You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@solr.apache.org by Karl Stoney <Ka...@autotrader.co.uk.INVALID> on 2021/03/17 19:16:07 UTC

REINDEXCOLLECTION gradually slows

Hey,
So we're trying to use REINDEXCOLLECTION again (solr 8.8) and similar to the last time we tried it (8.1) we see it gradually slow down; as you can see from the logs below.

I terminated it early but you can see the trend in the processingRatePerSecond.

I'm wondering if anyone has any idea's why this might happen? 🤷‍♂️. The machine isn't CPU or Memory constrained, so it makes reindexing larger collections really hard (we've had to code a separate app to do it).

Thanks



```
{"timestamp":"2021-03-17T19:04:44.703Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":156000,"targetDocs":787176,"percentComplete":20,"durationSeconds":125,"processingRatePerSecond":1248}
{"timestamp":"2021-03-17T19:04:49.719Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":156000,"targetDocs":787176,"percentComplete":20,"durationSeconds":130,"processingRatePerSecond":1200}
{"timestamp":"2021-03-17T19:04:54.736Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":172000,"targetDocs":787176,"percentComplete":22,"durationSeconds":135,"processingRatePerSecond":1274}
{"timestamp":"2021-03-17T19:04:59.751Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":172000,"targetDocs":787176,"percentComplete":22,"durationSeconds":140,"processingRatePerSecond":1229}
{"timestamp":"2021-03-17T19:05:04.765Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":188000,"targetDocs":787176,"percentComplete":24,"durationSeconds":145,"processingRatePerSecond":1297}
{"timestamp":"2021-03-17T19:05:09.782Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":188000,"targetDocs":787176,"percentComplete":24,"durationSeconds":150,"processingRatePerSecond":1253}
{"timestamp":"2021-03-17T19:05:14.798Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":202000,"targetDocs":787176,"percentComplete":26,"durationSeconds":155,"processingRatePerSecond":1303}
{"timestamp":"2021-03-17T19:05:19.816Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":202000,"targetDocs":787176,"percentComplete":26,"durationSeconds":161,"processingRatePerSecond":1255}
{"timestamp":"2021-03-17T19:05:24.832Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":212000,"targetDocs":787176,"percentComplete":27,"durationSeconds":166,"processingRatePerSecond":1277}
{"timestamp":"2021-03-17T19:05:29.850Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":212000,"targetDocs":787176,"percentComplete":27,"durationSeconds":171,"processingRatePerSecond":1240}
{"timestamp":"2021-03-17T19:05:34.866Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":220000,"targetDocs":787176,"percentComplete":28,"durationSeconds":176,"processingRatePerSecond":1250}
{"timestamp":"2021-03-17T19:05:39.884Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":220000,"targetDocs":787176,"percentComplete":28,"durationSeconds":181,"processingRatePerSecond":1215}
{"timestamp":"2021-03-17T19:05:44.899Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":228000,"targetDocs":787176,"percentComplete":29,"durationSeconds":186,"processingRatePerSecond":1226}
{"timestamp":"2021-03-17T19:05:49.916Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":228000,"targetDocs":787176,"percentComplete":29,"durationSeconds":191,"processingRatePerSecond":1194}
{"timestamp":"2021-03-17T19:05:54.931Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":236000,"targetDocs":787176,"percentComplete":30,"durationSeconds":196,"processingRatePerSecond":1204}
{"timestamp":"2021-03-17T19:05:59.947Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":236000,"targetDocs":787176,"percentComplete":30,"durationSeconds":201,"processingRatePerSecond":1174}
{"timestamp":"2021-03-17T19:06:04.963Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":244000,"targetDocs":787176,"percentComplete":31,"durationSeconds":206,"processingRatePerSecond":1184}
{"timestamp":"2021-03-17T19:06:09.977Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":244000,"targetDocs":787176,"percentComplete":31,"durationSeconds":211,"processingRatePerSecond":1156}
{"timestamp":"2021-03-17T19:06:14.993Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":254000,"targetDocs":787176,"percentComplete":32,"durationSeconds":216,"processingRatePerSecond":1176}
{"timestamp":"2021-03-17T19:06:20.007Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":254000,"targetDocs":787176,"percentComplete":32,"durationSeconds":221,"processingRatePerSecond":1149}
{"timestamp":"2021-03-17T19:06:25.024Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":264000,"targetDocs":787176,"percentComplete":34,"durationSeconds":226,"processingRatePerSecond":1168}
{"timestamp":"2021-03-17T19:06:30.037Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":264000,"targetDocs":787176,"percentComplete":34,"durationSeconds":231,"processingRatePerSecond":1143}
{"timestamp":"2021-03-17T19:06:35.053Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":272000,"targetDocs":787176,"percentComplete":35,"durationSeconds":236,"processingRatePerSecond":1153}
{"timestamp":"2021-03-17T19:06:40.067Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":272000,"targetDocs":787176,"percentComplete":35,"durationSeconds":241,"processingRatePerSecond":1129}
{"timestamp":"2021-03-17T19:06:45.083Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":282000,"targetDocs":787176,"percentComplete":36,"durationSeconds":246,"processingRatePerSecond":1146}
{"timestamp":"2021-03-17T19:06:50.099Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":282000,"targetDocs":787176,"percentComplete":36,"durationSeconds":251,"processingRatePerSecond":1124}
{"timestamp":"2021-03-17T19:06:55.114Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":292000,"targetDocs":787176,"percentComplete":37,"durationSeconds":256,"processingRatePerSecond":1141}
{"timestamp":"2021-03-17T19:07:00.140Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":292000,"targetDocs":787176,"percentComplete":37,"durationSeconds":261,"processingRatePerSecond":1119}
{"timestamp":"2021-03-17T19:07:05.155Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":304000,"targetDocs":787176,"percentComplete":39,"durationSeconds":266,"processingRatePerSecond":1143}
{"timestamp":"2021-03-17T19:07:10.174Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":304000,"targetDocs":787176,"percentComplete":39,"durationSeconds":271,"processingRatePerSecond":1122}
{"timestamp":"2021-03-17T19:07:15.189Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":316000,"targetDocs":787176,"percentComplete":40,"durationSeconds":276,"processingRatePerSecond":1145}
{"timestamp":"2021-03-17T19:07:20.203Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":328000,"targetDocs":787176,"percentComplete":42,"durationSeconds":281,"processingRatePerSecond":1167}
{"timestamp":"2021-03-17T19:07:25.220Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":328000,"targetDocs":787176,"percentComplete":42,"durationSeconds":286,"processingRatePerSecond":1147}
{"timestamp":"2021-03-17T19:07:30.237Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":344000,"targetDocs":787176,"percentComplete":44,"durationSeconds":291,"processingRatePerSecond":1182}
{"timestamp":"2021-03-17T19:07:35.253Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":344000,"targetDocs":787176,"percentComplete":44,"durationSeconds":296,"processingRatePerSecond":1162}
{"timestamp":"2021-03-17T19:07:40.268Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":356000,"targetDocs":787176,"percentComplete":45,"durationSeconds":301,"processingRatePerSecond":1183}
{"timestamp":"2021-03-17T19:07:45.283Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":356000,"targetDocs":787176,"percentComplete":45,"durationSeconds":306,"processingRatePerSecond":1163}
{"timestamp":"2021-03-17T19:07:50.297Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":368000,"targetDocs":787176,"percentComplete":47,"durationSeconds":311,"processingRatePerSecond":1183}
{"timestamp":"2021-03-17T19:07:55.313Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":368000,"targetDocs":787176,"percentComplete":47,"durationSeconds":316,"processingRatePerSecond":1165}
{"timestamp":"2021-03-17T19:08:00.329Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":376000,"targetDocs":787176,"percentComplete":48,"durationSeconds":321,"processingRatePerSecond":1171}
{"timestamp":"2021-03-17T19:08:05.345Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":376000,"targetDocs":787176,"percentComplete":48,"durationSeconds":326,"processingRatePerSecond":1153}
{"timestamp":"2021-03-17T19:08:10.360Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":386000,"targetDocs":787176,"percentComplete":49,"durationSeconds":331,"processingRatePerSecond":1166}
{"timestamp":"2021-03-17T19:08:15.377Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":386000,"targetDocs":787176,"percentComplete":49,"durationSeconds":336,"processingRatePerSecond":1149}
{"timestamp":"2021-03-17T19:08:20.391Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":394000,"targetDocs":787176,"percentComplete":50,"durationSeconds":341,"processingRatePerSecond":1155}
{"timestamp":"2021-03-17T19:08:25.408Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":394000,"targetDocs":787176,"percentComplete":50,"durationSeconds":346,"processingRatePerSecond":1139}
{"timestamp":"2021-03-17T19:08:30.423Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":404000,"targetDocs":787176,"percentComplete":51,"durationSeconds":351,"processingRatePerSecond":1151}
{"timestamp":"2021-03-17T19:08:35.440Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":404000,"targetDocs":787176,"percentComplete":51,"durationSeconds":356,"processingRatePerSecond":1135}
{"timestamp":"2021-03-17T19:08:40.457Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":414000,"targetDocs":787176,"percentComplete":53,"durationSeconds":361,"processingRatePerSecond":1147}
{"timestamp":"2021-03-17T19:08:45.471Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":414000,"targetDocs":787176,"percentComplete":53,"durationSeconds":366,"processingRatePerSecond":1131}
{"timestamp":"2021-03-17T19:08:50.490Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":422000,"targetDocs":787176,"percentComplete":54,"durationSeconds":371,"processingRatePerSecond":1137}
{"timestamp":"2021-03-17T19:08:55.506Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":422000,"targetDocs":787176,"percentComplete":54,"durationSeconds":376,"processingRatePerSecond":1122}
{"timestamp":"2021-03-17T19:09:00.523Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":432000,"targetDocs":787176,"percentComplete":55,"durationSeconds":381,"processingRatePerSecond":1134}
{"timestamp":"2021-03-17T19:09:05.538Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":432000,"targetDocs":787176,"percentComplete":55,"durationSeconds":386,"processingRatePerSecond":1119}
{"timestamp":"2021-03-17T19:09:10.554Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":440000,"targetDocs":787176,"percentComplete":56,"durationSeconds":391,"processingRatePerSecond":1125}
{"timestamp":"2021-03-17T19:09:15.571Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":440000,"targetDocs":787176,"percentComplete":56,"durationSeconds":396,"processingRatePerSecond":1111}
{"timestamp":"2021-03-17T19:09:20.587Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":452000,"targetDocs":787176,"percentComplete":57,"durationSeconds":401,"processingRatePerSecond":1127}
{"timestamp":"2021-03-17T19:09:25.603Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":452000,"targetDocs":787176,"percentComplete":57,"durationSeconds":406,"processingRatePerSecond":1113}
{"timestamp":"2021-03-17T19:09:30.618Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":464000,"targetDocs":787176,"percentComplete":59,"durationSeconds":411,"processingRatePerSecond":1129}
{"timestamp":"2021-03-17T19:09:35.635Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":464000,"targetDocs":787176,"percentComplete":59,"durationSeconds":416,"processingRatePerSecond":1115}
{"timestamp":"2021-03-17T19:09:40.651Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":476000,"targetDocs":787176,"percentComplete":60,"durationSeconds":421,"processingRatePerSecond":1131}
{"timestamp":"2021-03-17T19:09:45.668Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":476000,"targetDocs":787176,"percentComplete":60,"durationSeconds":426,"processingRatePerSecond":1117}
{"timestamp":"2021-03-17T19:09:50.683Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":486000,"targetDocs":787176,"percentComplete":62,"durationSeconds":431,"processingRatePerSecond":1128}
{"timestamp":"2021-03-17T19:09:55.698Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":486000,"targetDocs":787176,"percentComplete":62,"durationSeconds":436,"processingRatePerSecond":1115}
{"timestamp":"2021-03-17T19:10:00.714Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":496000,"targetDocs":787176,"percentComplete":63,"durationSeconds":441,"processingRatePerSecond":1125}
{"timestamp":"2021-03-17T19:10:05.731Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":496000,"targetDocs":787176,"percentComplete":63,"durationSeconds":446,"processingRatePerSecond":1112}
{"timestamp":"2021-03-17T19:10:10.746Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":506000,"targetDocs":787176,"percentComplete":64,"durationSeconds":451,"processingRatePerSecond":1122}
{"timestamp":"2021-03-17T19:10:15.763Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":506000,"targetDocs":787176,"percentComplete":64,"durationSeconds":456,"processingRatePerSecond":1110}
{"timestamp":"2021-03-17T19:10:20.778Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":516000,"targetDocs":787176,"percentComplete":66,"durationSeconds":461,"processingRatePerSecond":1119}
{"timestamp":"2021-03-17T19:10:25.792Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":516000,"targetDocs":787176,"percentComplete":66,"durationSeconds":466,"processingRatePerSecond":1107}
{"timestamp":"2021-03-17T19:10:30.808Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":528000,"targetDocs":787176,"percentComplete":67,"durationSeconds":472,"processingRatePerSecond":1119}
{"timestamp":"2021-03-17T19:10:35.821Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":528000,"targetDocs":787176,"percentComplete":67,"durationSeconds":477,"processingRatePerSecond":1107}
{"timestamp":"2021-03-17T19:10:40.840Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":538000,"targetDocs":787176,"percentComplete":68,"durationSeconds":482,"processingRatePerSecond":1116}
{"timestamp":"2021-03-17T19:10:45.854Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":538000,"targetDocs":787176,"percentComplete":68,"durationSeconds":487,"processingRatePerSecond":1105}
{"timestamp":"2021-03-17T19:10:50.870Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":544000,"targetDocs":787176,"percentComplete":69,"durationSeconds":492,"processingRatePerSecond":1106}
{"timestamp":"2021-03-17T19:10:55.887Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":544000,"targetDocs":787176,"percentComplete":69,"durationSeconds":497,"processingRatePerSecond":1095}
{"timestamp":"2021-03-17T19:11:00.901Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":548000,"targetDocs":787176,"percentComplete":70,"durationSeconds":502,"processingRatePerSecond":1092}
{"timestamp":"2021-03-17T19:11:05.916Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":548000,"targetDocs":787176,"percentComplete":70,"durationSeconds":507,"processingRatePerSecond":1081}
{"timestamp":"2021-03-17T19:11:10.930Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":552000,"targetDocs":787176,"percentComplete":70,"durationSeconds":512,"processingRatePerSecond":1078}
{"timestamp":"2021-03-17T19:11:15.946Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":552000,"targetDocs":787176,"percentComplete":70,"durationSeconds":517,"processingRatePerSecond":1068}
{"timestamp":"2021-03-17T19:11:20.962Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":554000,"targetDocs":787176,"percentComplete":70,"durationSeconds":522,"processingRatePerSecond":1061}
{"timestamp":"2021-03-17T19:11:25.978Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":554000,"targetDocs":787176,"percentComplete":70,"durationSeconds":527,"processingRatePerSecond":1051}
{"timestamp":"2021-03-17T19:11:30.993Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":558000,"targetDocs":787176,"percentComplete":71,"durationSeconds":532,"processingRatePerSecond":1049}
{"timestamp":"2021-03-17T19:11:36.012Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":558000,"targetDocs":787176,"percentComplete":71,"durationSeconds":537,"processingRatePerSecond":1039}
{"timestamp":"2021-03-17T19:11:41.038Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":562000,"targetDocs":787176,"percentComplete":71,"durationSeconds":542,"processingRatePerSecond":1037}
{"timestamp":"2021-03-17T19:11:46.055Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":562000,"targetDocs":787176,"percentComplete":71,"durationSeconds":547,"processingRatePerSecond":1027}
{"timestamp":"2021-03-17T19:11:51.071Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":566000,"targetDocs":787176,"percentComplete":72,"durationSeconds":552,"processingRatePerSecond":1025}
{"timestamp":"2021-03-17T19:11:56.087Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":566000,"targetDocs":787176,"percentComplete":72,"durationSeconds":557,"processingRatePerSecond":1016}
{"timestamp":"2021-03-17T19:12:01.104Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":570000,"targetDocs":787176,"percentComplete":72,"durationSeconds":562,"processingRatePerSecond":1014}
{"timestamp":"2021-03-17T19:12:06.123Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":570000,"targetDocs":787176,"percentComplete":72,"durationSeconds":567,"processingRatePerSecond":1005}
{"timestamp":"2021-03-17T19:12:11.143Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":576000,"targetDocs":787176,"percentComplete":73,"durationSeconds":572,"processingRatePerSecond":1007}
{"timestamp":"2021-03-17T19:12:16.158Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":576000,"targetDocs":787176,"percentComplete":73,"durationSeconds":577,"processingRatePerSecond":998}
{"timestamp":"2021-03-17T19:12:21.174Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":580000,"targetDocs":787176,"percentComplete":74,"durationSeconds":582,"processingRatePerSecond":997}
{"timestamp":"2021-03-17T19:12:26.189Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":580000,"targetDocs":787176,"percentComplete":74,"durationSeconds":587,"processingRatePerSecond":988}
{"timestamp":"2021-03-17T19:12:31.205Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":586000,"targetDocs":787176,"percentComplete":74,"durationSeconds":592,"processingRatePerSecond":990}
{"timestamp":"2021-03-17T19:12:36.220Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":586000,"targetDocs":787176,"percentComplete":74,"durationSeconds":597,"processingRatePerSecond":982}
{"timestamp":"2021-03-17T19:12:41.236Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":590000,"targetDocs":787176,"percentComplete":75,"durationSeconds":602,"processingRatePerSecond":980}
{"timestamp":"2021-03-17T19:12:46.252Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":590000,"targetDocs":787176,"percentComplete":75,"durationSeconds":607,"processingRatePerSecond":972}
{"timestamp":"2021-03-17T19:12:51.266Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":594000,"targetDocs":787176,"percentComplete":75,"durationSeconds":612,"processingRatePerSecond":971}
{"timestamp":"2021-03-17T19:12:56.282Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":594000,"targetDocs":787176,"percentComplete":75,"durationSeconds":617,"processingRatePerSecond":963}
```
Unless expressly stated otherwise in this email, this e-mail is sent on behalf of Auto Trader Limited Registered Office: 1 Tony Wilson Place, Manchester, Lancashire, M15 4FN (Registered in England No. 03909628). Auto Trader Limited is part of the Auto Trader Group Plc group. This email and any files transmitted with it are confidential and may be legally privileged, and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender. This email message has been swept for the presence of computer viruses.

Re: REINDEXCOLLECTION gradually slows

Posted by Karl Stoney <Ka...@autotrader.co.uk.INVALID>.

And a bit more... it's almost certainly to do with the commits happening after every batch.
When i set rows rather high (for example; 25k) I hold a sustained throughput for a longer duration and on the smaller index, things finish faster.

Downside is obviously more memory being used (but we have quite a bit).

Which really leads me down a route of "how to minimise the ever increasing costs of the commits.

FYI the index is about 30gb

This is my merge policy:

  <indexConfig>
    <ramBufferSizeMB>128</ramBufferSizeMB>
    <mergePolicyFactory class="org.apache.solr.index.TieredMergePolicyFactory">
      <int name="maxMergeAtOnce">5</int>
      <int name="segmentsPerTier">5</int>
      <int name="maxMergedSegmentMB">5000</int>
      <int name="deletesPctAllowed">25</int>
    </mergePolicyFactory>
    <mergeScheduler class="org.apache.lucene.index.ConcurrentMergeScheduler">
      <int name="maxThreadCount">10</int>
      <int name="maxMergeCount">15</int>
    </mergeScheduler>
    <mergedSegmentWarmer class="org.apache.lucene.index.SimpleMergedSegmentWarmer" />
  </indexConfig>

I'm thinking I could disable via configAPI cache autoWarming on the index for the reindex duration?
________________________________
From: Karl Stoney <Ka...@autotrader.co.uk>
Sent: 18 March 2021 10:21
To: solr-user@lucene.apache.org <so...@lucene.apache.org>; users@solr.apache.org <us...@solr.apache.org>
Subject: Re: REINDEXCOLLECTION gradually slows

A bit more data on this to help... really welcome any ideas as the slow down over time just means we're never really able to finish a reindex via REINDEXCOLLECTION

We have disabled auto commits on the destination collection during the reindex, however we're still seeing commits happen as the reindexing happens.  Does the REINDEXCOLLECTION perform a commit after every batch or every N seconds?  They seem to be bang on about 30 seconds?

```
/karl-test/config/updateHandler
{

  *
responseHeader:
{
     *
status: 0,
     *
QTime: 0
},
  *
config:
{
     *
updateHandler:
{
        *
indexWriter:
{
           *
closeWaitsForMerges: true
},
        *
commitWithin:
{
           *
softCommit: true
},
        *
autoCommit:
{
           *
maxDocs: -1,
           *
maxTime: -1,
           *
openSearcher: false
},
        *
autoSoftCommit:
{
           *
maxDocs: -1,
           *
maxTime: 900000
}
}
}

}
```

This in turn seems to be causing cache warming, which obviously gets increasingly slower and increasingly adds load.

________________________________
From: Karl Stoney <Ka...@autotrader.co.uk.INVALID>
Sent: 17 March 2021 19:16
To: solr-user@lucene.apache.org <so...@lucene.apache.org>
Subject: REINDEXCOLLECTION gradually slows

Hey,
So we're trying to use REINDEXCOLLECTION again (solr 8.8) and similar to the last time we tried it (8.1) we see it gradually slow down; as you can see from the logs below.

I terminated it early but you can see the trend in the processingRatePerSecond.

I'm wondering if anyone has any idea's why this might happen? 🤷‍♂️. The machine isn't CPU or Memory constrained, so it makes reindexing larger collections really hard (we've had to code a separate app to do it).

Thanks



```
{"timestamp":"2021-03-17T19:04:44.703Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":156000,"targetDocs":787176,"percentComplete":20,"durationSeconds":125,"processingRatePerSecond":1248}
{"timestamp":"2021-03-17T19:04:49.719Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":156000,"targetDocs":787176,"percentComplete":20,"durationSeconds":130,"processingRatePerSecond":1200}
{"timestamp":"2021-03-17T19:04:54.736Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":172000,"targetDocs":787176,"percentComplete":22,"durationSeconds":135,"processingRatePerSecond":1274}
{"timestamp":"2021-03-17T19:04:59.751Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":172000,"targetDocs":787176,"percentComplete":22,"durationSeconds":140,"processingRatePerSecond":1229}
{"timestamp":"2021-03-17T19:05:04.765Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":188000,"targetDocs":787176,"percentComplete":24,"durationSeconds":145,"processingRatePerSecond":1297}
{"timestamp":"2021-03-17T19:05:09.782Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":188000,"targetDocs":787176,"percentComplete":24,"durationSeconds":150,"processingRatePerSecond":1253}
{"timestamp":"2021-03-17T19:05:14.798Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":202000,"targetDocs":787176,"percentComplete":26,"durationSeconds":155,"processingRatePerSecond":1303}
{"timestamp":"2021-03-17T19:05:19.816Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":202000,"targetDocs":787176,"percentComplete":26,"durationSeconds":161,"processingRatePerSecond":1255}
{"timestamp":"2021-03-17T19:05:24.832Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":212000,"targetDocs":787176,"percentComplete":27,"durationSeconds":166,"processingRatePerSecond":1277}
{"timestamp":"2021-03-17T19:05:29.850Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":212000,"targetDocs":787176,"percentComplete":27,"durationSeconds":171,"processingRatePerSecond":1240}
{"timestamp":"2021-03-17T19:05:34.866Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":220000,"targetDocs":787176,"percentComplete":28,"durationSeconds":176,"processingRatePerSecond":1250}
{"timestamp":"2021-03-17T19:05:39.884Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":220000,"targetDocs":787176,"percentComplete":28,"durationSeconds":181,"processingRatePerSecond":1215}
{"timestamp":"2021-03-17T19:05:44.899Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":228000,"targetDocs":787176,"percentComplete":29,"durationSeconds":186,"processingRatePerSecond":1226}
{"timestamp":"2021-03-17T19:05:49.916Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":228000,"targetDocs":787176,"percentComplete":29,"durationSeconds":191,"processingRatePerSecond":1194}
{"timestamp":"2021-03-17T19:05:54.931Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":236000,"targetDocs":787176,"percentComplete":30,"durationSeconds":196,"processingRatePerSecond":1204}
{"timestamp":"2021-03-17T19:05:59.947Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":236000,"targetDocs":787176,"percentComplete":30,"durationSeconds":201,"processingRatePerSecond":1174}
{"timestamp":"2021-03-17T19:06:04.963Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":244000,"targetDocs":787176,"percentComplete":31,"durationSeconds":206,"processingRatePerSecond":1184}
{"timestamp":"2021-03-17T19:06:09.977Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":244000,"targetDocs":787176,"percentComplete":31,"durationSeconds":211,"processingRatePerSecond":1156}
{"timestamp":"2021-03-17T19:06:14.993Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":254000,"targetDocs":787176,"percentComplete":32,"durationSeconds":216,"processingRatePerSecond":1176}
{"timestamp":"2021-03-17T19:06:20.007Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":254000,"targetDocs":787176,"percentComplete":32,"durationSeconds":221,"processingRatePerSecond":1149}
{"timestamp":"2021-03-17T19:06:25.024Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":264000,"targetDocs":787176,"percentComplete":34,"durationSeconds":226,"processingRatePerSecond":1168}
{"timestamp":"2021-03-17T19:06:30.037Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":264000,"targetDocs":787176,"percentComplete":34,"durationSeconds":231,"processingRatePerSecond":1143}
{"timestamp":"2021-03-17T19:06:35.053Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":272000,"targetDocs":787176,"percentComplete":35,"durationSeconds":236,"processingRatePerSecond":1153}
{"timestamp":"2021-03-17T19:06:40.067Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":272000,"targetDocs":787176,"percentComplete":35,"durationSeconds":241,"processingRatePerSecond":1129}
{"timestamp":"2021-03-17T19:06:45.083Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":282000,"targetDocs":787176,"percentComplete":36,"durationSeconds":246,"processingRatePerSecond":1146}
{"timestamp":"2021-03-17T19:06:50.099Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":282000,"targetDocs":787176,"percentComplete":36,"durationSeconds":251,"processingRatePerSecond":1124}
{"timestamp":"2021-03-17T19:06:55.114Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":292000,"targetDocs":787176,"percentComplete":37,"durationSeconds":256,"processingRatePerSecond":1141}
{"timestamp":"2021-03-17T19:07:00.140Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":292000,"targetDocs":787176,"percentComplete":37,"durationSeconds":261,"processingRatePerSecond":1119}
{"timestamp":"2021-03-17T19:07:05.155Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":304000,"targetDocs":787176,"percentComplete":39,"durationSeconds":266,"processingRatePerSecond":1143}
{"timestamp":"2021-03-17T19:07:10.174Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":304000,"targetDocs":787176,"percentComplete":39,"durationSeconds":271,"processingRatePerSecond":1122}
{"timestamp":"2021-03-17T19:07:15.189Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":316000,"targetDocs":787176,"percentComplete":40,"durationSeconds":276,"processingRatePerSecond":1145}
{"timestamp":"2021-03-17T19:07:20.203Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":328000,"targetDocs":787176,"percentComplete":42,"durationSeconds":281,"processingRatePerSecond":1167}
{"timestamp":"2021-03-17T19:07:25.220Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":328000,"targetDocs":787176,"percentComplete":42,"durationSeconds":286,"processingRatePerSecond":1147}
{"timestamp":"2021-03-17T19:07:30.237Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":344000,"targetDocs":787176,"percentComplete":44,"durationSeconds":291,"processingRatePerSecond":1182}
{"timestamp":"2021-03-17T19:07:35.253Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":344000,"targetDocs":787176,"percentComplete":44,"durationSeconds":296,"processingRatePerSecond":1162}
{"timestamp":"2021-03-17T19:07:40.268Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":356000,"targetDocs":787176,"percentComplete":45,"durationSeconds":301,"processingRatePerSecond":1183}
{"timestamp":"2021-03-17T19:07:45.283Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":356000,"targetDocs":787176,"percentComplete":45,"durationSeconds":306,"processingRatePerSecond":1163}
{"timestamp":"2021-03-17T19:07:50.297Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":368000,"targetDocs":787176,"percentComplete":47,"durationSeconds":311,"processingRatePerSecond":1183}
{"timestamp":"2021-03-17T19:07:55.313Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":368000,"targetDocs":787176,"percentComplete":47,"durationSeconds":316,"processingRatePerSecond":1165}
{"timestamp":"2021-03-17T19:08:00.329Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":376000,"targetDocs":787176,"percentComplete":48,"durationSeconds":321,"processingRatePerSecond":1171}
{"timestamp":"2021-03-17T19:08:05.345Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":376000,"targetDocs":787176,"percentComplete":48,"durationSeconds":326,"processingRatePerSecond":1153}
{"timestamp":"2021-03-17T19:08:10.360Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":386000,"targetDocs":787176,"percentComplete":49,"durationSeconds":331,"processingRatePerSecond":1166}
{"timestamp":"2021-03-17T19:08:15.377Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":386000,"targetDocs":787176,"percentComplete":49,"durationSeconds":336,"processingRatePerSecond":1149}
{"timestamp":"2021-03-17T19:08:20.391Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":394000,"targetDocs":787176,"percentComplete":50,"durationSeconds":341,"processingRatePerSecond":1155}
{"timestamp":"2021-03-17T19:08:25.408Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":394000,"targetDocs":787176,"percentComplete":50,"durationSeconds":346,"processingRatePerSecond":1139}
{"timestamp":"2021-03-17T19:08:30.423Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":404000,"targetDocs":787176,"percentComplete":51,"durationSeconds":351,"processingRatePerSecond":1151}
{"timestamp":"2021-03-17T19:08:35.440Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":404000,"targetDocs":787176,"percentComplete":51,"durationSeconds":356,"processingRatePerSecond":1135}
{"timestamp":"2021-03-17T19:08:40.457Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":414000,"targetDocs":787176,"percentComplete":53,"durationSeconds":361,"processingRatePerSecond":1147}
{"timestamp":"2021-03-17T19:08:45.471Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":414000,"targetDocs":787176,"percentComplete":53,"durationSeconds":366,"processingRatePerSecond":1131}
{"timestamp":"2021-03-17T19:08:50.490Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":422000,"targetDocs":787176,"percentComplete":54,"durationSeconds":371,"processingRatePerSecond":1137}
{"timestamp":"2021-03-17T19:08:55.506Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":422000,"targetDocs":787176,"percentComplete":54,"durationSeconds":376,"processingRatePerSecond":1122}
{"timestamp":"2021-03-17T19:09:00.523Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":432000,"targetDocs":787176,"percentComplete":55,"durationSeconds":381,"processingRatePerSecond":1134}
{"timestamp":"2021-03-17T19:09:05.538Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":432000,"targetDocs":787176,"percentComplete":55,"durationSeconds":386,"processingRatePerSecond":1119}
{"timestamp":"2021-03-17T19:09:10.554Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":440000,"targetDocs":787176,"percentComplete":56,"durationSeconds":391,"processingRatePerSecond":1125}
{"timestamp":"2021-03-17T19:09:15.571Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":440000,"targetDocs":787176,"percentComplete":56,"durationSeconds":396,"processingRatePerSecond":1111}
{"timestamp":"2021-03-17T19:09:20.587Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":452000,"targetDocs":787176,"percentComplete":57,"durationSeconds":401,"processingRatePerSecond":1127}
{"timestamp":"2021-03-17T19:09:25.603Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":452000,"targetDocs":787176,"percentComplete":57,"durationSeconds":406,"processingRatePerSecond":1113}
{"timestamp":"2021-03-17T19:09:30.618Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":464000,"targetDocs":787176,"percentComplete":59,"durationSeconds":411,"processingRatePerSecond":1129}
{"timestamp":"2021-03-17T19:09:35.635Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":464000,"targetDocs":787176,"percentComplete":59,"durationSeconds":416,"processingRatePerSecond":1115}
{"timestamp":"2021-03-17T19:09:40.651Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":476000,"targetDocs":787176,"percentComplete":60,"durationSeconds":421,"processingRatePerSecond":1131}
{"timestamp":"2021-03-17T19:09:45.668Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":476000,"targetDocs":787176,"percentComplete":60,"durationSeconds":426,"processingRatePerSecond":1117}
{"timestamp":"2021-03-17T19:09:50.683Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":486000,"targetDocs":787176,"percentComplete":62,"durationSeconds":431,"processingRatePerSecond":1128}
{"timestamp":"2021-03-17T19:09:55.698Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":486000,"targetDocs":787176,"percentComplete":62,"durationSeconds":436,"processingRatePerSecond":1115}
{"timestamp":"2021-03-17T19:10:00.714Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":496000,"targetDocs":787176,"percentComplete":63,"durationSeconds":441,"processingRatePerSecond":1125}
{"timestamp":"2021-03-17T19:10:05.731Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":496000,"targetDocs":787176,"percentComplete":63,"durationSeconds":446,"processingRatePerSecond":1112}
{"timestamp":"2021-03-17T19:10:10.746Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":506000,"targetDocs":787176,"percentComplete":64,"durationSeconds":451,"processingRatePerSecond":1122}
{"timestamp":"2021-03-17T19:10:15.763Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":506000,"targetDocs":787176,"percentComplete":64,"durationSeconds":456,"processingRatePerSecond":1110}
{"timestamp":"2021-03-17T19:10:20.778Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":516000,"targetDocs":787176,"percentComplete":66,"durationSeconds":461,"processingRatePerSecond":1119}
{"timestamp":"2021-03-17T19:10:25.792Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":516000,"targetDocs":787176,"percentComplete":66,"durationSeconds":466,"processingRatePerSecond":1107}
{"timestamp":"2021-03-17T19:10:30.808Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":528000,"targetDocs":787176,"percentComplete":67,"durationSeconds":472,"processingRatePerSecond":1119}
{"timestamp":"2021-03-17T19:10:35.821Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":528000,"targetDocs":787176,"percentComplete":67,"durationSeconds":477,"processingRatePerSecond":1107}
{"timestamp":"2021-03-17T19:10:40.840Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":538000,"targetDocs":787176,"percentComplete":68,"durationSeconds":482,"processingRatePerSecond":1116}
{"timestamp":"2021-03-17T19:10:45.854Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":538000,"targetDocs":787176,"percentComplete":68,"durationSeconds":487,"processingRatePerSecond":1105}
{"timestamp":"2021-03-17T19:10:50.870Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":544000,"targetDocs":787176,"percentComplete":69,"durationSeconds":492,"processingRatePerSecond":1106}
{"timestamp":"2021-03-17T19:10:55.887Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":544000,"targetDocs":787176,"percentComplete":69,"durationSeconds":497,"processingRatePerSecond":1095}
{"timestamp":"2021-03-17T19:11:00.901Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":548000,"targetDocs":787176,"percentComplete":70,"durationSeconds":502,"processingRatePerSecond":1092}
{"timestamp":"2021-03-17T19:11:05.916Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":548000,"targetDocs":787176,"percentComplete":70,"durationSeconds":507,"processingRatePerSecond":1081}
{"timestamp":"2021-03-17T19:11:10.930Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":552000,"targetDocs":787176,"percentComplete":70,"durationSeconds":512,"processingRatePerSecond":1078}
{"timestamp":"2021-03-17T19:11:15.946Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":552000,"targetDocs":787176,"percentComplete":70,"durationSeconds":517,"processingRatePerSecond":1068}
{"timestamp":"2021-03-17T19:11:20.962Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":554000,"targetDocs":787176,"percentComplete":70,"durationSeconds":522,"processingRatePerSecond":1061}
{"timestamp":"2021-03-17T19:11:25.978Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":554000,"targetDocs":787176,"percentComplete":70,"durationSeconds":527,"processingRatePerSecond":1051}
{"timestamp":"2021-03-17T19:11:30.993Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":558000,"targetDocs":787176,"percentComplete":71,"durationSeconds":532,"processingRatePerSecond":1049}
{"timestamp":"2021-03-17T19:11:36.012Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":558000,"targetDocs":787176,"percentComplete":71,"durationSeconds":537,"processingRatePerSecond":1039}
{"timestamp":"2021-03-17T19:11:41.038Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":562000,"targetDocs":787176,"percentComplete":71,"durationSeconds":542,"processingRatePerSecond":1037}
{"timestamp":"2021-03-17T19:11:46.055Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":562000,"targetDocs":787176,"percentComplete":71,"durationSeconds":547,"processingRatePerSecond":1027}
{"timestamp":"2021-03-17T19:11:51.071Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":566000,"targetDocs":787176,"percentComplete":72,"durationSeconds":552,"processingRatePerSecond":1025}
{"timestamp":"2021-03-17T19:11:56.087Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":566000,"targetDocs":787176,"percentComplete":72,"durationSeconds":557,"processingRatePerSecond":1016}
{"timestamp":"2021-03-17T19:12:01.104Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":570000,"targetDocs":787176,"percentComplete":72,"durationSeconds":562,"processingRatePerSecond":1014}
{"timestamp":"2021-03-17T19:12:06.123Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":570000,"targetDocs":787176,"percentComplete":72,"durationSeconds":567,"processingRatePerSecond":1005}
{"timestamp":"2021-03-17T19:12:11.143Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":576000,"targetDocs":787176,"percentComplete":73,"durationSeconds":572,"processingRatePerSecond":1007}
{"timestamp":"2021-03-17T19:12:16.158Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":576000,"targetDocs":787176,"percentComplete":73,"durationSeconds":577,"processingRatePerSecond":998}
{"timestamp":"2021-03-17T19:12:21.174Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":580000,"targetDocs":787176,"percentComplete":74,"durationSeconds":582,"processingRatePerSecond":997}
{"timestamp":"2021-03-17T19:12:26.189Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":580000,"targetDocs":787176,"percentComplete":74,"durationSeconds":587,"processingRatePerSecond":988}
{"timestamp":"2021-03-17T19:12:31.205Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":586000,"targetDocs":787176,"percentComplete":74,"durationSeconds":592,"processingRatePerSecond":990}
{"timestamp":"2021-03-17T19:12:36.220Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":586000,"targetDocs":787176,"percentComplete":74,"durationSeconds":597,"processingRatePerSecond":982}
{"timestamp":"2021-03-17T19:12:41.236Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":590000,"targetDocs":787176,"percentComplete":75,"durationSeconds":602,"processingRatePerSecond":980}
{"timestamp":"2021-03-17T19:12:46.252Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":590000,"targetDocs":787176,"percentComplete":75,"durationSeconds":607,"processingRatePerSecond":972}
{"timestamp":"2021-03-17T19:12:51.266Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":594000,"targetDocs":787176,"percentComplete":75,"durationSeconds":612,"processingRatePerSecond":971}
{"timestamp":"2021-03-17T19:12:56.282Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":594000,"targetDocs":787176,"percentComplete":75,"durationSeconds":617,"processingRatePerSecond":963}
```
Unless expressly stated otherwise in this email, this e-mail is sent on behalf of Auto Trader Limited Registered Office: 1 Tony Wilson Place, Manchester, Lancashire, M15 4FN (Registered in England No. 03909628). Auto Trader Limited is part of the Auto Trader Group Plc group. This email and any files transmitted with it are confidential and may be legally privileged, and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender. This email message has been swept for the presence of computer viruses.
Unless expressly stated otherwise in this email, this e-mail is sent on behalf of Auto Trader Limited Registered Office: 1 Tony Wilson Place, Manchester, Lancashire, M15 4FN (Registered in England No. 03909628). Auto Trader Limited is part of the Auto Trader Group Plc group. This email and any files transmitted with it are confidential and may be legally privileged, and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender. This email message has been swept for the presence of computer viruses.

Re: REINDEXCOLLECTION gradually slows

Posted by Karl Stoney <Ka...@autotrader.co.uk.INVALID>.

A bit more data on this to help... really welcome any ideas as the slow down over time just means we're never really able to finish a reindex via REINDEXCOLLECTION

We have disabled auto commits on the destination collection during the reindex, however we're still seeing commits happen as the reindexing happens.  Does the REINDEXCOLLECTION perform a commit after every batch or every N seconds?  They seem to be bang on about 30 seconds?

```
/karl-test/config/updateHandler
{

  *
responseHeader:
{
     *
status: 0,
     *
QTime: 0
},
  *
config:
{
     *
updateHandler:
{
        *
indexWriter:
{
           *
closeWaitsForMerges: true
},
        *
commitWithin:
{
           *
softCommit: true
},
        *
autoCommit:
{
           *
maxDocs: -1,
           *
maxTime: -1,
           *
openSearcher: false
},
        *
autoSoftCommit:
{
           *
maxDocs: -1,
           *
maxTime: 900000
}
}
}

}
```

This in turn seems to be causing cache warming, which obviously gets increasingly slower and increasingly adds load.

________________________________
From: Karl Stoney <Ka...@autotrader.co.uk.INVALID>
Sent: 17 March 2021 19:16
To: solr-user@lucene.apache.org <so...@lucene.apache.org>
Subject: REINDEXCOLLECTION gradually slows

Hey,
So we're trying to use REINDEXCOLLECTION again (solr 8.8) and similar to the last time we tried it (8.1) we see it gradually slow down; as you can see from the logs below.

I terminated it early but you can see the trend in the processingRatePerSecond.

I'm wondering if anyone has any idea's why this might happen? 🤷‍♂️. The machine isn't CPU or Memory constrained, so it makes reindexing larger collections really hard (we've had to code a separate app to do it).

Thanks



```
{"timestamp":"2021-03-17T19:04:44.703Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":156000,"targetDocs":787176,"percentComplete":20,"durationSeconds":125,"processingRatePerSecond":1248}
{"timestamp":"2021-03-17T19:04:49.719Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":156000,"targetDocs":787176,"percentComplete":20,"durationSeconds":130,"processingRatePerSecond":1200}
{"timestamp":"2021-03-17T19:04:54.736Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":172000,"targetDocs":787176,"percentComplete":22,"durationSeconds":135,"processingRatePerSecond":1274}
{"timestamp":"2021-03-17T19:04:59.751Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":172000,"targetDocs":787176,"percentComplete":22,"durationSeconds":140,"processingRatePerSecond":1229}
{"timestamp":"2021-03-17T19:05:04.765Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":188000,"targetDocs":787176,"percentComplete":24,"durationSeconds":145,"processingRatePerSecond":1297}
{"timestamp":"2021-03-17T19:05:09.782Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":188000,"targetDocs":787176,"percentComplete":24,"durationSeconds":150,"processingRatePerSecond":1253}
{"timestamp":"2021-03-17T19:05:14.798Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":202000,"targetDocs":787176,"percentComplete":26,"durationSeconds":155,"processingRatePerSecond":1303}
{"timestamp":"2021-03-17T19:05:19.816Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":202000,"targetDocs":787176,"percentComplete":26,"durationSeconds":161,"processingRatePerSecond":1255}
{"timestamp":"2021-03-17T19:05:24.832Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":212000,"targetDocs":787176,"percentComplete":27,"durationSeconds":166,"processingRatePerSecond":1277}
{"timestamp":"2021-03-17T19:05:29.850Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":212000,"targetDocs":787176,"percentComplete":27,"durationSeconds":171,"processingRatePerSecond":1240}
{"timestamp":"2021-03-17T19:05:34.866Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":220000,"targetDocs":787176,"percentComplete":28,"durationSeconds":176,"processingRatePerSecond":1250}
{"timestamp":"2021-03-17T19:05:39.884Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":220000,"targetDocs":787176,"percentComplete":28,"durationSeconds":181,"processingRatePerSecond":1215}
{"timestamp":"2021-03-17T19:05:44.899Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":228000,"targetDocs":787176,"percentComplete":29,"durationSeconds":186,"processingRatePerSecond":1226}
{"timestamp":"2021-03-17T19:05:49.916Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":228000,"targetDocs":787176,"percentComplete":29,"durationSeconds":191,"processingRatePerSecond":1194}
{"timestamp":"2021-03-17T19:05:54.931Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":236000,"targetDocs":787176,"percentComplete":30,"durationSeconds":196,"processingRatePerSecond":1204}
{"timestamp":"2021-03-17T19:05:59.947Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":236000,"targetDocs":787176,"percentComplete":30,"durationSeconds":201,"processingRatePerSecond":1174}
{"timestamp":"2021-03-17T19:06:04.963Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":244000,"targetDocs":787176,"percentComplete":31,"durationSeconds":206,"processingRatePerSecond":1184}
{"timestamp":"2021-03-17T19:06:09.977Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":244000,"targetDocs":787176,"percentComplete":31,"durationSeconds":211,"processingRatePerSecond":1156}
{"timestamp":"2021-03-17T19:06:14.993Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":254000,"targetDocs":787176,"percentComplete":32,"durationSeconds":216,"processingRatePerSecond":1176}
{"timestamp":"2021-03-17T19:06:20.007Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":254000,"targetDocs":787176,"percentComplete":32,"durationSeconds":221,"processingRatePerSecond":1149}
{"timestamp":"2021-03-17T19:06:25.024Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":264000,"targetDocs":787176,"percentComplete":34,"durationSeconds":226,"processingRatePerSecond":1168}
{"timestamp":"2021-03-17T19:06:30.037Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":264000,"targetDocs":787176,"percentComplete":34,"durationSeconds":231,"processingRatePerSecond":1143}
{"timestamp":"2021-03-17T19:06:35.053Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":272000,"targetDocs":787176,"percentComplete":35,"durationSeconds":236,"processingRatePerSecond":1153}
{"timestamp":"2021-03-17T19:06:40.067Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":272000,"targetDocs":787176,"percentComplete":35,"durationSeconds":241,"processingRatePerSecond":1129}
{"timestamp":"2021-03-17T19:06:45.083Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":282000,"targetDocs":787176,"percentComplete":36,"durationSeconds":246,"processingRatePerSecond":1146}
{"timestamp":"2021-03-17T19:06:50.099Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":282000,"targetDocs":787176,"percentComplete":36,"durationSeconds":251,"processingRatePerSecond":1124}
{"timestamp":"2021-03-17T19:06:55.114Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":292000,"targetDocs":787176,"percentComplete":37,"durationSeconds":256,"processingRatePerSecond":1141}
{"timestamp":"2021-03-17T19:07:00.140Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":292000,"targetDocs":787176,"percentComplete":37,"durationSeconds":261,"processingRatePerSecond":1119}
{"timestamp":"2021-03-17T19:07:05.155Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":304000,"targetDocs":787176,"percentComplete":39,"durationSeconds":266,"processingRatePerSecond":1143}
{"timestamp":"2021-03-17T19:07:10.174Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":304000,"targetDocs":787176,"percentComplete":39,"durationSeconds":271,"processingRatePerSecond":1122}
{"timestamp":"2021-03-17T19:07:15.189Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":316000,"targetDocs":787176,"percentComplete":40,"durationSeconds":276,"processingRatePerSecond":1145}
{"timestamp":"2021-03-17T19:07:20.203Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":328000,"targetDocs":787176,"percentComplete":42,"durationSeconds":281,"processingRatePerSecond":1167}
{"timestamp":"2021-03-17T19:07:25.220Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":328000,"targetDocs":787176,"percentComplete":42,"durationSeconds":286,"processingRatePerSecond":1147}
{"timestamp":"2021-03-17T19:07:30.237Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":344000,"targetDocs":787176,"percentComplete":44,"durationSeconds":291,"processingRatePerSecond":1182}
{"timestamp":"2021-03-17T19:07:35.253Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":344000,"targetDocs":787176,"percentComplete":44,"durationSeconds":296,"processingRatePerSecond":1162}
{"timestamp":"2021-03-17T19:07:40.268Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":356000,"targetDocs":787176,"percentComplete":45,"durationSeconds":301,"processingRatePerSecond":1183}
{"timestamp":"2021-03-17T19:07:45.283Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":356000,"targetDocs":787176,"percentComplete":45,"durationSeconds":306,"processingRatePerSecond":1163}
{"timestamp":"2021-03-17T19:07:50.297Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":368000,"targetDocs":787176,"percentComplete":47,"durationSeconds":311,"processingRatePerSecond":1183}
{"timestamp":"2021-03-17T19:07:55.313Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":368000,"targetDocs":787176,"percentComplete":47,"durationSeconds":316,"processingRatePerSecond":1165}
{"timestamp":"2021-03-17T19:08:00.329Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":376000,"targetDocs":787176,"percentComplete":48,"durationSeconds":321,"processingRatePerSecond":1171}
{"timestamp":"2021-03-17T19:08:05.345Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":376000,"targetDocs":787176,"percentComplete":48,"durationSeconds":326,"processingRatePerSecond":1153}
{"timestamp":"2021-03-17T19:08:10.360Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":386000,"targetDocs":787176,"percentComplete":49,"durationSeconds":331,"processingRatePerSecond":1166}
{"timestamp":"2021-03-17T19:08:15.377Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":386000,"targetDocs":787176,"percentComplete":49,"durationSeconds":336,"processingRatePerSecond":1149}
{"timestamp":"2021-03-17T19:08:20.391Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":394000,"targetDocs":787176,"percentComplete":50,"durationSeconds":341,"processingRatePerSecond":1155}
{"timestamp":"2021-03-17T19:08:25.408Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":394000,"targetDocs":787176,"percentComplete":50,"durationSeconds":346,"processingRatePerSecond":1139}
{"timestamp":"2021-03-17T19:08:30.423Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":404000,"targetDocs":787176,"percentComplete":51,"durationSeconds":351,"processingRatePerSecond":1151}
{"timestamp":"2021-03-17T19:08:35.440Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":404000,"targetDocs":787176,"percentComplete":51,"durationSeconds":356,"processingRatePerSecond":1135}
{"timestamp":"2021-03-17T19:08:40.457Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":414000,"targetDocs":787176,"percentComplete":53,"durationSeconds":361,"processingRatePerSecond":1147}
{"timestamp":"2021-03-17T19:08:45.471Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":414000,"targetDocs":787176,"percentComplete":53,"durationSeconds":366,"processingRatePerSecond":1131}
{"timestamp":"2021-03-17T19:08:50.490Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":422000,"targetDocs":787176,"percentComplete":54,"durationSeconds":371,"processingRatePerSecond":1137}
{"timestamp":"2021-03-17T19:08:55.506Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":422000,"targetDocs":787176,"percentComplete":54,"durationSeconds":376,"processingRatePerSecond":1122}
{"timestamp":"2021-03-17T19:09:00.523Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":432000,"targetDocs":787176,"percentComplete":55,"durationSeconds":381,"processingRatePerSecond":1134}
{"timestamp":"2021-03-17T19:09:05.538Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":432000,"targetDocs":787176,"percentComplete":55,"durationSeconds":386,"processingRatePerSecond":1119}
{"timestamp":"2021-03-17T19:09:10.554Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":440000,"targetDocs":787176,"percentComplete":56,"durationSeconds":391,"processingRatePerSecond":1125}
{"timestamp":"2021-03-17T19:09:15.571Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":440000,"targetDocs":787176,"percentComplete":56,"durationSeconds":396,"processingRatePerSecond":1111}
{"timestamp":"2021-03-17T19:09:20.587Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":452000,"targetDocs":787176,"percentComplete":57,"durationSeconds":401,"processingRatePerSecond":1127}
{"timestamp":"2021-03-17T19:09:25.603Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":452000,"targetDocs":787176,"percentComplete":57,"durationSeconds":406,"processingRatePerSecond":1113}
{"timestamp":"2021-03-17T19:09:30.618Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":464000,"targetDocs":787176,"percentComplete":59,"durationSeconds":411,"processingRatePerSecond":1129}
{"timestamp":"2021-03-17T19:09:35.635Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":464000,"targetDocs":787176,"percentComplete":59,"durationSeconds":416,"processingRatePerSecond":1115}
{"timestamp":"2021-03-17T19:09:40.651Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":476000,"targetDocs":787176,"percentComplete":60,"durationSeconds":421,"processingRatePerSecond":1131}
{"timestamp":"2021-03-17T19:09:45.668Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":476000,"targetDocs":787176,"percentComplete":60,"durationSeconds":426,"processingRatePerSecond":1117}
{"timestamp":"2021-03-17T19:09:50.683Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":486000,"targetDocs":787176,"percentComplete":62,"durationSeconds":431,"processingRatePerSecond":1128}
{"timestamp":"2021-03-17T19:09:55.698Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":486000,"targetDocs":787176,"percentComplete":62,"durationSeconds":436,"processingRatePerSecond":1115}
{"timestamp":"2021-03-17T19:10:00.714Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":496000,"targetDocs":787176,"percentComplete":63,"durationSeconds":441,"processingRatePerSecond":1125}
{"timestamp":"2021-03-17T19:10:05.731Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":496000,"targetDocs":787176,"percentComplete":63,"durationSeconds":446,"processingRatePerSecond":1112}
{"timestamp":"2021-03-17T19:10:10.746Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":506000,"targetDocs":787176,"percentComplete":64,"durationSeconds":451,"processingRatePerSecond":1122}
{"timestamp":"2021-03-17T19:10:15.763Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":506000,"targetDocs":787176,"percentComplete":64,"durationSeconds":456,"processingRatePerSecond":1110}
{"timestamp":"2021-03-17T19:10:20.778Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":516000,"targetDocs":787176,"percentComplete":66,"durationSeconds":461,"processingRatePerSecond":1119}
{"timestamp":"2021-03-17T19:10:25.792Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":516000,"targetDocs":787176,"percentComplete":66,"durationSeconds":466,"processingRatePerSecond":1107}
{"timestamp":"2021-03-17T19:10:30.808Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":528000,"targetDocs":787176,"percentComplete":67,"durationSeconds":472,"processingRatePerSecond":1119}
{"timestamp":"2021-03-17T19:10:35.821Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":528000,"targetDocs":787176,"percentComplete":67,"durationSeconds":477,"processingRatePerSecond":1107}
{"timestamp":"2021-03-17T19:10:40.840Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":538000,"targetDocs":787176,"percentComplete":68,"durationSeconds":482,"processingRatePerSecond":1116}
{"timestamp":"2021-03-17T19:10:45.854Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":538000,"targetDocs":787176,"percentComplete":68,"durationSeconds":487,"processingRatePerSecond":1105}
{"timestamp":"2021-03-17T19:10:50.870Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":544000,"targetDocs":787176,"percentComplete":69,"durationSeconds":492,"processingRatePerSecond":1106}
{"timestamp":"2021-03-17T19:10:55.887Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":544000,"targetDocs":787176,"percentComplete":69,"durationSeconds":497,"processingRatePerSecond":1095}
{"timestamp":"2021-03-17T19:11:00.901Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":548000,"targetDocs":787176,"percentComplete":70,"durationSeconds":502,"processingRatePerSecond":1092}
{"timestamp":"2021-03-17T19:11:05.916Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":548000,"targetDocs":787176,"percentComplete":70,"durationSeconds":507,"processingRatePerSecond":1081}
{"timestamp":"2021-03-17T19:11:10.930Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":552000,"targetDocs":787176,"percentComplete":70,"durationSeconds":512,"processingRatePerSecond":1078}
{"timestamp":"2021-03-17T19:11:15.946Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":552000,"targetDocs":787176,"percentComplete":70,"durationSeconds":517,"processingRatePerSecond":1068}
{"timestamp":"2021-03-17T19:11:20.962Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":554000,"targetDocs":787176,"percentComplete":70,"durationSeconds":522,"processingRatePerSecond":1061}
{"timestamp":"2021-03-17T19:11:25.978Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":554000,"targetDocs":787176,"percentComplete":70,"durationSeconds":527,"processingRatePerSecond":1051}
{"timestamp":"2021-03-17T19:11:30.993Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":558000,"targetDocs":787176,"percentComplete":71,"durationSeconds":532,"processingRatePerSecond":1049}
{"timestamp":"2021-03-17T19:11:36.012Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":558000,"targetDocs":787176,"percentComplete":71,"durationSeconds":537,"processingRatePerSecond":1039}
{"timestamp":"2021-03-17T19:11:41.038Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":562000,"targetDocs":787176,"percentComplete":71,"durationSeconds":542,"processingRatePerSecond":1037}
{"timestamp":"2021-03-17T19:11:46.055Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":562000,"targetDocs":787176,"percentComplete":71,"durationSeconds":547,"processingRatePerSecond":1027}
{"timestamp":"2021-03-17T19:11:51.071Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":566000,"targetDocs":787176,"percentComplete":72,"durationSeconds":552,"processingRatePerSecond":1025}
{"timestamp":"2021-03-17T19:11:56.087Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":566000,"targetDocs":787176,"percentComplete":72,"durationSeconds":557,"processingRatePerSecond":1016}
{"timestamp":"2021-03-17T19:12:01.104Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":570000,"targetDocs":787176,"percentComplete":72,"durationSeconds":562,"processingRatePerSecond":1014}
{"timestamp":"2021-03-17T19:12:06.123Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":570000,"targetDocs":787176,"percentComplete":72,"durationSeconds":567,"processingRatePerSecond":1005}
{"timestamp":"2021-03-17T19:12:11.143Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":576000,"targetDocs":787176,"percentComplete":73,"durationSeconds":572,"processingRatePerSecond":1007}
{"timestamp":"2021-03-17T19:12:16.158Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":576000,"targetDocs":787176,"percentComplete":73,"durationSeconds":577,"processingRatePerSecond":998}
{"timestamp":"2021-03-17T19:12:21.174Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":580000,"targetDocs":787176,"percentComplete":74,"durationSeconds":582,"processingRatePerSecond":997}
{"timestamp":"2021-03-17T19:12:26.189Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":580000,"targetDocs":787176,"percentComplete":74,"durationSeconds":587,"processingRatePerSecond":988}
{"timestamp":"2021-03-17T19:12:31.205Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":586000,"targetDocs":787176,"percentComplete":74,"durationSeconds":592,"processingRatePerSecond":990}
{"timestamp":"2021-03-17T19:12:36.220Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":586000,"targetDocs":787176,"percentComplete":74,"durationSeconds":597,"processingRatePerSecond":982}
{"timestamp":"2021-03-17T19:12:41.236Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":590000,"targetDocs":787176,"percentComplete":75,"durationSeconds":602,"processingRatePerSecond":980}
{"timestamp":"2021-03-17T19:12:46.252Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":590000,"targetDocs":787176,"percentComplete":75,"durationSeconds":607,"processingRatePerSecond":972}
{"timestamp":"2021-03-17T19:12:51.266Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":594000,"targetDocs":787176,"percentComplete":75,"durationSeconds":612,"processingRatePerSecond":971}
{"timestamp":"2021-03-17T19:12:56.282Z","level":"info","module":"solr","message":"reindex still running","sourceCollection":"at-uk-002","destinationCollection":"karl-test","configName":"at-uk-002-509","rows":2000,"processedDocs":594000,"targetDocs":787176,"percentComplete":75,"durationSeconds":617,"processingRatePerSecond":963}
```
Unless expressly stated otherwise in this email, this e-mail is sent on behalf of Auto Trader Limited Registered Office: 1 Tony Wilson Place, Manchester, Lancashire, M15 4FN (Registered in England No. 03909628). Auto Trader Limited is part of the Auto Trader Group Plc group. This email and any files transmitted with it are confidential and may be legally privileged, and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender. This email message has been swept for the presence of computer viruses.
Unless expressly stated otherwise in this email, this e-mail is sent on behalf of Auto Trader Limited Registered Office: 1 Tony Wilson Place, Manchester, Lancashire, M15 4FN (Registered in England No. 03909628). Auto Trader Limited is part of the Auto Trader Group Plc group. This email and any files transmitted with it are confidential and may be legally privileged, and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender. This email message has been swept for the presence of computer viruses.