You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "raj.yadav" <ra...@cse.ism.ac.in> on 2020/11/04 06:46:41 UTC
Commits (with openSearcher = true) are too slow in solr 8
Hi everyone,
We have two parallel system one is solr 8.5.2 and other one is solr 5.4
In solr_5.4 commit time with opensearcher true is 10 to 12 minutes while in
solr_8 it's around 25 minutes.
This is our current caching policy of solr_8
<filterCache class="solr.CaffeineCache"
size="32768"
initialSize="6000"
autowarmCount="6000"/>
<queryResultCache class="solr.CaffeineCache"
size="25600"
initialSize="6000"
autowarmCount="0"/>
<documentCache class="solr.CaffeineCache"
size="32768"
initialSize="6144"
autowarmCount="0"/>
In solr 5, we are using FastLRUCache (instead of CaffeineCache) and other
parameters are same.
While debugging this we came across this page.
https://cwiki.apache.org/confluence/display/SOLR/SolrPerformanceProblems#SolrPerformanceProblems-Slowcommits
Here one of the reasons for slow commit is mentioned as:
*/`Heap size issues. Problems from the heap being too big will tend to be
infrequent, while problems from the heap being too small will tend to happen
consistently.`/*
Can anyone please help me understand the above point?
System config:
disk size: 250 GB
cpu: (8 vcpus, 64 GiB memory)
Index size: 11 GB
JVM heap size: 30 GB
--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Commits (with openSearcher = true) are too slow in solr 8
Posted by "raj.yadav" <ra...@cse.ism.ac.in>.
Hi everyone,
As per suggestions in previous post (by Erick and Shawn) we did following
changes.
OLD
<filterCache class="solr.CaffeineCache"
size="32768"
initialSize="6000"
autowarmCount="6000"/>
<queryResultCache class="solr.CaffeineCache"
size="25600"
initialSize="6000"
autowarmCount="0"/>
<documentCache class="solr.CaffeineCache"
size="32768"
initialSize="6144"
autowarmCount="0"/>
NEW
<filterCache class="solr.CaffeineCache"
size="8192"
initialSize="512"
autowarmCount="512"/>
<queryResultCache class="solr.CaffeineCache"
size="8192"
initialSize="3000"
autowarmCount="0"/>
<documentCache class="solr.CaffeineCache"
size="8192"
initialSize="3072"
autowarmCount="0"/>
*Reduced JVM heap size from 30GB to 26GB*
GC setting:
GC_TUNE=" \
-XX:+UseG1GC \
-XX:+PerfDisableSharedMem \
-XX:+ParallelRefProcEnabled \
-XX:G1HeapRegionSize=8m \
-XX:MaxGCPauseMillis=150 \
-XX:InitiatingHeapOccupancyPercent=60 \
-XX:+UseLargePages \
-XX:+AggressiveOpts \
"
Solr Collection details: (running in solrCloud mode)
It has 6 shards, and each shard has only one replica (which is also a
leader) and replica type is NRT
Each shard Index size: 11 GB
avg size/doc: 1.0Kb
We are running indexing on this collection:
*Indexing rate: 2.4 million per hour*
*The query rate is zero. Still commit with opensearcher=true is taking 25 to
28 minutes.*
Is this because of heavy indexing? Also, with an increase in the number of
documents in collection commit time is increasing.
This is not our production system. In the prod system generally, our
indexing rate is 5k/hour.
Is it expected to have such high commit time with the above indexing rate?
--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Commits (with openSearcher = true) are too slow in solr 8
Posted by "raj.yadav" <ra...@cse.ism.ac.in>.
Thanks, Shawn and Erick.
We are step by step trying out the changes suggested in your post.
Will get back once we have some numbers.
--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Commits (with openSearcher = true) are too slow in solr 8
Posted by "raj.yadav" <ra...@cse.ism.ac.in>.
matthew sporleder wrote
> I would stick to soft commits and schedule hard-commits as
> spaced-out-as-possible in regular maintenance windows until you can
> find the culprit of the timeout.
>
> This way you will have very focused windows for intense monitoring
> during the hard-commit runs.
*Little correction:*
In my last post, I had mentioned that softCommit is working fine and there
no delay or error message.
Here is what happening:
1. Hard commit with openSearcher=true
curl
"http://<solr_ip>:solr_port/solr/my_collection/update?openSearcher=true&commit=true&wt=json"
All the cores started processing commit except , the one hosted `<solr_ip>`.
Also we are getting timeout error on this.
2. softCommit
curl
"http://<solr_ip>:solr_port/solr/my_collection/update?softCommit=true&wt=json"
Same as 1.
3.Hard commit with openSearcher=false
curl
"http://<solr_ip>:solr_port/solr/my_collection/update?openSearcher=false&commit=true&wt=json"
All the cores started processing commit immediately and there is no error.
Solr commands used to set up system
Solr start command
#/var/solr-8.5.2/bin/solr start -c -p solr_port -z
zk_host1:zk_port,zk_host1:zk_port,zk_host1:zk_port -s
/var/node_my_collection_1/solr-8.5.2/server/solr -h <solr_ip> -m 26g
-DzkClientTimeout=30000 -force
Creat Collection
1.upload config to zookeper
#var/solr-8.5.2/server/scripts/cloud-scripts/./zkcli.sh -z
zk_host1:zk_port,zk_host1:zk_port,zk_host1:zk_port -cmd upconfig -confname
my_collection -confdir /<local_config_path>
2. Cretaed collection with 3 shards (shard1,shard2,shard3),
#curl
"http://<solr_ip>:solr_port/solr/admin/collections?action=CREATE&name=my_collection&numShards=3&replicationFactor=1&maxShardsPerNode=1&collection.configName=my_collection&createNodeSet=solr_node1:solr_port,solr_node2:solr_port,solr_node3:solr_port"
3. Used SPLITSHARD command to split each shards into two half
(shard1_1,shard1_0,shard2_0,...)
e.g
#curl
"http://<solr_ip>:solr_port/solr/admin/collections?action=SPLITSHARD&collection=my_collection&shard=shard1
4. Used DELETESHARD command to delete old shatds (shard1,shard2,shard3).
e.g
#curl
"http://<solr_ip>:solr_port/solr/admin/collections?action=DELETESHARD&collection=my_collection&shard=shard1
--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Commits (with openSearcher = true) are too slow in solr 8
Posted by matthew sporleder <ms...@gmail.com>.
I would stick to soft commits and schedule hard-commits as
spaced-out-as-possible in regular maintenance windows until you can
find the culprit of the timeout.
This way you will have very focused windows for intense monitoring
during the hard-commit runs.
On Mon, Dec 7, 2020 at 9:24 AM raj.yadav <ra...@cse.ism.ac.in> wrote:
>
> Hi Folks,
>
> Do let me know if any more information required to debug this.
>
>
> Regards,
> Raj
>
>
>
> --
> Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Commits (with openSearcher = true) are too slow in solr 8
Posted by "raj.yadav" <ra...@cse.ism.ac.in>.
Hi Folks,
Do let me know if any more information required to debug this.
Regards,
Raj
--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Commits (with openSearcher = true) are too slow in solr 8
Posted by "raj.yadav" <ra...@cse.ism.ac.in>.
matthew sporleder wrote
> Is zookeeper on the solr hosts or on its own? Have you tried
> opensearcher=false (soft commit?)
1. we are using zookeeper in ensemble mode. Its hosted on 3 seperate node.
2. Soft commit (opensearcher=false) is working fine. All the shards are
getting commit request immediately and its got processed within second.
--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Commits (with openSearcher = true) are too slow in solr 8
Posted by matthew sporleder <ms...@gmail.com>.
Is zookeeper on the solr hosts or on its own? Have you tried
opensearcher=false (soft commit?)
On Sun, Dec 6, 2020 at 6:19 PM raj.yadav <ra...@cse.ism.ac.in> wrote:
>
> Hi Everyone,
>
>
> matthew sporleder wrote
> > Are you stuck in iowait during that commit?
>
> During commit operation, there is no iowait.
> Infact most of the time cpu utilization percentage is very low.
>
> /*As I mentioned in my previous post that we are getting `SolrCmdDistributor
> org.apache.solr.client.solrj.SolrServerException: Timeout occured while
> waiting response from server` and `DistributedZkUpdateProcessor` ERROR on
> one of the shards. And this error is always occurring on the shard that is
> used (in culr command) to issue commit. (See below example for better
> understanding)*/
>
> Here is shard and corresponding node details:
> shard1_0=>solr_199
> shard1_1=>solr_200
> shard2_0=> solr_254
> shard2_1=> solr_132
> shard3_0=>solr_133
> shard3_1=>solr_198
>
> We are using the following command to issue commit:
> /curl
> "http://solr_node:8389/solr/my_collection/update?openSearcher=true&commit=true&wt=json"/
>
> For example, in the above command, if we replace solr_node with solr_254,
> then it's throwing SolrCmdDistributor and DistributedZkUpdateProcessor
> errors on shard2_0. Similarly, if we replace solr_node with solr_200 its
> throws errors on shard1_1.
>
> *I'm not able to figure out why this is happening. Is there any connection
> timeout setting that is affecting this? Is there any limit that, at a time
> only N number of shards can run commit ops simultaneously or is it some
> network related issue?*
>
>
> For a better understanding of what's happening in SOLR logs. I will
> demonstrate here one commit operation.
>
> I used the below command to issue commit at `2020-12-06 18:37:40` (approx)
> curl
> "http://solr_200:8389/solr/my_collection/update?openSearcher=true&commit=true&wt=json"
>
>
> /*shard2_0 (node: solr_254) Logs:*/
>
>
> *Commit is received at `2020-12-06 18:37:47` and got over by `2020-12-06
> 18:37:47` since there were no changes to commit. And CPU utilization during
> the whole period is around 2%.*
>
>
> 2020-12-06 18:37:47.023 INFO (qtp2034610694-31355) [c:my_collection
> s:shard2_0 r:core_node13 x:my_collection_shard2_0_replica_n11]
> o.a.s.u.DirectUpdateHandler2 start
> commit{_version_=1685355093842460672,optimize=false,ope
> nSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
> 2020-12-06 18:37:47.023 INFO No uncommitted changes. Skipping IW.commit.
> 2020-12-06 18:37:47.023 INFO end_commit_flush
> 2020-12-06 18:37:47.023 INFO (qtp2034610694-31355) [c:my_collection
> s:shard2_0 r:core_node13 x:my_collection_shard2_0_replica_n11]
> o.a.s.u.p.LogUpdateProcessorFactory [my_collection_shard2_0_replica_n11]
> webapp=/solr path=/update
>
> params={update.distrib=TOLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://solr_200:8389/solr/my_collection_shard1_1_replica_n19/&commit_end_point=leaders&wt=javabi
> n&version=2&expungeDeletes=false}{commit=} 0 3
>
> /*shard2_1 (node: solr_132) Logs:*/
>
> *Commit is received at `2020-12-06 18:37:47` and got over by `2020-12-06
> 18:50:46` in between there were some external file reloading operations (our
> solr-5.4.2 system is also taking similar time to reload external files so
> right now this is not a major concern for us)
> CPU utilization before commit (i.e `2020-12-06 18:37:47` timestamp) is 2%
> and between commit ops (i.e from `2020-12-06 18:37:47` to `2020-12-06
> 18:50:46` timestamp) is 14% and after commit operation is done it agains
> fall back to 2%*
>
>
> 2020-12-06 18:37:47.024 INFO (qtp2034610694-30058) [c:my_collection
> s:shard2_1 r:core_node22 x:my_collection_shard2_1_replica_n21]
> o.a.s.u.DirectUpdateHandler2 start
> commit{_version_=1685355093844557824,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
>
> 2020-12-06 18:50:46.218 INFO (qtp2034610694-30058) [c:my_collection
> s:shard2_1 r:core_node22 x:my_collection_shard2_1_replica_n21]
> o.a.s.u.p.LogUpdateProcessorFactory [my_collection_shard2_1_replica_n21]
> webapp=/solr path=/update
> params={update.distrib=TOLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://solr_200:8389/solr/my_collection_shard1_1_replica_n19/&commit_end_point=leaders&wt=javabin&version=2&expungeDeletes=false}{commit=}
> 0 779196
>
>
> /*shard3_0 (node: solr_133) logs*/
>
> Same as shard2_1, commit received at `2020-12-06 18:37:47` and got over by
> `2020-12-06 18:49:24`.
> CPU utilization pattern is same is shard2_1.
>
> /*shard3_1 (node: solr_198) logs.*/
>
> Same as shard2_1, commit received at `2020-12-06 18:37:47` and got over by
> `2020-12-06 18:53:57`.
> CPU utilization pattern is same is shard2_1.
>
> /*shard1_0 (node: solr_199) logs.*/
>
> Same as shard2_1, commit received at `2020-12-06 18:37:47` and got over by
> `2020-12-06 18:54:51`.
> CPU utilization pattern is same is shard2_1.
>
> /*shard1_1 (node: solr_200) logs.*/
>
> /This is the same solr_node which we used in curl command to issue commit.
> As expected we got SolrCmdDistributor and DistributedZkUpdateProcessor error
> on it./
>
> /Till `2020-12-06 18:46:50` timestamp there is no `start commit` request
> received. Also CPU utilization is 2% till this time./
> /*Received follwing error at `2020-12-06 18:47:47` timestamp*/
>
> 2020-12-06 18:47:47.013 ERROR
> (updateExecutor-5-thread-6-processing-n:solr_200:8389_solr
> x:my_collection_shard1_1_replica_n19 c:my_collection s:shard1_1
> r:core_node20) [c:my_collection s:shard1_1 r:core_node20
> x:my_collection_shard1_1_replica_n19] o.a.s.u.SolrCmdDistributor
> org.apache.solr.client.solrj.SolrServerException: Timeout occured while
> waiting response from server at:
> http://solr_198:8389/solr/my_collection_shard3_1_replica_n23/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsolr_200%3A8389%2Fsolr%2Fmy_collection_shard1_1_replica_n19%2F
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:407)
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:753)
> at
> org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:369)
> at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290)
> at
> org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:344)
> at
> org.apache.solr.update.SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:333)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.util.concurrent.TimeoutException
> at
> org.eclipse.jetty.client.util.InputStreamResponseListener.get(InputStreamResponseListener.java:216)
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:398)
> ... 13 more
>
> 2020-12-06 18:47:47.018 ERROR
> (updateExecutor-5-thread-2-processing-n:solr_200:8389_solr
> x:my_collection_shard1_1_replica_n19 c:my_collection s:shard1_1
> r:core_node20) [c:my_collection s:shard1_1 r:core_node20
> x:my_collection_shard1_1_replica_n19] o.a.s.u.SolrCmdDistributor
> org.apache.solr.client.solrj.SolrServerException: Timeout occured while
> waiting response from server at:
> http://solr_199:8389/solr/my_collection_shard1_0_replica_n7/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsolr_200%3A8389%2Fsolr%2Fmy_collection_shard1_1_replica_n19%2F
> ........
> *(REST ERROR MESSAGE SAME AS ABOVE)*
>
>
>
> 2020-12-06 18:47:47.018 ERROR
> (updateExecutor-5-thread-4-processing-n:solr_200:8389_solr
> x:my_collection_shard1_1_replica_n19 c:my_collection s:shard1_1
> r:core_node20) [c:my_collection s:shard1_1 r:core_node20
> x:my_collection_shard1_1_replica_n19] o.a.s.u.SolrCmdDistributor
> org.apache.solr.client.solrj.SolrServerException: Timeout occured while
> waiting response from server at:
> http://solr_132:8389/solr/my_collection_shard2_1_replica_n21/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsolr_200%3A8389%2Fsolr%2Fmy_collection_shard1_1_replica_n19%2F
> ........
>
>
> 2020-12-06 18:47:47.018 ERROR
> (updateExecutor-5-thread-5-processing-n:solr_200:8389_solr
> x:my_collection_shard1_1_replica_n19 c:my_collection s:shard1_1
> r:core_node20) [c:my_collection s:shard1_1 r:core_node20
> x:my_collection_shard1_1_replica_n19] o.a.s.u.SolrCmdDistributor
> org.apache.solr.client.solrj.SolrServerException: Timeout occured while
> waiting response from server at:
> http://solr_133:8389/solr/my_collection_shard3_0_replica_n15/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsolr_200%3A8389%2Fsolr%2Fmy_collection_shard1_1_replica_n19%2F
> ........
>
> 2020-12-06 18:47:47.020 INFO (qtp1267082248-29872) [c:my_collection
> s:shard1_1 r:core_node20 x:my_collection_shard1_1_replica_n19]
> o.a.s.u.SolrCmdDistributor SolrCmdDistributor found 4 errors
>
>
> /*Then at `2020-12-06 18:47:47` received `start commit` request. Also by
> this time CPU utilization is moved up to 14%. */
>
>
> 2020-12-06 18:47:47.020 INFO (qtp1267082248-29872) [c:my_collection
> s:shard1_1 r:core_node20 x:my_collection_shard1_1_replica_n19]
> o.a.s.u.DirectUpdateHandler2 start
> commit{_version_=1685355722985963520,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
>
>
> *THEN IN BW THERE WERE SOME EXTERNAL FILE RELOADING OPERATION. CPU
> utilization is still around 14%.*
>
> *Then at `2020-12-06 19:06:09` timestamp got following warning.*
>
> 2020-12-06 19:06:09.672 WARN (qtp1267082248-29872) [c:my_collection
> s:shard1_1 r:core_node20 x:my_collection_shard1_1_replica_n19]
> o.a.s.u.p.DistributedZkUpdateProcessor Error sending update to
> http://solr_198:8389/solr =>
> org.apache.solr.client.solrj.SolrServerException: Timeout occured while
> waiting response from server at:
> http://solr_198:8389/solr/my_collection_shard3_1_replica_n23/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsolr_200%3A8389%2Fsolr%2Fmy_collection_shard1_1_replica_n19%2F
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:407)
> org.apache.solr.client.solrj.SolrServerException: Timeout occured while
> waiting response from server at:
> http://solr_198:8389/solr/my_collection_shard3_1_replica_n23/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsolr_200%3A8389%2Fsolr%2Fmy_collection_shard1_1_replica_n19%2F
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:407)
> ~[?:?]
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:753)
> ~[?:?]
> at
> org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:369)
> ~[?:?]
> at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290)
> ~[?:?]
> at
> org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:344)
> ~[?:?]
> at
> org.apache.solr.update.SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:333)
> ~[?:?]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_262]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[?:1.8.0_262]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_262]
> at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
> ~[metrics-core-4.1.2.jar:4.1.2]
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
> ~[?:?]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_262]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_262]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_262]
> Caused by: java.util.concurrent.TimeoutException
> at
> org.eclipse.jetty.client.util.InputStreamResponseListener.get(InputStreamResponseListener.java:216)
> ~[?:?]
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:398)
> ~[?:?]
> ... 13 more
>
>
> 2020-12-06 19:06:09.673 WARN (qtp1267082248-29872) [c:my_collection
> s:shard1_1 r:core_node20 x:my_collection_shard1_1_replica_n19]
> o.a.s.u.p.DistributedZkUpdateProcessor Error sending update to
> http://solr_199:8389/solr =>
> org.apache.solr.client.solrj.SolrServerException: Timeout occured while
> waiting response from server at: http://solr_199:8389/solr/........
>
> 2020-12-06 19:06:09.673 WARN (qtp1267082248-29872) [c:my_collection
> s:shard1_1 r:core_node20 x:my_collection_shard1_1_replica_n19]
> o.a.s.u.p.DistributedZkUpdateProcessor Error sending update to
> http://solr_132:8389/solr =>
> org.apache.solr.client.solrj.SolrServerException: Timeout occured while
> waiting response from server at: http://solr_132:8389/solr/........
>
>
> 2020-12-06 19:06:09.673 WARN (qtp1267082248-29872) [c:my_collection
> s:shard1_1 r:core_node20 x:my_collection_shard1_1_replica_n19]
> o.a.s.u.p.DistributedZkUpdateProcessor Error sending update to
> http://solr_133:8389/solr =>
> org.apache.solr.client.solrj.SolrServerException: Timeout occured while
> waiting response from server at: http://solr_133:8389/solr/........
>
> /*
> Then at `2020-12-06 19:06:09` timestamp we got following. Which I'm assuming
> indicates that commit got over, since after this CPU utilization fall back
> again to 2%. And We also got curl reponse by this timestamp with status=0. *
> /
>
>
> 2020-12-06 19:06:09.673 INFO (qtp1267082248-29872) [c:my_collection
> s:shard1_1 r:core_node20 x:my_collection_shard1_1_replica_n19]
> o.a.s.u.p.LogUpdateProcessorFactory [my_collection_shard1_1_replica_n19]
> webapp=/solr path=/update
> params={openSearcher=true&commit=true&wt=json}{commit=} 0 1702717
>
>
> In short to summarize:
> Commit was issued at `2020-12-06 18:37:40` (approx) timestamp.
> Till `2020-12-06 18:46:50` timestamp there was no `start commit` received
> and CPU was at 2% till it throws `SolrCmdDistributor` error at `2020-12-06
> 18:47:47` timestamp. Then at around the same timestamp `2020-12-06 18:47:47`
> received the `start commit` request. And by this time CPU utilization is
> moved up to 14%.
> Then at `2020-12-06 19:06:09` timestamp got `DistributedZkUpdateProcessor`
> error message and then at `2020-12-06 19:06:09.673` timestamp got some
> commit related message (which I'm assuming its indicates end of commit) and
> CPU is again fall back to 2%.
>
>
> Due to these timeout errors overall commit time is getting double. Can
> someone guide me to resolve this issue.
>
>
>
>
>
>
>
>
>
>
> --
> Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Commits (with openSearcher = true) are too slow in solr 8
Posted by "raj.yadav" <ra...@cse.ism.ac.in>.
Hi All,
I tried debugging but unable to find any solution. Do let me know in case
details/logs shared by me are not suffiecient/clear.
Regards,
Raj
--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Commits (with openSearcher = true) are too slow in solr 8
Posted by "raj.yadav" <ra...@cse.ism.ac.in>.
Hi Everyone,
matthew sporleder wrote
> Are you stuck in iowait during that commit?
During commit operation, there is no iowait.
Infact most of the time cpu utilization percentage is very low.
/*As I mentioned in my previous post that we are getting `SolrCmdDistributor
org.apache.solr.client.solrj.SolrServerException: Timeout occured while
waiting response from server` and `DistributedZkUpdateProcessor` ERROR on
one of the shards. And this error is always occurring on the shard that is
used (in culr command) to issue commit. (See below example for better
understanding)*/
Here is shard and corresponding node details:
shard1_0=>solr_199
shard1_1=>solr_200
shard2_0=> solr_254
shard2_1=> solr_132
shard3_0=>solr_133
shard3_1=>solr_198
We are using the following command to issue commit:
/curl
"http://solr_node:8389/solr/my_collection/update?openSearcher=true&commit=true&wt=json"/
For example, in the above command, if we replace solr_node with solr_254,
then it's throwing SolrCmdDistributor and DistributedZkUpdateProcessor
errors on shard2_0. Similarly, if we replace solr_node with solr_200 its
throws errors on shard1_1.
*I'm not able to figure out why this is happening. Is there any connection
timeout setting that is affecting this? Is there any limit that, at a time
only N number of shards can run commit ops simultaneously or is it some
network related issue?*
For a better understanding of what's happening in SOLR logs. I will
demonstrate here one commit operation.
I used the below command to issue commit at `2020-12-06 18:37:40` (approx)
curl
"http://solr_200:8389/solr/my_collection/update?openSearcher=true&commit=true&wt=json"
/*shard2_0 (node: solr_254) Logs:*/
*Commit is received at `2020-12-06 18:37:47` and got over by `2020-12-06
18:37:47` since there were no changes to commit. And CPU utilization during
the whole period is around 2%.*
2020-12-06 18:37:47.023 INFO (qtp2034610694-31355) [c:my_collection
s:shard2_0 r:core_node13 x:my_collection_shard2_0_replica_n11]
o.a.s.u.DirectUpdateHandler2 start
commit{_version_=1685355093842460672,optimize=false,ope
nSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
2020-12-06 18:37:47.023 INFO No uncommitted changes. Skipping IW.commit.
2020-12-06 18:37:47.023 INFO end_commit_flush
2020-12-06 18:37:47.023 INFO (qtp2034610694-31355) [c:my_collection
s:shard2_0 r:core_node13 x:my_collection_shard2_0_replica_n11]
o.a.s.u.p.LogUpdateProcessorFactory [my_collection_shard2_0_replica_n11]
webapp=/solr path=/update
params={update.distrib=TOLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://solr_200:8389/solr/my_collection_shard1_1_replica_n19/&commit_end_point=leaders&wt=javabi
n&version=2&expungeDeletes=false}{commit=} 0 3
/*shard2_1 (node: solr_132) Logs:*/
*Commit is received at `2020-12-06 18:37:47` and got over by `2020-12-06
18:50:46` in between there were some external file reloading operations (our
solr-5.4.2 system is also taking similar time to reload external files so
right now this is not a major concern for us)
CPU utilization before commit (i.e `2020-12-06 18:37:47` timestamp) is 2%
and between commit ops (i.e from `2020-12-06 18:37:47` to `2020-12-06
18:50:46` timestamp) is 14% and after commit operation is done it agains
fall back to 2%*
2020-12-06 18:37:47.024 INFO (qtp2034610694-30058) [c:my_collection
s:shard2_1 r:core_node22 x:my_collection_shard2_1_replica_n21]
o.a.s.u.DirectUpdateHandler2 start
commit{_version_=1685355093844557824,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
2020-12-06 18:50:46.218 INFO (qtp2034610694-30058) [c:my_collection
s:shard2_1 r:core_node22 x:my_collection_shard2_1_replica_n21]
o.a.s.u.p.LogUpdateProcessorFactory [my_collection_shard2_1_replica_n21]
webapp=/solr path=/update
params={update.distrib=TOLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://solr_200:8389/solr/my_collection_shard1_1_replica_n19/&commit_end_point=leaders&wt=javabin&version=2&expungeDeletes=false}{commit=}
0 779196
/*shard3_0 (node: solr_133) logs*/
Same as shard2_1, commit received at `2020-12-06 18:37:47` and got over by
`2020-12-06 18:49:24`.
CPU utilization pattern is same is shard2_1.
/*shard3_1 (node: solr_198) logs.*/
Same as shard2_1, commit received at `2020-12-06 18:37:47` and got over by
`2020-12-06 18:53:57`.
CPU utilization pattern is same is shard2_1.
/*shard1_0 (node: solr_199) logs.*/
Same as shard2_1, commit received at `2020-12-06 18:37:47` and got over by
`2020-12-06 18:54:51`.
CPU utilization pattern is same is shard2_1.
/*shard1_1 (node: solr_200) logs.*/
/This is the same solr_node which we used in curl command to issue commit.
As expected we got SolrCmdDistributor and DistributedZkUpdateProcessor error
on it./
/Till `2020-12-06 18:46:50` timestamp there is no `start commit` request
received. Also CPU utilization is 2% till this time./
/*Received follwing error at `2020-12-06 18:47:47` timestamp*/
2020-12-06 18:47:47.013 ERROR
(updateExecutor-5-thread-6-processing-n:solr_200:8389_solr
x:my_collection_shard1_1_replica_n19 c:my_collection s:shard1_1
r:core_node20) [c:my_collection s:shard1_1 r:core_node20
x:my_collection_shard1_1_replica_n19] o.a.s.u.SolrCmdDistributor
org.apache.solr.client.solrj.SolrServerException: Timeout occured while
waiting response from server at:
http://solr_198:8389/solr/my_collection_shard3_1_replica_n23/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsolr_200%3A8389%2Fsolr%2Fmy_collection_shard1_1_replica_n19%2F
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:407)
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:753)
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:369)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290)
at
org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:344)
at
org.apache.solr.update.SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:333)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.TimeoutException
at
org.eclipse.jetty.client.util.InputStreamResponseListener.get(InputStreamResponseListener.java:216)
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:398)
... 13 more
2020-12-06 18:47:47.018 ERROR
(updateExecutor-5-thread-2-processing-n:solr_200:8389_solr
x:my_collection_shard1_1_replica_n19 c:my_collection s:shard1_1
r:core_node20) [c:my_collection s:shard1_1 r:core_node20
x:my_collection_shard1_1_replica_n19] o.a.s.u.SolrCmdDistributor
org.apache.solr.client.solrj.SolrServerException: Timeout occured while
waiting response from server at:
http://solr_199:8389/solr/my_collection_shard1_0_replica_n7/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsolr_200%3A8389%2Fsolr%2Fmy_collection_shard1_1_replica_n19%2F
........
*(REST ERROR MESSAGE SAME AS ABOVE)*
2020-12-06 18:47:47.018 ERROR
(updateExecutor-5-thread-4-processing-n:solr_200:8389_solr
x:my_collection_shard1_1_replica_n19 c:my_collection s:shard1_1
r:core_node20) [c:my_collection s:shard1_1 r:core_node20
x:my_collection_shard1_1_replica_n19] o.a.s.u.SolrCmdDistributor
org.apache.solr.client.solrj.SolrServerException: Timeout occured while
waiting response from server at:
http://solr_132:8389/solr/my_collection_shard2_1_replica_n21/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsolr_200%3A8389%2Fsolr%2Fmy_collection_shard1_1_replica_n19%2F
........
2020-12-06 18:47:47.018 ERROR
(updateExecutor-5-thread-5-processing-n:solr_200:8389_solr
x:my_collection_shard1_1_replica_n19 c:my_collection s:shard1_1
r:core_node20) [c:my_collection s:shard1_1 r:core_node20
x:my_collection_shard1_1_replica_n19] o.a.s.u.SolrCmdDistributor
org.apache.solr.client.solrj.SolrServerException: Timeout occured while
waiting response from server at:
http://solr_133:8389/solr/my_collection_shard3_0_replica_n15/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsolr_200%3A8389%2Fsolr%2Fmy_collection_shard1_1_replica_n19%2F
........
2020-12-06 18:47:47.020 INFO (qtp1267082248-29872) [c:my_collection
s:shard1_1 r:core_node20 x:my_collection_shard1_1_replica_n19]
o.a.s.u.SolrCmdDistributor SolrCmdDistributor found 4 errors
/*Then at `2020-12-06 18:47:47` received `start commit` request. Also by
this time CPU utilization is moved up to 14%. */
2020-12-06 18:47:47.020 INFO (qtp1267082248-29872) [c:my_collection
s:shard1_1 r:core_node20 x:my_collection_shard1_1_replica_n19]
o.a.s.u.DirectUpdateHandler2 start
commit{_version_=1685355722985963520,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
*THEN IN BW THERE WERE SOME EXTERNAL FILE RELOADING OPERATION. CPU
utilization is still around 14%.*
*Then at `2020-12-06 19:06:09` timestamp got following warning.*
2020-12-06 19:06:09.672 WARN (qtp1267082248-29872) [c:my_collection
s:shard1_1 r:core_node20 x:my_collection_shard1_1_replica_n19]
o.a.s.u.p.DistributedZkUpdateProcessor Error sending update to
http://solr_198:8389/solr =>
org.apache.solr.client.solrj.SolrServerException: Timeout occured while
waiting response from server at:
http://solr_198:8389/solr/my_collection_shard3_1_replica_n23/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsolr_200%3A8389%2Fsolr%2Fmy_collection_shard1_1_replica_n19%2F
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:407)
org.apache.solr.client.solrj.SolrServerException: Timeout occured while
waiting response from server at:
http://solr_198:8389/solr/my_collection_shard3_1_replica_n23/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsolr_200%3A8389%2Fsolr%2Fmy_collection_shard1_1_replica_n19%2F
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:407)
~[?:?]
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:753)
~[?:?]
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:369)
~[?:?]
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290)
~[?:?]
at
org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:344)
~[?:?]
at
org.apache.solr.update.SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:333)
~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_262]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[?:1.8.0_262]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_262]
at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
~[metrics-core-4.1.2.jar:4.1.2]
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
~[?:1.8.0_262]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
~[?:1.8.0_262]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_262]
Caused by: java.util.concurrent.TimeoutException
at
org.eclipse.jetty.client.util.InputStreamResponseListener.get(InputStreamResponseListener.java:216)
~[?:?]
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:398)
~[?:?]
... 13 more
2020-12-06 19:06:09.673 WARN (qtp1267082248-29872) [c:my_collection
s:shard1_1 r:core_node20 x:my_collection_shard1_1_replica_n19]
o.a.s.u.p.DistributedZkUpdateProcessor Error sending update to
http://solr_199:8389/solr =>
org.apache.solr.client.solrj.SolrServerException: Timeout occured while
waiting response from server at: http://solr_199:8389/solr/........
2020-12-06 19:06:09.673 WARN (qtp1267082248-29872) [c:my_collection
s:shard1_1 r:core_node20 x:my_collection_shard1_1_replica_n19]
o.a.s.u.p.DistributedZkUpdateProcessor Error sending update to
http://solr_132:8389/solr =>
org.apache.solr.client.solrj.SolrServerException: Timeout occured while
waiting response from server at: http://solr_132:8389/solr/........
2020-12-06 19:06:09.673 WARN (qtp1267082248-29872) [c:my_collection
s:shard1_1 r:core_node20 x:my_collection_shard1_1_replica_n19]
o.a.s.u.p.DistributedZkUpdateProcessor Error sending update to
http://solr_133:8389/solr =>
org.apache.solr.client.solrj.SolrServerException: Timeout occured while
waiting response from server at: http://solr_133:8389/solr/........
/*
Then at `2020-12-06 19:06:09` timestamp we got following. Which I'm assuming
indicates that commit got over, since after this CPU utilization fall back
again to 2%. And We also got curl reponse by this timestamp with status=0. *
/
2020-12-06 19:06:09.673 INFO (qtp1267082248-29872) [c:my_collection
s:shard1_1 r:core_node20 x:my_collection_shard1_1_replica_n19]
o.a.s.u.p.LogUpdateProcessorFactory [my_collection_shard1_1_replica_n19]
webapp=/solr path=/update
params={openSearcher=true&commit=true&wt=json}{commit=} 0 1702717
In short to summarize:
Commit was issued at `2020-12-06 18:37:40` (approx) timestamp.
Till `2020-12-06 18:46:50` timestamp there was no `start commit` received
and CPU was at 2% till it throws `SolrCmdDistributor` error at `2020-12-06
18:47:47` timestamp. Then at around the same timestamp `2020-12-06 18:47:47`
received the `start commit` request. And by this time CPU utilization is
moved up to 14%.
Then at `2020-12-06 19:06:09` timestamp got `DistributedZkUpdateProcessor`
error message and then at `2020-12-06 19:06:09.673` timestamp got some
commit related message (which I'm assuming its indicates end of commit) and
CPU is again fall back to 2%.
Due to these timeout errors overall commit time is getting double. Can
someone guide me to resolve this issue.
--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Commits (with openSearcher = true) are too slow in solr 8
Posted by "raj.yadav" <ra...@cse.ism.ac.in>.
matthew sporleder wrote
> On unix the top command will tell you. On windows you need to find
> the disk latency stuff.
Will check this and report here
matthew sporleder wrote
> Are you on a spinning disk or on a (good) SSD?
we are using SSD
matthew sporleder wrote
> Anyway, my theory is that trying to do too many commits in parallel
> (too many or not enough shards) is causing iowait = high latency to
> work through.
Can you please elaborate more about this.
--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Commits (with openSearcher = true) are too slow in solr 8
Posted by matthew sporleder <ms...@gmail.com>.
On unix the top command will tell you. On windows you need to find
the disk latency stuff.
Are you on a spinning disk or on a (good) SSD?
Anyway my theory is that trying to do too many commits in parallel
(too many or not enough shards) is causing iowait = high latency to
work through.
On Sun, Dec 6, 2020 at 9:05 AM raj.yadav <ra...@cse.ism.ac.in> wrote:
>
> matthew sporleder wrote
> > Are you stuck in iowait during that commit?
>
> I am not sure how do I determine that, could you help me here.
>
>
>
>
> --
> Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Commits (with openSearcher = true) are too slow in solr 8
Posted by "raj.yadav" <ra...@cse.ism.ac.in>.
matthew sporleder wrote
> Are you stuck in iowait during that commit?
I am not sure how do I determine that, could you help me here.
--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Commits (with openSearcher = true) are too slow in solr 8
Posted by "raj.yadav" <ra...@cse.ism.ac.in>.
Hey Matthew,
matthew sporleder wrote
> Are you stuck in iowait during that commit?
I am not sure how do I determine that, could you help me here.
--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Commits (with openSearcher = true) are too slow in solr 8
Posted by matthew sporleder <ms...@gmail.com>.
Are you stuck in iowait during that commit?
On Fri, Dec 4, 2020 at 6:28 AM raj.yadav <ra...@cse.ism.ac.in> wrote:
>
> Hi everyone,
>
> As per suggestions in previous post (by Erick and Shawn) we did following
> changes.
>
> OLD CACHE CONFIG
> <filterCache class="solr.CaffeineCache"
> size="32768"
> initialSize="6000"
> autowarmCount="6000"/>
>
> <queryResultCache class="solr.CaffeineCache"
> size="25600"
> initialSize="6000"
> autowarmCount="0"/>
>
> <documentCache class="solr.CaffeineCache"
> size="32768"
> initialSize="6144"
> autowarmCount="0"/>
>
> NEW CACHE CONFIG
> <filterCache class="solr.CaffeineCache"
> size="8192"
> initialSize="512"
> autowarmCount="512"/>
>
> <queryResultCache class="solr.CaffeineCache"
> size="8192"
> initialSize="3000"
> autowarmCount="0"/>
>
> <documentCache class="solr.CaffeineCache"
> size="8192"
> initialSize="3072"
> autowarmCount="0"/>
>
>
> *Reduced JVM heap size from 30GB to 26GB*
>
>
>
> *Currently query request rate on the system is zero.
> But still, commit with openSearcher=true is taking 25 mins.*
>
> We looked into solr logs, and observed the following things:
>
> 1. /Once the commit is issued, five (shard1_0, shard1_1, shard2_2, shard3_0,
> shard3_1) of the six shards have immediately started processing commit but
> on one shard (shard2_1) we are getting follwing error:/
>
> 2020-12-03 12:29:17.518 ERROR
> (updateExecutor-5-thread-6-processing-n:solr_132:8389_solr
> x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
> r:core_node22) [c:my_collection s:shard2_1 r:core_node22
> x:my_collection_shard2_1_replica_n21] o.a.s.u.SolrCmdDistributor
> org.apache.solr.client.solrj.SolrServerException: Timeout occured while
> waiting response from server at:
> http://solr_198:8389/solr/my_collection_shard3_1_replica_n23/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsolr_132%3A8389%2Fsolr%2Fmy_collection_shard2_1_replica_n21%2F
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:407)
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:753)
> at
> org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:369)
> at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290)
> at
> org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:344)
> at
> org.apache.solr.update.SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:333)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.util.concurrent.TimeoutException
> at
> org.eclipse.jetty.client.util.InputStreamResponseListener.get(InputStreamResponseListener.java:216)
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:398)
> ... 13 more
>
> 2020-12-03 12:29:17.518 ERROR
> (updateExecutor-5-thread-2-processing-n:solr_132:8389_solr
> x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
> r:core_node22) [c:my_collection s:shard2_1 r:core_node22
> x:my_collection_shard2_1_replica_n21] o.a.s.u.SolrCmdDistributor
> org.apache.solr.client.solrj.SolrServerException: Timeout occured while
> waiting response from server at:
> http://solr_199:8389/solr/my_collection_shard1_0_replica_n7/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsolr_132%3A8389%2Fsolr%2Fmy_collection_shard2_1_replica_n21%2F
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:407)
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:753)
> at
> org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:369)
> at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290)
> at
> org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:344)
> at
> org.apache.solr.update.SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:333)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.util.concurrent.TimeoutException
> at
> org.eclipse.jetty.client.util.InputStreamResponseListener.get(InputStreamResponseListener.java:216)
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:398)
> ... 13 more
>
> 2020-12-03 12:29:17.518 ERROR
> (updateExecutor-5-thread-3-processing-n:solr_132:8389_solr
> x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
> r:core_node22) [c:my_collection s:shard2_1 r:core_node22
> x:my_collection_shard2_1_replica_n21] o.a.s.u.SolrCmdDistributor
> org.apache.solr.client.solrj.SolrServerException: Timeout occured while
> waiting response from server at:
> http://solr_200:8389/solr/my_collection_shard1_1_replica_n19/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsolr_132%3A8389%2Fsolr%2Fmy_collection_shard2_1_replica_n21%2F
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:407)
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:753)
> at
> org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:369)
> at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290)
> at
> org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:344)
> at
> org.apache.solr.update.SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:333)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.util.concurrent.TimeoutException
> at
> org.eclipse.jetty.client.util.InputStreamResponseListener.get(InputStreamResponseListener.java:216)
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:398)
> ... 13 more
>
> 2020-12-03 12:29:17.518 ERROR
> (updateExecutor-5-thread-5-processing-n:solr_132:8389_solr
> x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
> r:core_node22) [c:my_collection s:shard2_1 r:core_node22
> x:my_collection_shard2_1_replica_n21] o.a.s.u.SolrCmdDistributor
> org.apache.solr.client.solrj.SolrServerException: Timeout occured while
> waiting response from server at:
> http://solr_133:8389/solr/my_collection_shard3_0_replica_n15/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsolr_132%3A8389%2Fsolr%2Fmy_collection_shard2_1_replica_n21%2F
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:407)
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:753)
> at
> org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:369)
> at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290)
> at
> org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:344)
> at
> org.apache.solr.update.SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:333)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.util.concurrent.TimeoutException
> at
> org.eclipse.jetty.client.util.InputStreamResponseListener.get(InputStreamResponseListener.java:216)
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:398)
> ... 13 more
>
> 2020-12-03 12:29:17.518 ERROR
> (updateExecutor-5-thread-4-processing-n:solr_132:8389_solr
> x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
> r:core_node22) [c:my_collection s:shard2_1 r:core_node22
> x:my_collection_shard2_1_replica_n21] o.a.s.u.SolrCmdDistributor
> org.apache.solr.client.solrj.SolrServerException: Timeout occured while
> waiting response from server at:
> http://solr_254:8389/solr/my_collection_shard2_0_replica_n11/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsolr_132%3A8389%2Fsolr%2Fmy_collection_shard2_1_replica_n21%2F
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:407)
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:753)
> at
> org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:369)
> at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290)
> at
> org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:344)
> at
> org.apache.solr.update.SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:333)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.util.concurrent.TimeoutException
> at
> org.eclipse.jetty.client.util.InputStreamResponseListener.get(InputStreamResponseListener.java:216)
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:398)
> ... 13 more
>
> 2020-12-03 12:29:17.522 INFO (qtp2034610694-29830) [c:my_collection
> s:shard2_1 r:core_node22 x:my_collection_shard2_1_replica_n21]
> o.a.s.u.SolrCmdDistributor SolrCmdDistributor found 5 errors
> 2020-12-03 12:29:17.522 INFO (qtp2034610694-29830) [c:my_collection
> s:shard2_1 r:core_node22 x:my_collection_shard2_1_replica_n21]
> o.a.s.u.DirectUpdateHandler2 start
> commit{_version_=1685060119452188672,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
>
-- snipped --
>
>
>
> --
> Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Commits (with openSearcher = true) are too slow in solr 8
Posted by "raj.yadav" <ra...@cse.ism.ac.in>.
Hi everyone,
As per suggestions in previous post (by Erick and Shawn) we did following
changes.
OLD CACHE CONFIG
<filterCache class="solr.CaffeineCache"
size="32768"
initialSize="6000"
autowarmCount="6000"/>
<queryResultCache class="solr.CaffeineCache"
size="25600"
initialSize="6000"
autowarmCount="0"/>
<documentCache class="solr.CaffeineCache"
size="32768"
initialSize="6144"
autowarmCount="0"/>
NEW CACHE CONFIG
<filterCache class="solr.CaffeineCache"
size="8192"
initialSize="512"
autowarmCount="512"/>
<queryResultCache class="solr.CaffeineCache"
size="8192"
initialSize="3000"
autowarmCount="0"/>
<documentCache class="solr.CaffeineCache"
size="8192"
initialSize="3072"
autowarmCount="0"/>
*Reduced JVM heap size from 30GB to 26GB*
*Currently query request rate on the system is zero.
But still, commit with openSearcher=true is taking 25 mins.*
We looked into solr logs, and observed the following things:
1. /Once the commit is issued, five (shard1_0, shard1_1, shard2_2, shard3_0,
shard3_1) of the six shards have immediately started processing commit but
on one shard (shard2_1) we are getting follwing error:/
2020-12-03 12:29:17.518 ERROR
(updateExecutor-5-thread-6-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.u.SolrCmdDistributor
org.apache.solr.client.solrj.SolrServerException: Timeout occured while
waiting response from server at:
http://solr_198:8389/solr/my_collection_shard3_1_replica_n23/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsolr_132%3A8389%2Fsolr%2Fmy_collection_shard2_1_replica_n21%2F
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:407)
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:753)
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:369)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290)
at
org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:344)
at
org.apache.solr.update.SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:333)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.TimeoutException
at
org.eclipse.jetty.client.util.InputStreamResponseListener.get(InputStreamResponseListener.java:216)
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:398)
... 13 more
2020-12-03 12:29:17.518 ERROR
(updateExecutor-5-thread-2-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.u.SolrCmdDistributor
org.apache.solr.client.solrj.SolrServerException: Timeout occured while
waiting response from server at:
http://solr_199:8389/solr/my_collection_shard1_0_replica_n7/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsolr_132%3A8389%2Fsolr%2Fmy_collection_shard2_1_replica_n21%2F
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:407)
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:753)
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:369)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290)
at
org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:344)
at
org.apache.solr.update.SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:333)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.TimeoutException
at
org.eclipse.jetty.client.util.InputStreamResponseListener.get(InputStreamResponseListener.java:216)
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:398)
... 13 more
2020-12-03 12:29:17.518 ERROR
(updateExecutor-5-thread-3-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.u.SolrCmdDistributor
org.apache.solr.client.solrj.SolrServerException: Timeout occured while
waiting response from server at:
http://solr_200:8389/solr/my_collection_shard1_1_replica_n19/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsolr_132%3A8389%2Fsolr%2Fmy_collection_shard2_1_replica_n21%2F
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:407)
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:753)
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:369)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290)
at
org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:344)
at
org.apache.solr.update.SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:333)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.TimeoutException
at
org.eclipse.jetty.client.util.InputStreamResponseListener.get(InputStreamResponseListener.java:216)
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:398)
... 13 more
2020-12-03 12:29:17.518 ERROR
(updateExecutor-5-thread-5-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.u.SolrCmdDistributor
org.apache.solr.client.solrj.SolrServerException: Timeout occured while
waiting response from server at:
http://solr_133:8389/solr/my_collection_shard3_0_replica_n15/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsolr_132%3A8389%2Fsolr%2Fmy_collection_shard2_1_replica_n21%2F
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:407)
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:753)
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:369)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290)
at
org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:344)
at
org.apache.solr.update.SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:333)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.TimeoutException
at
org.eclipse.jetty.client.util.InputStreamResponseListener.get(InputStreamResponseListener.java:216)
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:398)
... 13 more
2020-12-03 12:29:17.518 ERROR
(updateExecutor-5-thread-4-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.u.SolrCmdDistributor
org.apache.solr.client.solrj.SolrServerException: Timeout occured while
waiting response from server at:
http://solr_254:8389/solr/my_collection_shard2_0_replica_n11/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsolr_132%3A8389%2Fsolr%2Fmy_collection_shard2_1_replica_n21%2F
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:407)
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:753)
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:369)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290)
at
org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:344)
at
org.apache.solr.update.SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:333)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.TimeoutException
at
org.eclipse.jetty.client.util.InputStreamResponseListener.get(InputStreamResponseListener.java:216)
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:398)
... 13 more
2020-12-03 12:29:17.522 INFO (qtp2034610694-29830) [c:my_collection
s:shard2_1 r:core_node22 x:my_collection_shard2_1_replica_n21]
o.a.s.u.SolrCmdDistributor SolrCmdDistributor found 5 errors
2020-12-03 12:29:17.522 INFO (qtp2034610694-29830) [c:my_collection
s:shard2_1 r:core_node22 x:my_collection_shard2_1_replica_n21]
o.a.s.u.DirectUpdateHandler2 start
commit{_version_=1685060119452188672,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
2. /As per the logs, on five shards (shard1_0, shard1_1, shard2_2, shard3_0,
shard3_1) commit is getting completed in around 12 to 14 mins (which is same
as our solr-5.4.2 system), and a large part of commit time is spent on
reloading external files. You can see in logs that commit is received on
five shards at around *2020-12-03 12:19 timestamp* , but on shard2_0 there
are error messages at *2020-12-03 12:29 timestamp* and commit is received at
*2020-12-03 12:29 timestamp.*
/
3. /shard2_1 (the one on which we got the above error) has started the
commit process after 10 min of issuing the commit and it is taking another
12 to 14 mins to complete the commit task. Due to this our overall commit
time is around 25 mins./
I'm not able to figure out why we are getting this error on one shard. Also,
this error shard is not always sahrd2_1 sometime we get a similar error on
shard2_0 but in that case, shard2_1 and other fours work fine
I'm attaching detail solr.log and system information:
GC setting:
GC_TUNE=" \
-XX:+UseG1GC \
-XX:+PerfDisableSharedMem \
-XX:+ParallelRefProcEnabled \
-XX:G1HeapRegionSize=8m \
-XX:MaxGCPauseMillis=150 \
-XX:InitiatingHeapOccupancyPercent=60 \
-XX:+UseLargePages \
-XX:+AggressiveOpts \
"
Solr version => solr-8.5.2
Solr Collection details: (running in solrCloud mode)
It has 6 shards, and each shard has only one replica (which is also a
leader) and replica type is NRT
Total doc in collection: 77 million
Each shard Index size: 11 GB
avg size/doc: 1.0Kb
Shard_name => host_machine
shard1_0=>solr_199
shard1_1=>solr_200
shard2_0=> solr_254
shard2_1=> solr_132
shard3_0=>solr_133
shard3_1=>solr_198
Zookeeper => We are using external zookeeper ensemble (3 node cluster)
*SOLR LOGS*
@#@###### => solr_254
2020-12-03 12:16:51.361 INFO (qtp2034610694-30245) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_0.replica_n11:QUERY./select.requests&key=solr.core.my_collection.shard2_0.replica_n11:UPDATE./update.requests&key=solr.core.my_collection.shard2_0.replica_n11:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:16:51.364 INFO (qtp2034610694-30520) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=2
2020-12-03 12:17:51.579 INFO (qtp2034610694-30245) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_0.replica_n11:QUERY./select.requests&key=solr.core.my_collection.shard2_0.replica_n11:UPDATE./update.requests&key=solr.core.my_collection.shard2_0.replica_n11:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:17:51.581 INFO (qtp2034610694-30520) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:18:51.779 INFO (qtp2034610694-30245) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_0.replica_n11:QUERY./select.requests&key=solr.core.my_collection.shard2_0.replica_n11:UPDATE./update.requests&key=solr.core.my_collection.shard2_0.replica_n11:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:18:51.780 INFO (qtp2034610694-30293) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:19:17.526 INFO (qtp2034610694-30293) [c:my_collection
s:shard2_0 r:core_node13 x:my_collection_shard2_0_replica_n11]
o.a.s.u.DirectUpdateHandler2 start
commit{_version_=1685059490310782976,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
2020-12-03 12:19:17.526 INFO (qtp2034610694-30293) [c:my_collection
s:shard2_0 r:core_node13 x:my_collection_shard2_0_replica_n11]
o.a.s.u.SolrIndexWriter Calling setCommitData with
IW:org.apache.solr.update.SolrIndexWriter@7999d49e
commitCommandVersion:1685059490310782976
2020-12-03 12:19:17.730 INFO (qtp2034610694-30293) [c:my_collection
s:shard2_0 r:core_node13 x:my_collection_shard2_0_replica_n11]
o.a.s.s.SolrIndexSearcher Opening
[Searcher@3ecd5548[my_collection_shard2_0_replica_n11] main]
2020-12-03 12:19:17.732 INFO (qtp2034610694-30293) [c:my_collection
s:shard2_0 r:core_node13 x:my_collection_shard2_0_replica_n11]
o.a.s.u.DirectUpdateHandler2 end_commit_flush
2020-12-03 12:19:51.956 INFO (qtp2034610694-30245) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_0.replica_n11:QUERY./select.requests&key=solr.core.my_collection.shard2_0.replica_n11:UPDATE./update.requests&key=solr.core.my_collection.shard2_0.replica_n11:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:19:51.959 INFO (qtp2034610694-30401) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=1
2020-12-03 12:20:45.128 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_AF :5234563 missing keys [WWKeNEOp,
P9hl8bGD, Htqj5VOu, NOTOQMim, tP8VO0QL, VxHndrnP, fIqqfPkp, N_gHFlI_,
54yONW2l, AJFFGJ8j]
2020-12-03 12:20:45.200 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_manualboost_ACs :3045 missing keys
[blXh9_dw, uhvA7VfX, fqvfSzuB, ardQkLJ6, LoUgZFaE, BFV775k9, L_T7S2tB,
OMjJj4q4, lKBa6juj, rE2TnBHh]
2020-12-03 12:20:52.164 INFO (qtp2034610694-30245) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_0.replica_n11:QUERY./select.requests&key=solr.core.my_collection.shard2_0.replica_n11:UPDATE./update.requests&key=solr.core.my_collection.shard2_0.replica_n11:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:20:52.166 INFO (qtp2034610694-30401) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:21:10.358 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_AB_level :661658 missing keys [23647754,
10529959, 23263400, 20939560, 11299581, 14108851, 19203485, 2934559,
18385364, 19273969]
2020-12-03 12:21:12.511 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_malayalam
2020-12-03 12:21:13.388 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_malayalam :51996 missing keys [cZn2f6Xs,
4Z6lW4Li, 5RJuPUK4, GNMHIYVb, ugLDMeC1, e6TYULIH, pYxX0twz, C5n2jwht,
EDbM64uJ, a-SVfjW9]
2020-12-03 12:21:13.388 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_malayalam reloaded
2020-12-03 12:21:13.388 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_hindi
2020-12-03 12:21:18.665 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_hindi :314390 missing keys [peUFCam7,
ynaCbJxu, Z7zDcZFn, L-Lqsq6X, 9ly_5w8m, jPKgJcWj, Nb7Equ17, XFYVtslF,
qOrQRrra, hpsM3g77]
2020-12-03 12:21:18.665 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_hindi reloaded
2020-12-03 12:21:18.665 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_bengali
2020-12-03 12:21:20.564 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_bengali :108944 missing keys [yJpx_SpM,
STjLw6br, 0b06UNFk, krextkJu, UhnWkIs1, S6IhfFgC, Ae9Jz0Ah, RVPzenj_,
MbQCj-IA, GZu-zly4]
2020-12-03 12:21:20.564 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_bengali reloaded
2020-12-03 12:21:20.564 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_urdu
2020-12-03 12:21:21.187 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_urdu :33186 missing keys [ocPqghAA,
3E2mc6yo, 6TatU1en, Qjs9S8mY, kQCvEN9Y, y7pyiqx-, tG4nP8RA, m5V4L80Z,
bnA-25w7, E7BZ91he]
2020-12-03 12:21:21.187 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_urdu reloaded
2020-12-03 12:21:21.187 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_AE_primary
2020-12-03 12:21:52.354 INFO (qtp2034610694-30218) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_0.replica_n11:QUERY./select.requests&key=solr.core.my_collection.shard2_0.replica_n11:UPDATE./update.requests&key=solr.core.my_collection.shard2_0.replica_n11:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:21:52.357 INFO (qtp2034610694-30245) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=1
2020-12-03 12:22:52.780 INFO (qtp2034610694-30218) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_0.replica_n11:QUERY./select.requests&key=solr.core.my_collection.shard2_0.replica_n11:UPDATE./update.requests&key=solr.core.my_collection.shard2_0.replica_n11:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:22:52.782 INFO (qtp2034610694-30245) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:23:53.214 INFO (qtp2034610694-30218) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_0.replica_n11:QUERY./select.requests&key=solr.core.my_collection.shard2_0.replica_n11:UPDATE./update.requests&key=solr.core.my_collection.shard2_0.replica_n11:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:23:53.216 INFO (qtp2034610694-30520) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:24:08.277 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_AE_primary :9946008 missing keys
[DfGJXwII, L7AaaN_9, 1IgF5TwC, MOzvgwfX, 5dzFIeup, JP7SqZsj, 1pjDJpgX,
zYikBLhg, Mzk1ZmRh, QWNSeS25]
2020-12-03 12:24:08.277 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_AE_primary reloaded
2020-12-03 12:24:08.277 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_punjabi
2020-12-03 12:24:10.269 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_punjabi :117925 missing keys [mo6fMaVh,
tzAwxvtj, 7tpfikGT, XNFeyrdr, tkPFUx_I, J7t4TbgR, 0B-BndhI, aEPNxZIY,
jKNnSkuj, XMWoK3VG]
2020-12-03 12:24:10.269 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_punjabi reloaded
2020-12-03 12:24:10.269 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_marathi
2020-12-03 12:24:11.153 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_marathi :49601 missing keys [ZuHbnZrs,
O4tzIeqB, 9M6B1pMh, SmQJINlD, ba93lzRA, a8te1FzS, h5KnrFk7, 8FqvXXSk,
Mq5G1H7o, 7e0F61HV]
2020-12-03 12:24:11.153 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_marathi reloaded
2020-12-03 12:24:11.153 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_english
2020-12-03 12:24:45.947 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_english :2060987 missing keys [qX0rzqI_,
0Y_wd_2f, avPGWOXX, pHSsQA8T, LBZl3cbT, U5GynSQN, muPE6G_Y, cgDvU6uP,
bE8hgzKe, gwonomiZ]
2020-12-03 12:24:45.947 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_english reloaded
2020-12-03 12:24:45.947 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_AC_level
2020-12-03 12:24:53.797 INFO (qtp2034610694-30218) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_0.replica_n11:QUERY./select.requests&key=solr.core.my_collection.shard2_0.replica_n11:UPDATE./update.requests&key=solr.core.my_collection.shard2_0.replica_n11:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:24:53.798 INFO (qtp2034610694-30520) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:25:54.106 INFO (qtp2034610694-30218) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_0.replica_n11:QUERY./select.requests&key=solr.core.my_collection.shard2_0.replica_n11:UPDATE./update.requests&key=solr.core.my_collection.shard2_0.replica_n11:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:25:54.109 INFO (qtp2034610694-30245) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=1
2020-12-03 12:26:54.289 INFO (qtp2034610694-30441) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_0.replica_n11:QUERY./select.requests&key=solr.core.my_collection.shard2_0.replica_n11:UPDATE./update.requests&key=solr.core.my_collection.shard2_0.replica_n11:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:26:54.291 INFO (qtp2034610694-30218) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:27:39.954 INFO (qtp2034610694-30218) [c:my_collection
s:shard2_0 r:core_node13 x:my_collection_shard2_0_replica_n11]
o.a.s.c.S.Request [my_collection_shard2_0_replica_n11] webapp=/solr
path=/select
params={mm=100%25&df=AC&distrib=false&fl=pid&fl=score&shards.purpose=16388&start=0&fsv=true&shard.url=http://solr_254:8389/solr/my_collection_shard2_0_replica_n11/&_forwardedCount=1&rows=10&version=2&q=pid:"avPGWOXX"&defType=edismax&omitHeader=false&NOW=1606998459943&isShard=true&boost=0&sow=true&timeAllowed=1000&wt=javabin&_=1606996632415}
hits=0 status=0 QTime=3
2020-12-03 12:27:55.102 INFO (qtp2034610694-30441) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_0.replica_n11:QUERY./select.requests&key=solr.core.my_collection.shard2_0.replica_n11:UPDATE./update.requests&key=solr.core.my_collection.shard2_0.replica_n11:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:27:55.104 INFO (qtp2034610694-30218) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:28:55.388 INFO (qtp2034610694-30441) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_0.replica_n11:QUERY./select.requests&key=solr.core.my_collection.shard2_0.replica_n11:UPDATE./update.requests&key=solr.core.my_collection.shard2_0.replica_n11:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:28:55.390 INFO (qtp2034610694-30218) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:29:17.535 WARN (qtp2034610694-30182) [c:my_collection
s:shard2_0 r:core_node13 x:my_collection_shard2_0_replica_n11]
o.e.j.s.HttpChannelState org.eclipse.jetty.io.EofException: Reset
cancel_stream_error
2020-12-03 12:29:47.518 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_AC_level :18479876 missing keys
[umI3MWuQ, JvUe6G-_, RFfZKs4w, 8clNfIsa, DN2BAnDu, 32jVSnYH, M-D-9pAR,
8NAodfoK, CHn_juqL, hDJR6XaL]
2020-12-03 12:29:47.518 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_AC_level reloaded
2020-12-03 12:29:47.518 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_AF
2020-12-03 12:29:55.702 INFO (qtp2034610694-30441) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_0.replica_n11:QUERY./select.requests&key=solr.core.my_collection.shard2_0.replica_n11:UPDATE./update.requests&key=solr.core.my_collection.shard2_0.replica_n11:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:29:55.703 INFO (qtp2034610694-30123) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:30:01.843 INFO (qtp2034610694-30218) [c:my_collection
s:shard2_0 r:core_node13 x:my_collection_shard2_0_replica_n11]
o.a.s.u.DirectUpdateHandler2 start
commit{_version_=1685060165926125568,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
2020-12-03 12:30:01.843 INFO (qtp2034610694-30218) [c:my_collection
s:shard2_0 r:core_node13 x:my_collection_shard2_0_replica_n11]
o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
2020-12-03 12:30:01.847 INFO (qtp2034610694-30218) [c:my_collection
s:shard2_0 r:core_node13 x:my_collection_shard2_0_replica_n11]
o.a.s.u.DirectUpdateHandler2 end_commit_flush
2020-12-03 12:30:01.849 INFO (qtp2034610694-30218) [c:my_collection
s:shard2_0 r:core_node13 x:my_collection_shard2_0_replica_n11]
o.a.s.u.p.LogUpdateProcessorFactory [my_collection_shard2_0_replica_n11]
webapp=/solr path=/update
params={openSearcher=false&commit=true&env=qa&wt=json}{commit=} 0 79
2020-12-03 12:30:55.879 INFO (qtp2034610694-30441) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_0.replica_n11:QUERY./select.requests&key=solr.core.my_collection.shard2_0.replica_n11:UPDATE./update.requests&key=solr.core.my_collection.shard2_0.replica_n11:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:30:55.880 INFO (qtp2034610694-30123) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:31:15.729 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_AF :5234563 missing keys [WWKeNEOp,
P9hl8bGD, Htqj5VOu, NOTOQMim, tP8VO0QL, VxHndrnP, fIqqfPkp, N_gHFlI_,
54yONW2l, AJFFGJ8j]
2020-12-03 12:31:15.729 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_AF reloaded
2020-12-03 12:31:15.729 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_bhojpuri
2020-12-03 12:31:17.135 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_bhojpuri :80914 missing keys [21JkD4TV,
pSIWYJuE, PQlcukgI, cSn0v29w, vMSo9khA, b-ep7NPS, 8uq97zzX, syEpVNmJ,
IlhTQeWZ, EIPeYz28]
2020-12-03 12:31:17.135 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_bhojpuri reloaded
2020-12-03 12:31:17.135 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_kannada
2020-12-03 12:31:18.299 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_kannada :62782 missing keys [2ZmXX6SV,
V1IlmJ18, AsFLJ3Yn, c-yhRYDT, qOJerBJq, 7iICrIty, J4lm2n8W, is4KI6Eq,
ffVY59it, KyoMJBa9]
2020-12-03 12:31:18.299 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_kannada reloaded
2020-12-03 12:31:18.299 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_gujarati
2020-12-03 12:31:19.434 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_gujarati :64719 missing keys [y0IfmBUR,
z2fNs6SH, tVe_aBfK, GsUc51io, EeGK1WIO, 6H2kniND, zN92bAP2, nd69KcB9,
Gp11jTLi, H_Z6Wg5B]
2020-12-03 12:31:19.434 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_gujarati reloaded
2020-12-03 12:31:19.434 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_odia
2020-12-03 12:31:19.665 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_odia :12539 missing keys [LVsQn_cL,
jDOQ9oWW, xTB2Duf9, 3CGn8vI9, 7BOUDdZJ, JBHr_d8J, 2EgrM24O, SN0J0wDV,
UXRmfWCb, d1aer4Mr]
2020-12-03 12:31:19.665 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_odia reloaded
2020-12-03 12:31:19.665 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_tamil
2020-12-03 12:31:21.247 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_tamil :89467 missing keys [drpalXsJ,
Hccjv831, VQ65X2Jv, jCOoIObQ, hj2tmPIx, GIiZO90Q, ixAqa05O, t2QMsATn,
1-pHt4E1, -_wW2ycj]
2020-12-03 12:31:21.247 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_tamil reloaded
2020-12-03 12:31:21.247 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_haryanvi
2020-12-03 12:31:21.434 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_haryanvi :9988 missing keys [fLqY7IfH,
SHafvTkd, Kizpgxr1, S7qpm8hO, _YSKGq5x, 7nmN9XB1, eoN7t6XQ, vXxCHq6Y,
RYqFrXZk, 3dAvFFTP]
2020-12-03 12:31:21.434 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_haryanvi reloaded
2020-12-03 12:31:21.434 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_rajasthani
2020-12-03 12:31:21.852 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_rajasthani :24388 missing keys [lkA_r3o7,
1dxsfwx1, Q2qOgk5L, n2lJdK3D, V2gr4MU3, CnVRGKwN, g5MLk0P0, imRLTiHe,
jeNdTUKy, mu1WYLQ-]
2020-12-03 12:31:21.852 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_rajasthani reloaded
2020-12-03 12:31:21.852 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_AC_ADs
2020-12-03 12:31:22.870 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_AC_ADs :57874 missing keys [pc7Tr5jO,
7IY_lge_, MNLWeqBq, -EMNXOHX, F-ji7fTD, 3Lh3vyZc, 7eVi_9t8, HAB12hE9,
pXR8RWHg, UltKIW0J]
2020-12-03 12:31:22.870 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_AC_ADs reloaded
2020-12-03 12:31:22.870 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_telugu
2020-12-03 12:31:24.053 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_telugu :68460 missing keys [kYgt06qn,
q6Z_ik8h, AT9Hesvv, f63iQ1NV, pMbagei9, xfzlU5dT, kL9aVqI9, m3R-UYnC,
2KrJysAj, heKVGbUJ]
2020-12-03 12:31:24.053 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_telugu reloaded
2020-12-03 12:31:24.053 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_manualboost_ACs
2020-12-03 12:31:24.113 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_manualboost_ACs :3045 missing keys
[blXh9_dw, uhvA7VfX, fqvfSzuB, ardQkLJ6, LoUgZFaE, BFV775k9, L_T7S2tB,
OMjJj4q4, lKBa6juj, rE2TnBHh]
2020-12-03 12:31:24.114 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_manualboost_ACs reloaded
2020-12-03 12:31:24.114 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_AA
2020-12-03 12:31:24.307 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_AA :10477 missing keys [sfSKzJ2I,
3IX30VNb, hXZLZtJa, Xhks2_wg, gCB9jM9t, zcRsVJSX, L-DZdfHe, eIyVP4Yd,
yExPB61v, Mpusokq5]
2020-12-03 12:31:24.307 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_AA reloaded
2020-12-03 12:31:24.307 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_AB_level
2020-12-03 12:31:49.856 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_AB_level :661658 missing keys [23647754,
10529959, 23263400, 20939560, 11299581, 14108851, 19203485, 2934559,
18385364, 19273969]
2020-12-03 12:31:49.856 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_AB_level reloaded
2020-12-03 12:31:49.856 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_AE
2020-12-03 12:31:56.061 INFO (qtp2034610694-30245) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_0.replica_n11:QUERY./select.requests&key=solr.core.my_collection.shard2_0.replica_n11:UPDATE./update.requests&key=solr.core.my_collection.shard2_0.replica_n11:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:31:56.063 INFO (qtp2034610694-30441) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:32:56.250 INFO (qtp2034610694-30245) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_0.replica_n11:QUERY./select.requests&key=solr.core.my_collection.shard2_0.replica_n11:UPDATE./update.requests&key=solr.core.my_collection.shard2_0.replica_n11:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:32:56.253 INFO (qtp2034610694-30441) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=1
2020-12-03 12:33:56.417 INFO (qtp2034610694-30245) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_0.replica_n11:QUERY./select.requests&key=solr.core.my_collection.shard2_0.replica_n11:UPDATE./update.requests&key=solr.core.my_collection.shard2_0.replica_n11:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:33:56.418 INFO (qtp2034610694-30182) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:34:53.944 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_AE :10980807 missing keys [nfSSYlGn,
WWKeNEOp, P9hl8bGD, gxyKPooc, YjYyMTcy, Htqj5VOu, bZ1v3yJW, NOTOQMim,
qF0KRB7q, tP8VO0QL]
2020-12-03 12:34:53.944 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_AE reloaded
2020-12-03 12:34:53.945 INFO
(searcherExecutor-117-thread-1-processing-n:solr_254:8389_solr
x:my_collection_shard2_0_replica_n11 c:my_collection s:shard2_0
r:core_node13) [c:my_collection s:shard2_0 r:core_node13
x:my_collection_shard2_0_replica_n11] o.a.s.c.SolrCore
[my_collection_shard2_0_replica_n11] Registered new searcher
Searcher@3ecd5548[my_collection_shard2_0_replica_n11]
main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_26(8.5.2):C3369913/111702:[diagnostics={os=Linux,
java.version=1.8.0_262, os.arch=amd64, java.runtime.version=1.8.0_262-b10,
source=merge, os.version=3.10.0-1062.9.1.el7.x86_64, java.vendor=Oracle
Corporation, java.vm.version=25.262-b10, lucene.version=8.5.2,
mergeMaxNumSegments=-1, mergeFactor=5,
timestamp=1606789817461}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]:delGen=74)
Uninverting(_3l(8.5.2):C3812921/58987:[diagnostics={os=Linux,
java.version=1.8.0_262, os.arch=amd64, java.runtime.version=1.8.0_262-b10,
source=merge, os.version=3.10.0-1062.9.1.el7.x86_64, java.vendor=Oracle
Corporation, java.vm.version=25.262-b10, lucene.version=8.5.2,
mergeMaxNumSegments=-1, mergeFactor=5,
timestamp=1606834818111}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]:delGen=58)
Uninverting(_5v(8.5.2):C2678626/57512:[diagnostics={os=Linux,
java.version=1.8.0_262, os.arch=amd64, java.runtime.version=1.8.0_262-b10,
source=merge, os.version=3.10.0-1062.9.1.el7.x86_64, java.vendor=Oracle
Corporation, java.vm.version=25.262-b10, lucene.version=8.5.2,
mergeMaxNumSegments=-1, mergeFactor=5,
timestamp=1606939804937}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]:delGen=21)
Uninverting(_4w(8.5.2):C3196601/26045:[diagnostics={os=Linux,
java.version=1.8.0_262, os.arch=amd64, java.runtime.version=1.8.0_262-b10,
source=merge, os.version=3.10.0-1062.9.1.el7.x86_64, java.vendor=Oracle
Corporation, java.vm.version=25.262-b10, lucene.version=8.5.2,
mergeMaxNumSegments=-1, mergeFactor=5,
timestamp=1606875006051}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]:delGen=40)
Uninverting(_5x(8.5.2):C485034/88026:[diagnostics={os=Linux,
java.version=1.8.0_262, os.arch=amd64, java.runtime.version=1.8.0_262-b10,
source=merge, os.version=3.10.0-1062.9.1.el7.x86_64, java.vendor=Oracle
Corporation, java.vm.version=25.262-b10, lucene.version=8.5.2,
mergeMaxNumSegments=-1, mergeFactor=5,
timestamp=1606940250272}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]:delGen=21)
Uninverting(_6h(8.5.2):C92098/14:[diagnostics={os=Linux,
java.version=1.8.0_262, os.arch=amd64, java.runtime.version=1.8.0_262-b10,
source=merge, os.version=3.10.0-1062.9.1.el7.x86_64, java.vendor=Oracle
Corporation, java.vm.version=25.262-b10, lucene.version=8.5.2,
mergeMaxNumSegments=-1, mergeFactor=5,
timestamp=1606990203326}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]:delGen=2)
Uninverting(_67(8.5.2):C88423/10467:[diagnostics={os=Linux,
java.version=1.8.0_262, os.arch=amd64, java.runtime.version=1.8.0_262-b10,
source=merge, os.version=3.10.0-1062.9.1.el7.x86_64, java.vendor=Oracle
Corporation, java.vm.version=25.262-b10, lucene.version=8.5.2,
mergeMaxNumSegments=-1, mergeFactor=5,
timestamp=1606961403398}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]:delGen=10)
Uninverting(_6c(8.5.2):C94180/5129:[diagnostics={os=Linux,
java.version=1.8.0_262, os.arch=amd64, java.runtime.version=1.8.0_262-b10,
source=merge, os.version=3.10.0-1062.9.1.el7.x86_64, java.vendor=Oracle
Corporation, java.vm.version=25.262-b10, lucene.version=8.5.2,
mergeMaxNumSegments=-1, mergeFactor=5,
timestamp=1606971604790}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]:delGen=6)
Uninverting(_6i(8.5.2):C3451/273:[diagnostics={os=Linux, java.vendor=Oracle
Corporation, java.version=1.8.0_262, java.vm.version=25.262-b10,
lucene.version=8.5.2, os.arch=amd64, java.runtime.version=1.8.0_262-b10,
source=flush, os.version=3.10.0-1062.9.1.el7.x86_64,
timestamp=1606995602508}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]:delGen=1)
Uninverting(_6j(8.5.2):C18:[diagnostics={os=Linux, java.vendor=Oracle
Corporation, java.version=1.8.0_262, java.vm.version=25.262-b10,
lucene.version=8.5.2, os.arch=amd64, java.runtime.version=1.8.0_262-b10,
source=flush, os.version=3.10.0-1062.9.1.el7.x86_64,
timestamp=1606997957538}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}])))}
2020-12-03 12:34:53.950 INFO (qtp2034610694-30293) [c:my_collection
s:shard2_0 r:core_node13 x:my_collection_shard2_0_replica_n11]
o.a.s.u.p.LogUpdateProcessorFactory [my_collection_shard2_0_replica_n11]
webapp=/solr path=/update
params={update.distrib=TOLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://solr_132:8389/solr/my_collection_shard2_1_replica_n21/&commit_end_point=leaders&wt=javabin&version=2&expungeDeletes=false}{commit=}
0 936426
2020-12-03 12:34:56.595 INFO (qtp2034610694-30245) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_0.replica_n11:QUERY./select.requests&key=solr.core.my_collection.shard2_0.replica_n11:UPDATE./update.requests&key=solr.core.my_collection.shard2_0.replica_n11:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:34:56.596 INFO (qtp2034610694-30293) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
@#@###### => solr_132
2020-12-03 12:15:51.229 INFO (qtp2034610694-29927) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_1.replica_n21:QUERY./select.requests&key=solr.core.my_collection.shard2_1.replica_n21:UPDATE./update.requests&key=solr.core.my_collection.shard2_1.replica_n21:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:15:51.230 INFO (qtp2034610694-30010) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:16:51.425 INFO (qtp2034610694-29903) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_1.replica_n21:QUERY./select.requests&key=solr.core.my_collection.shard2_1.replica_n21:UPDATE./update.requests&key=solr.core.my_collection.shard2_1.replica_n21:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:16:51.428 INFO (qtp2034610694-29927) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=1
2020-12-03 12:17:51.662 INFO (qtp2034610694-29903) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_1.replica_n21:QUERY./select.requests&key=solr.core.my_collection.shard2_1.replica_n21:UPDATE./update.requests&key=solr.core.my_collection.shard2_1.replica_n21:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:17:51.664 INFO (qtp2034610694-29998) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:18:51.841 INFO (qtp2034610694-29903) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_1.replica_n21:QUERY./select.requests&key=solr.core.my_collection.shard2_1.replica_n21:UPDATE./update.requests&key=solr.core.my_collection.shard2_1.replica_n21:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:18:51.842 INFO (qtp2034610694-29998) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:19:52.052 INFO (qtp2034610694-29746) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_1.replica_n21:QUERY./select.requests&key=solr.core.my_collection.shard2_1.replica_n21:UPDATE./update.requests&key=solr.core.my_collection.shard2_1.replica_n21:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:19:52.053 INFO (qtp2034610694-29903) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:20:52.246 INFO (qtp2034610694-29927) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_1.replica_n21:QUERY./select.requests&key=solr.core.my_collection.shard2_1.replica_n21:UPDATE./update.requests&key=solr.core.my_collection.shard2_1.replica_n21:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:20:52.249 INFO (qtp2034610694-29903) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=1
2020-12-03 12:21:52.559 INFO (qtp2034610694-30010) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_1.replica_n21:QUERY./select.requests&key=solr.core.my_collection.shard2_1.replica_n21:UPDATE./update.requests&key=solr.core.my_collection.shard2_1.replica_n21:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:21:52.562 INFO (qtp2034610694-30013) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=1
2020-12-03 12:22:53.081 INFO (qtp2034610694-30010) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_1.replica_n21:QUERY./select.requests&key=solr.core.my_collection.shard2_1.replica_n21:UPDATE./update.requests&key=solr.core.my_collection.shard2_1.replica_n21:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:22:53.082 INFO (qtp2034610694-30013) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:23:53.433 INFO (qtp2034610694-30010) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_1.replica_n21:QUERY./select.requests&key=solr.core.my_collection.shard2_1.replica_n21:UPDATE./update.requests&key=solr.core.my_collection.shard2_1.replica_n21:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:23:53.435 INFO (qtp2034610694-29746) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:24:53.856 INFO (qtp2034610694-30010) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_1.replica_n21:QUERY./select.requests&key=solr.core.my_collection.shard2_1.replica_n21:UPDATE./update.requests&key=solr.core.my_collection.shard2_1.replica_n21:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:24:53.857 INFO (qtp2034610694-29746) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:25:54.174 INFO (qtp2034610694-30010) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_1.replica_n21:QUERY./select.requests&key=solr.core.my_collection.shard2_1.replica_n21:UPDATE./update.requests&key=solr.core.my_collection.shard2_1.replica_n21:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:25:54.177 INFO (qtp2034610694-29927) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=1
2020-12-03 12:26:54.829 INFO (qtp2034610694-29862) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_1.replica_n21:QUERY./select.requests&key=solr.core.my_collection.shard2_1.replica_n21:UPDATE./update.requests&key=solr.core.my_collection.shard2_1.replica_n21:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:26:54.840 INFO (qtp2034610694-30010) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:27:39.952 INFO (qtp2034610694-29925) [c:my_collection
s:shard2_1 r:core_node22 x:my_collection_shard2_1_replica_n21]
o.a.s.c.S.Request [my_collection_shard2_1_replica_n21] webapp=/solr
path=/select
params={mm=100%25&df=AC&distrib=false&fl=pid&fl=score&shards.purpose=16388&start=0&fsv=true&shard.url=http://solr_132:8389/solr/my_collection_shard2_1_replica_n21/&_forwardedCount=1&rows=10&version=2&q=pid:"avPGWOXX"&defType=edismax&omitHeader=false&NOW=1606998459943&isShard=true&boost=0&sow=true&timeAllowed=1000&wt=javabin&_=1606996632415}
hits=0 status=0 QTime=2
2020-12-03 12:27:39.959 INFO (qtp2034610694-29903) [c:my_collection
s:shard2_1 r:core_node22 x:my_collection_shard2_1_replica_n21]
o.a.s.c.S.Request [my_collection_shard2_1_replica_n21] webapp=/solr
path=/select params={q=pid:"avPGWOXX"&_forwardedCount=1&_=1606996632415}
hits=0 status=0 QTime=15
2020-12-03 12:27:55.170 INFO (qtp2034610694-29862) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_1.replica_n21:QUERY./select.requests&key=solr.core.my_collection.shard2_1.replica_n21:UPDATE./update.requests&key=solr.core.my_collection.shard2_1.replica_n21:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:27:55.172 INFO (qtp2034610694-29903) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:28:55.448 INFO (qtp2034610694-29862) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_1.replica_n21:QUERY./select.requests&key=solr.core.my_collection.shard2_1.replica_n21:UPDATE./update.requests&key=solr.core.my_collection.shard2_1.replica_n21:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:28:55.450 INFO (qtp2034610694-29903) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:29:17.518 ERROR
(updateExecutor-5-thread-6-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.u.SolrCmdDistributor
org.apache.solr.client.solrj.SolrServerException: Timeout occured while
waiting response from server at:
http://solr_198:8389/solr/my_collection_shard3_1_replica_n23/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsolr_132%3A8389%2Fsolr%2Fmy_collection_shard2_1_replica_n21%2F
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:407)
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:753)
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:369)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290)
at
org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:344)
at
org.apache.solr.update.SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:333)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.TimeoutException
at
org.eclipse.jetty.client.util.InputStreamResponseListener.get(InputStreamResponseListener.java:216)
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:398)
... 13 more
2020-12-03 12:29:17.518 ERROR
(updateExecutor-5-thread-2-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.u.SolrCmdDistributor
org.apache.solr.client.solrj.SolrServerException: Timeout occured while
waiting response from server at:
http://solr_199:8389/solr/my_collection_shard1_0_replica_n7/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsolr_132%3A8389%2Fsolr%2Fmy_collection_shard2_1_replica_n21%2F
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:407)
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:753)
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:369)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290)
at
org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:344)
at
org.apache.solr.update.SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:333)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.TimeoutException
at
org.eclipse.jetty.client.util.InputStreamResponseListener.get(InputStreamResponseListener.java:216)
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:398)
... 13 more
2020-12-03 12:29:17.518 ERROR
(updateExecutor-5-thread-3-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.u.SolrCmdDistributor
org.apache.solr.client.solrj.SolrServerException: Timeout occured while
waiting response from server at:
http://solr_200:8389/solr/my_collection_shard1_1_replica_n19/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsolr_132%3A8389%2Fsolr%2Fmy_collection_shard2_1_replica_n21%2F
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:407)
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:753)
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:369)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290)
at
org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:344)
at
org.apache.solr.update.SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:333)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.TimeoutException
at
org.eclipse.jetty.client.util.InputStreamResponseListener.get(InputStreamResponseListener.java:216)
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:398)
... 13 more
2020-12-03 12:29:17.518 ERROR
(updateExecutor-5-thread-5-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.u.SolrCmdDistributor
org.apache.solr.client.solrj.SolrServerException: Timeout occured while
waiting response from server at:
http://solr_133:8389/solr/my_collection_shard3_0_replica_n15/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsolr_132%3A8389%2Fsolr%2Fmy_collection_shard2_1_replica_n21%2F
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:407)
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:753)
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:369)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290)
at
org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:344)
at
org.apache.solr.update.SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:333)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.TimeoutException
at
org.eclipse.jetty.client.util.InputStreamResponseListener.get(InputStreamResponseListener.java:216)
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:398)
... 13 more
2020-12-03 12:29:17.518 ERROR
(updateExecutor-5-thread-4-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.u.SolrCmdDistributor
org.apache.solr.client.solrj.SolrServerException: Timeout occured while
waiting response from server at:
http://solr_254:8389/solr/my_collection_shard2_0_replica_n11/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsolr_132%3A8389%2Fsolr%2Fmy_collection_shard2_1_replica_n21%2F
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:407)
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:753)
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:369)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290)
at
org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:344)
at
org.apache.solr.update.SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:333)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.TimeoutException
at
org.eclipse.jetty.client.util.InputStreamResponseListener.get(InputStreamResponseListener.java:216)
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:398)
... 13 more
2020-12-03 12:29:17.522 INFO (qtp2034610694-29830) [c:my_collection
s:shard2_1 r:core_node22 x:my_collection_shard2_1_replica_n21]
o.a.s.u.SolrCmdDistributor SolrCmdDistributor found 5 errors
2020-12-03 12:29:17.522 INFO (qtp2034610694-29830) [c:my_collection
s:shard2_1 r:core_node22 x:my_collection_shard2_1_replica_n21]
o.a.s.u.DirectUpdateHandler2 start
commit{_version_=1685060119452188672,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
2020-12-03 12:29:17.522 INFO (qtp2034610694-29830) [c:my_collection
s:shard2_1 r:core_node22 x:my_collection_shard2_1_replica_n21]
o.a.s.u.SolrIndexWriter Calling setCommitData with
IW:org.apache.solr.update.SolrIndexWriter@44786edb
commitCommandVersion:1685060119452188672
2020-12-03 12:29:17.725 INFO (qtp2034610694-29830) [c:my_collection
s:shard2_1 r:core_node22 x:my_collection_shard2_1_replica_n21]
o.a.s.s.SolrIndexSearcher Opening
[Searcher@63f636ac[my_collection_shard2_1_replica_n21] main]
2020-12-03 12:29:17.726 INFO (qtp2034610694-29830) [c:my_collection
s:shard2_1 r:core_node22 x:my_collection_shard2_1_replica_n21]
o.a.s.u.DirectUpdateHandler2 end_commit_flush
2020-12-03 12:29:55.762 INFO (qtp2034610694-29862) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_1.replica_n21:QUERY./select.requests&key=solr.core.my_collection.shard2_1.replica_n21:UPDATE./update.requests&key=solr.core.my_collection.shard2_1.replica_n21:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:29:55.765 INFO (qtp2034610694-29998) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=1
2020-12-03 12:30:01.788 INFO (qtp2034610694-29998) [c:my_collection
s:shard2_1 r:core_node22 x:my_collection_shard2_1_replica_n21]
o.a.s.u.DirectUpdateHandler2 start
commit{_version_=1685060165868453888,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
2020-12-03 12:30:01.788 INFO (qtp2034610694-29998) [c:my_collection
s:shard2_1 r:core_node22 x:my_collection_shard2_1_replica_n21]
o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
2020-12-03 12:30:01.791 INFO (qtp2034610694-29998) [c:my_collection
s:shard2_1 r:core_node22 x:my_collection_shard2_1_replica_n21]
o.a.s.u.DirectUpdateHandler2 end_commit_flush
2020-12-03 12:30:01.791 INFO (qtp2034610694-29998) [c:my_collection
s:shard2_1 r:core_node22 x:my_collection_shard2_1_replica_n21]
o.a.s.u.p.LogUpdateProcessorFactory [my_collection_shard2_1_replica_n21]
webapp=/solr path=/update
params={update.distrib=TOLEADER&waitSearcher=true&openSearcher=false&commit=true&softCommit=false&distrib.from=http://solr_254:8389/solr/my_collection_shard2_0_replica_n11/&commit_end_point=leaders&wt=javabin&version=2&expungeDeletes=false}{commit=}
0 4
2020-12-03 12:30:43.622 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_AF :5260351 missing keys [_r6A0GUX,
Htqj5VOu, NOTOQMim, tP8VO0QL, 2O2V8UOY, VxHndrnP, fIqqfPkp, N_gHFlI_,
dsx_kph1, AJFFGJ8j]
2020-12-03 12:30:55.947 INFO (qtp2034610694-29862) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_1.replica_n21:QUERY./select.requests&key=solr.core.my_collection.shard2_1.replica_n21:UPDATE./update.requests&key=solr.core.my_collection.shard2_1.replica_n21:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:30:55.949 INFO (qtp2034610694-29862) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:31:09.598 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_AB_level :660960 missing keys [17198479,
10529959, 23263400, 20939560, 11299581, 19841955, 14108851, 22958257,
19203485, 18385364]
2020-12-03 12:31:09.664 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_manualboost_ACs :2987 missing keys
[03nlYmrp, uhvA7VfX, Hupc0M1Y, ardQkLJ6, LoUgZFaE, BFV775k9, L_T7S2tB,
OMjJj4q4, rE2TnBHh, UNTsGnao]
2020-12-03 12:31:11.611 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_malayalam
2020-12-03 12:31:12.464 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_malayalam :52013 missing keys [cZn2f6Xs,
5RJuPUK4, flW9TLCW, GNMHIYVb, A22JCYnw, ugLDMeC1, e6TYULIH, pYxX0twz,
EDbM64uJ, iaqOLujH]
2020-12-03 12:31:12.464 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_malayalam reloaded
2020-12-03 12:31:12.464 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_hindi
2020-12-03 12:31:17.671 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_hindi :314570 missing keys [peUFCam7,
ynaCbJxu, Z7zDcZFn, Tv_G7PFw, L-Lqsq6X, 9ly_5w8m, jPKgJcWj, aL-ArcIE,
Nb7Equ17, XFYVtslF]
2020-12-03 12:31:17.671 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_hindi reloaded
2020-12-03 12:31:17.671 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_bengali
2020-12-03 12:31:19.451 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_bengali :109130 missing keys [yJpx_SpM,
0b06UNFk, xIWvtmkD, krextkJu, S6IhfFgC, MbQCj-IA, GZu-zly4, 4O8u3PNM,
WmuhbPZ7, yFqlKKyA]
2020-12-03 12:31:19.451 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_bengali reloaded
2020-12-03 12:31:19.451 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_urdu
2020-12-03 12:31:19.993 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_urdu :33271 missing keys [ocPqghAA,
3E2mc6yo, 49s7qbRF, y7pyiqx-, tG4nP8RA, bnA-25w7, E7BZ91he, vFKCaMmX,
yeZPGUlw, yY--oZQ_]
2020-12-03 12:31:19.993 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_urdu reloaded
2020-12-03 12:31:19.993 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_AE_primary
2020-12-03 12:31:56.126 INFO (qtp2034610694-29862) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_1.replica_n21:QUERY./select.requests&key=solr.core.my_collection.shard2_1.replica_n21:UPDATE./update.requests&key=solr.core.my_collection.shard2_1.replica_n21:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:31:56.128 INFO (qtp2034610694-29998) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:32:56.312 INFO (qtp2034610694-29862) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_1.replica_n21:QUERY./select.requests&key=solr.core.my_collection.shard2_1.replica_n21:UPDATE./update.requests&key=solr.core.my_collection.shard2_1.replica_n21:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:32:56.315 INFO (qtp2034610694-29927) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=1
2020-12-03 12:33:56.479 INFO (qtp2034610694-29862) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_1.replica_n21:QUERY./select.requests&key=solr.core.my_collection.shard2_1.replica_n21:UPDATE./update.requests&key=solr.core.my_collection.shard2_1.replica_n21:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:33:56.481 INFO (qtp2034610694-29927) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:33:59.559 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_AE_primary :10066010 missing keys
[L7AaaN_9, 1IgF5TwC, sint5dtC, 5dzFIeup, JP7SqZsj, 1pjDJpgX, Mzk1ZmRh,
foeGg92o, eRzi1ydZ, sNLdl3gm]
2020-12-03 12:33:59.559 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_AE_primary reloaded
2020-12-03 12:33:59.559 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_punjabi
2020-12-03 12:34:01.555 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_punjabi :118081 missing keys [mo6fMaVh,
tzAwxvtj, 7tpfikGT, XNFeyrdr, tkPFUx_I, aEPNxZIY, jKNnSkuj, lu9JJBnL,
XMWoK3VG, sufV1nuR]
2020-12-03 12:34:01.555 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_punjabi reloaded
2020-12-03 12:34:01.555 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_marathi
2020-12-03 12:34:02.394 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_marathi :49629 missing keys [O4tzIeqB,
4Ah64rJ5, 9M6B1pMh, SmQJINlD, a8te1FzS, Sl_VpyeP, h5KnrFk7, Mq5G1H7o,
7e0F61HV, HfGP8AD_]
2020-12-03 12:34:02.394 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_marathi reloaded
2020-12-03 12:34:02.394 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_english
2020-12-03 12:34:36.061 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_english :2066976 missing keys [qX0rzqI_,
avPGWOXX, LBZl3cbT, U5GynSQN, muPE6G_Y, cgDvU6uP, bE8hgzKe, tGj-s5DN,
f573XQ9m, ao8BEhG8]
2020-12-03 12:34:36.061 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_english reloaded
2020-12-03 12:34:36.061 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_AC_level
2020-12-03 12:34:56.663 INFO (qtp2034610694-29862) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_1.replica_n21:QUERY./select.requests&key=solr.core.my_collection.shard2_1.replica_n21:UPDATE./update.requests&key=solr.core.my_collection.shard2_1.replica_n21:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:34:56.664 INFO (qtp2034610694-30010) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:35:56.851 INFO (qtp2034610694-29903) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_1.replica_n21:QUERY./select.requests&key=solr.core.my_collection.shard2_1.replica_n21:UPDATE./update.requests&key=solr.core.my_collection.shard2_1.replica_n21:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:35:56.854 INFO (qtp2034610694-29862) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=1
2020-12-03 12:36:57.066 INFO (qtp2034610694-29903) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_1.replica_n21:QUERY./select.requests&key=solr.core.my_collection.shard2_1.replica_n21:UPDATE./update.requests&key=solr.core.my_collection.shard2_1.replica_n21:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:36:57.068 INFO (qtp2034610694-29998) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:37:57.237 INFO (qtp2034610694-29903) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_1.replica_n21:QUERY./select.requests&key=solr.core.my_collection.shard2_1.replica_n21:UPDATE./update.requests&key=solr.core.my_collection.shard2_1.replica_n21:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:37:57.239 INFO (qtp2034610694-29998) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=1
2020-12-03 12:38:57.424 INFO (qtp2034610694-29903) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_1.replica_n21:QUERY./select.requests&key=solr.core.my_collection.shard2_1.replica_n21:UPDATE./update.requests&key=solr.core.my_collection.shard2_1.replica_n21:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:38:57.426 INFO (qtp2034610694-29927) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:39:40.599 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_AC_level :18472171 missing keys
[umI3MWuQ, JvUe6G-_, RFfZKs4w, sOWMUQwa, 32jVSnYH, 8NAodfoK, CHn_juqL,
6Xk6FNWH, ZTc5ODBm, U7QrWY67]
2020-12-03 12:39:40.599 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_AC_level reloaded
2020-12-03 12:39:40.599 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_AF
2020-12-03 12:39:57.604 INFO (qtp2034610694-29903) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_1.replica_n21:QUERY./select.requests&key=solr.core.my_collection.shard2_1.replica_n21:UPDATE./update.requests&key=solr.core.my_collection.shard2_1.replica_n21:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:39:57.606 INFO (qtp2034610694-29927) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:40:57.777 INFO (qtp2034610694-30013) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_1.replica_n21:QUERY./select.requests&key=solr.core.my_collection.shard2_1.replica_n21:UPDATE./update.requests&key=solr.core.my_collection.shard2_1.replica_n21:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:40:57.780 INFO (qtp2034610694-29746) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=1
2020-12-03 12:41:04.051 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_AF :5260351 missing keys [_r6A0GUX,
Htqj5VOu, NOTOQMim, tP8VO0QL, 2O2V8UOY, VxHndrnP, fIqqfPkp, N_gHFlI_,
dsx_kph1, AJFFGJ8j]
2020-12-03 12:41:04.052 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_AF reloaded
2020-12-03 12:41:04.052 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_bhojpuri
2020-12-03 12:41:05.406 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_bhojpuri :81120 missing keys [21JkD4TV,
pSIWYJuE, vMSo9khA, inBIv57N, FBZa5Jyi, Q5XzXs-4, b-ep7NPS, 4L10e47O,
8uq97zzX, syEpVNmJ]
2020-12-03 12:41:05.406 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_bhojpuri reloaded
2020-12-03 12:41:05.406 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_kannada
2020-12-03 12:41:06.425 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_kannada :62621 missing keys [vMrm556n,
2ZmXX6SV, V1IlmJ18, BImn1IMM, AsFLJ3Yn, c-yhRYDT, qOJerBJq, 7iICrIty,
UMdY9_yd, GLTOLU6m]
2020-12-03 12:41:06.425 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_kannada reloaded
2020-12-03 12:41:06.425 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_gujarati
2020-12-03 12:41:07.490 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_gujarati :64756 missing keys [y0IfmBUR,
tVe_aBfK, GsUc51io, 6H2kniND, zN92bAP2, nd69KcB9, GFh2YpQG, Gp11jTLi,
H_Z6Wg5B, 5aO7JJLF]
2020-12-03 12:41:07.490 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_gujarati reloaded
2020-12-03 12:41:07.490 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_odia
2020-12-03 12:41:07.704 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_odia :12449 missing keys [LVsQn_cL,
jDOQ9oWW, xTB2Duf9, 3CGn8vI9, 7BOUDdZJ, JBHr_d8J, iB-GKjwD, d3B8AVwg,
2EgrM24O, SN0J0wDV]
2020-12-03 12:41:07.704 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_odia reloaded
2020-12-03 12:41:07.704 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_tamil
2020-12-03 12:41:09.234 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_tamil :89440 missing keys [drpalXsJ,
Hccjv831, hJXlUq7Y, VQ65X2Jv, UgRr_rA4, GIiZO90Q, ixAqa05O, t2QMsATn,
jpvfnTKC, 1-pHt4E1]
2020-12-03 12:41:09.234 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_tamil reloaded
2020-12-03 12:41:09.234 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_haryanvi
2020-12-03 12:41:09.426 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_haryanvi :10044 missing keys [fLqY7IfH,
SHafvTkd, Kizpgxr1, S7qpm8hO, 7nmN9XB1, eoN7t6XQ, P4xJ6mZ4, vXxCHq6Y,
RYqFrXZk, 3dAvFFTP]
2020-12-03 12:41:09.426 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_haryanvi reloaded
2020-12-03 12:41:09.426 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_rajasthani
2020-12-03 12:41:09.843 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_rajasthani :24314 missing keys [lkA_r3o7,
1dxsfwx1, Q2qOgk5L, n2lJdK3D, V2gr4MU3, CnVRGKwN, g5MLk0P0, imRLTiHe,
R9sMiFZZ, jeNdTUKy]
2020-12-03 12:41:09.843 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_rajasthani reloaded
2020-12-03 12:41:09.843 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_AC_ADs
2020-12-03 12:41:10.886 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_AC_ADs :57956 missing keys [7IY_lge_,
MNLWeqBq, -EMNXOHX, F-ji7fTD, 5dS1u4Lk, 3Lh3vyZc, mQrlqOOh, 7eVi_9t8,
HAB12hE9, VdpacKsK]
2020-12-03 12:41:10.886 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_AC_ADs reloaded
2020-12-03 12:41:10.886 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_telugu
2020-12-03 12:41:12.064 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_telugu :68441 missing keys [kYgt06qn,
f63iQ1NV, pMbagei9, qLs554tH, xfzlU5dT, kL9aVqI9, m3R-UYnC, 2KrJysAj,
heKVGbUJ, 6SlIIrml]
2020-12-03 12:41:12.064 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_telugu reloaded
2020-12-03 12:41:12.064 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_manualboost_ACs
2020-12-03 12:41:12.128 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_manualboost_ACs :2987 missing keys
[03nlYmrp, uhvA7VfX, Hupc0M1Y, ardQkLJ6, LoUgZFaE, BFV775k9, L_T7S2tB,
OMjJj4q4, rE2TnBHh, UNTsGnao]
2020-12-03 12:41:12.128 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_manualboost_ACs reloaded
2020-12-03 12:41:12.128 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_AA
2020-12-03 12:41:12.314 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_AA :10433 missing keys [sfSKzJ2I,
hXZLZtJa, wNBMEGde, Xhks2_wg, gCB9jM9t, L-DZdfHe, v4yNfaic, yExPB61v,
Mpusokq5, UVXeGvBe]
2020-12-03 12:41:12.314 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_AA reloaded
2020-12-03 12:41:12.314 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_AB_level
2020-12-03 12:41:37.463 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_AB_level :660960 missing keys [17198479,
10529959, 23263400, 20939560, 11299581, 19841955, 14108851, 22958257,
19203485, 18385364]
2020-12-03 12:41:37.463 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_AB_level reloaded
2020-12-03 12:41:37.463 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_AE
2020-12-03 12:41:57.952 INFO (qtp2034610694-30013) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_1.replica_n21:QUERY./select.requests&key=solr.core.my_collection.shard2_1.replica_n21:UPDATE./update.requests&key=solr.core.my_collection.shard2_1.replica_n21:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:41:57.955 INFO (qtp2034610694-29746) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:42:58.123 INFO (qtp2034610694-30013) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_1.replica_n21:QUERY./select.requests&key=solr.core.my_collection.shard2_1.replica_n21:UPDATE./update.requests&key=solr.core.my_collection.shard2_1.replica_n21:INDEX.sizeInBytes}
status=0 QTime=2
2020-12-03 12:42:58.125 INFO (qtp2034610694-29998) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:43:58.290 INFO (qtp2034610694-30013) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_1.replica_n21:QUERY./select.requests&key=solr.core.my_collection.shard2_1.replica_n21:UPDATE./update.requests&key=solr.core.my_collection.shard2_1.replica_n21:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:43:58.292 INFO (qtp2034610694-29998) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:44:31.849 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_AE :11090597 missing keys [gxyKPooc,
_r6A0GUX, YjYyMTcy, Htqj5VOu, bZ1v3yJW, 7uvcshv6, E6V-Aiwk, NOTOQMim,
qF0KRB7q, tP8VO0QL]
2020-12-03 12:44:31.849 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_AE reloaded
2020-12-03 12:44:31.850 INFO
(searcherExecutor-38-thread-1-processing-n:solr_132:8389_solr
x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1
r:core_node22) [c:my_collection s:shard2_1 r:core_node22
x:my_collection_shard2_1_replica_n21] o.a.s.c.SolrCore
[my_collection_shard2_1_replica_n21] Registered new searcher
Searcher@63f636ac[my_collection_shard2_1_replica_n21]
main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_21(8.5.2):C3309082/111149:[diagnostics={os=Linux,
java.version=1.8.0_262, os.arch=amd64, java.runtime.version=1.8.0_262-b10,
source=merge, os.version=3.10.0-1062.9.1.el7.x86_64, java.vendor=Oracle
Corporation, java.vm.version=25.262-b10, lucene.version=8.5.2,
mergeMaxNumSegments=-1, mergeFactor=5,
timestamp=1606789802191}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]:delGen=75)
Uninverting(_3m(8.5.2):C3823194/59618:[diagnostics={os=Linux,
java.version=1.8.0_262, os.arch=amd64, java.runtime.version=1.8.0_262-b10,
source=merge, os.version=3.10.0-1062.9.1.el7.x86_64, java.vendor=Oracle
Corporation, java.vm.version=25.262-b10, lucene.version=8.5.2,
mergeMaxNumSegments=-1, mergeFactor=5,
timestamp=1606840176762}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]:delGen=57)
Uninverting(_4w(8.5.2):C3694758/47663:[diagnostics={os=Linux,
java.version=1.8.0_262, os.arch=amd64, java.runtime.version=1.8.0_262-b10,
source=merge, os.version=3.10.0-1062.9.1.el7.x86_64, java.vendor=Oracle
Corporation, java.vm.version=25.262-b10, lucene.version=8.5.2,
mergeMaxNumSegments=-1, mergeFactor=5,
timestamp=1606874402977}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]:delGen=42)
Uninverting(_4x(8.5.2):C659207/6680:[diagnostics={os=Linux,
java.version=1.8.0_262, os.arch=amd64, java.runtime.version=1.8.0_262-b10,
source=merge, os.version=3.10.0-1062.9.1.el7.x86_64, java.vendor=Oracle
Corporation, java.vm.version=25.262-b10, lucene.version=8.5.2,
mergeMaxNumSegments=-1, mergeFactor=5,
timestamp=1606874971154}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]:delGen=42)
Uninverting(_65(8.5.2):C149509/12553:[diagnostics={os=Linux,
java.version=1.8.0_262, os.arch=amd64, java.runtime.version=1.8.0_262-b10,
source=merge, os.version=3.10.0-1062.9.1.el7.x86_64, java.vendor=Oracle
Corporation, java.vm.version=25.262-b10, lucene.version=8.5.2,
mergeMaxNumSegments=-1, mergeFactor=5,
timestamp=1606951803156}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]:delGen=17)
Uninverting(_5f(8.5.2):C751946/31228:[diagnostics={os=Linux,
java.version=1.8.0_262, os.arch=amd64, java.runtime.version=1.8.0_262-b10,
source=merge, os.version=3.10.0-1062.9.1.el7.x86_64, java.vendor=Oracle
Corporation, java.vm.version=25.262-b10, lucene.version=8.5.2,
mergeMaxNumSegments=-1, mergeFactor=5,
timestamp=1606892402466}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]:delGen=36)
Uninverting(_5a(8.5.2):C756138/17992:[diagnostics={os=Linux,
java.version=1.8.0_262, os.arch=amd64, java.runtime.version=1.8.0_262-b10,
source=merge, os.version=3.10.0-1062.9.1.el7.x86_64, java.vendor=Oracle
Corporation, java.vm.version=25.262-b10, lucene.version=8.5.2,
mergeMaxNumSegments=-1, mergeFactor=5,
timestamp=1606888776244}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]:delGen=37)
Uninverting(_6a(8.5.2):C216064/23372:[diagnostics={os=Linux,
java.version=1.8.0_262, os.arch=amd64, java.runtime.version=1.8.0_262-b10,
source=merge, os.version=3.10.0-1062.9.1.el7.x86_64, java.vendor=Oracle
Corporation, java.vm.version=25.262-b10, lucene.version=8.5.2,
mergeMaxNumSegments=-1, mergeFactor=5,
timestamp=1606960803148}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]:delGen=12)
Uninverting(_5g(8.5.2):C151901/3797:[diagnostics={os=Linux,
java.vendor=Oracle Corporation, java.version=1.8.0_262,
java.vm.version=25.262-b10, lucene.version=8.5.2, os.arch=amd64,
java.runtime.version=1.8.0_262-b10, source=flush,
os.version=3.10.0-1062.9.1.el7.x86_64,
timestamp=1606894147143}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]:delGen=33)
Uninverting(_5j(8.5.2):C159053/441:[diagnostics={os=Linux,
java.vendor=Oracle Corporation, java.version=1.8.0_262,
java.vm.version=25.262-b10, lucene.version=8.5.2, os.arch=amd64,
java.runtime.version=1.8.0_262-b10, source=flush,
os.version=3.10.0-1062.9.1.el7.x86_64,
timestamp=1606896047339}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]:delGen=31)
Uninverting(_6k(8.5.2):C109394/2508:[diagnostics={os=Linux,
java.version=1.8.0_262, os.arch=amd64, java.runtime.version=1.8.0_262-b10,
source=merge, os.version=3.10.0-1062.9.1.el7.x86_64, java.vendor=Oracle
Corporation, java.vm.version=25.262-b10, lucene.version=8.5.2,
mergeMaxNumSegments=-1, mergeFactor=5,
timestamp=1606987803450}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]:delGen=4)
Uninverting(_6l(8.5.2):C10459:[diagnostics={os=Linux, java.vendor=Oracle
Corporation, java.version=1.8.0_262, java.vm.version=25.262-b10,
lucene.version=8.5.2, os.arch=amd64, java.runtime.version=1.8.0_262-b10,
source=flush, os.version=3.10.0-1062.9.1.el7.x86_64,
timestamp=1606989602781}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}])
Uninverting(_6m(8.5.2):C3511/262:[diagnostics={os=Linux, java.vendor=Oracle
Corporation, java.version=1.8.0_262, java.vm.version=25.262-b10,
lucene.version=8.5.2, os.arch=amd64, java.runtime.version=1.8.0_262-b10,
source=flush, os.version=3.10.0-1062.9.1.el7.x86_64,
timestamp=1606995002466}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]:delGen=1)
Uninverting(_6n(8.5.2):C19:[diagnostics={os=Linux, java.vendor=Oracle
Corporation, java.version=1.8.0_262, java.vm.version=25.262-b10,
lucene.version=8.5.2, os.arch=amd64, java.runtime.version=1.8.0_262-b10,
source=flush, os.version=3.10.0-1062.9.1.el7.x86_64,
timestamp=1606998557533}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}])))}
2020-12-03 12:44:31.860 WARN (qtp2034610694-29830) [c:my_collection
s:shard2_1 r:core_node22 x:my_collection_shard2_1_replica_n21]
o.a.s.u.p.DistributedZkUpdateProcessor Error sending update to
http://solr_198:8389/solr =>
org.apache.solr.client.solrj.SolrServerException: Timeout occured while
waiting response from server at:
http://solr_198:8389/solr/my_collection_shard3_1_replica_n23/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsolr_132%3A8389%2Fsolr%2Fmy_collection_shard2_1_replica_n21%2F
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:407)
org.apache.solr.client.solrj.SolrServerException: Timeout occured while
waiting response from server at:
http://solr_198:8389/solr/my_collection_shard3_1_replica_n23/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsolr_132%3A8389%2Fsolr%2Fmy_collection_shard2_1_replica_n21%2F
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:407)
~[?:?]
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:753)
~[?:?]
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:369)
~[?:?]
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290)
~[?:?]
at
org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:344)
~[?:?]
at
org.apache.solr.update.SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:333)
~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_262]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[?:1.8.0_262]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_262]
at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
~[metrics-core-4.1.2.jar:4.1.2]
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
~[?:1.8.0_262]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
~[?:1.8.0_262]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_262]
Caused by: java.util.concurrent.TimeoutException
at
org.eclipse.jetty.client.util.InputStreamResponseListener.get(InputStreamResponseListener.java:216)
~[?:?]
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:398)
~[?:?]
... 13 more
2020-12-03 12:44:31.860 WARN (qtp2034610694-29830) [c:my_collection
s:shard2_1 r:core_node22 x:my_collection_shard2_1_replica_n21]
o.a.s.u.p.DistributedZkUpdateProcessor Error sending update to
http://solr_254:8389/solr =>
org.apache.solr.client.solrj.SolrServerException: Timeout occured while
waiting response from server at:
http://solr_254:8389/solr/my_collection_shard2_0_replica_n11/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsolr_132%3A8389%2Fsolr%2Fmy_collection_shard2_1_replica_n21%2F
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:407)
org.apache.solr.client.solrj.SolrServerException: Timeout occured while
waiting response from server at:
http://solr_254:8389/solr/my_collection_shard2_0_replica_n11/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsolr_132%3A8389%2Fsolr%2Fmy_collection_shard2_1_replica_n21%2F
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:407)
~[?:?]
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:753)
~[?:?]
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:369)
~[?:?]
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290)
~[?:?]
at
org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:344)
~[?:?]
at
org.apache.solr.update.SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:333)
~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_262]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[?:1.8.0_262]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_262]
at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
~[metrics-core-4.1.2.jar:4.1.2]
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
~[?:1.8.0_262]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
~[?:1.8.0_262]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_262]
Caused by: java.util.concurrent.TimeoutException
at
org.eclipse.jetty.client.util.InputStreamResponseListener.get(InputStreamResponseListener.java:216)
~[?:?]
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:398)
~[?:?]
... 13 more
2020-12-03 12:44:31.860 WARN (qtp2034610694-29830) [c:my_collection
s:shard2_1 r:core_node22 x:my_collection_shard2_1_replica_n21]
o.a.s.u.p.DistributedZkUpdateProcessor Error sending update to
http://solr_133:8389/solr =>
org.apache.solr.client.solrj.SolrServerException: Timeout occured while
waiting response from server at:
http://solr_133:8389/solr/my_collection_shard3_0_replica_n15/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsolr_132%3A8389%2Fsolr%2Fmy_collection_shard2_1_replica_n21%2F
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:407)
org.apache.solr.client.solrj.SolrServerException: Timeout occured while
waiting response from server at:
http://solr_133:8389/solr/my_collection_shard3_0_replica_n15/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsolr_132%3A8389%2Fsolr%2Fmy_collection_shard2_1_replica_n21%2F
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:407)
~[?:?]
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:753)
~[?:?]
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:369)
~[?:?]
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290)
~[?:?]
at
org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:344)
~[?:?]
at
org.apache.solr.update.SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:333)
~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_262]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[?:1.8.0_262]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_262]
at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
~[metrics-core-4.1.2.jar:4.1.2]
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
~[?:1.8.0_262]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
~[?:1.8.0_262]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_262]
Caused by: java.util.concurrent.TimeoutException
at
org.eclipse.jetty.client.util.InputStreamResponseListener.get(InputStreamResponseListener.java:216)
~[?:?]
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:398)
~[?:?]
... 13 more
2020-12-03 12:44:31.860 WARN (qtp2034610694-29830) [c:my_collection
s:shard2_1 r:core_node22 x:my_collection_shard2_1_replica_n21]
o.a.s.u.p.DistributedZkUpdateProcessor Error sending update to
http://solr_200:8389/solr =>
org.apache.solr.client.solrj.SolrServerException: Timeout occured while
waiting response from server at:
http://solr_200:8389/solr/my_collection_shard1_1_replica_n19/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsolr_132%3A8389%2Fsolr%2Fmy_collection_shard2_1_replica_n21%2F
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:407)
org.apache.solr.client.solrj.SolrServerException: Timeout occured while
waiting response from server at:
http://solr_200:8389/solr/my_collection_shard1_1_replica_n19/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsolr_132%3A8389%2Fsolr%2Fmy_collection_shard2_1_replica_n21%2F
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:407)
~[?:?]
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:753)
~[?:?]
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:369)
~[?:?]
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290)
~[?:?]
at
org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:344)
~[?:?]
at
org.apache.solr.update.SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:333)
~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_262]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[?:1.8.0_262]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_262]
at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
~[metrics-core-4.1.2.jar:4.1.2]
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
~[?:1.8.0_262]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
~[?:1.8.0_262]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_262]
Caused by: java.util.concurrent.TimeoutException
at
org.eclipse.jetty.client.util.InputStreamResponseListener.get(InputStreamResponseListener.java:216)
~[?:?]
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:398)
~[?:?]
... 13 more
2020-12-03 12:44:31.860 WARN (qtp2034610694-29830) [c:my_collection
s:shard2_1 r:core_node22 x:my_collection_shard2_1_replica_n21]
o.a.s.u.p.DistributedZkUpdateProcessor Error sending update to
http://solr_199:8389/solr =>
org.apache.solr.client.solrj.SolrServerException: Timeout occured while
waiting response from server at:
http://solr_199:8389/solr/my_collection_shard1_0_replica_n7/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsolr_132%3A8389%2Fsolr%2Fmy_collection_shard2_1_replica_n21%2F
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:407)
org.apache.solr.client.solrj.SolrServerException: Timeout occured while
waiting response from server at:
http://solr_199:8389/solr/my_collection_shard1_0_replica_n7/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsolr_132%3A8389%2Fsolr%2Fmy_collection_shard2_1_replica_n21%2F
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:407)
~[?:?]
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:753)
~[?:?]
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:369)
~[?:?]
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290)
~[?:?]
at
org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:344)
~[?:?]
at
org.apache.solr.update.SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:333)
~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_262]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[?:1.8.0_262]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_262]
at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
~[metrics-core-4.1.2.jar:4.1.2]
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
~[?:1.8.0_262]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
~[?:1.8.0_262]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_262]
Caused by: java.util.concurrent.TimeoutException
at
org.eclipse.jetty.client.util.InputStreamResponseListener.get(InputStreamResponseListener.java:216)
~[?:?]
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:398)
~[?:?]
... 13 more
2020-12-03 12:44:31.860 INFO (qtp2034610694-29830) [c:my_collection
s:shard2_1 r:core_node22 x:my_collection_shard2_1_replica_n21]
o.a.s.u.p.LogUpdateProcessorFactory [my_collection_shard2_1_replica_n21]
webapp=/solr path=/update
params={openSearcher=true&commit=true&_forwardedCount=1&wt=json}{commit=} 0
1514373
2020-12-03 12:44:58.814 INFO (qtp2034610694-30013) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_1.replica_n21:QUERY./select.requests&key=solr.core.my_collection.shard2_1.replica_n21:UPDATE./update.requests&key=solr.core.my_collection.shard2_1.replica_n21:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:44:58.817 INFO (qtp2034610694-29998) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=1
2020-12-03 12:45:59.151 INFO (qtp2034610694-29925) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_1.replica_n21:QUERY./select.requests&key=solr.core.my_collection.shard2_1.replica_n21:UPDATE./update.requests&key=solr.core.my_collection.shard2_1.replica_n21:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:45:59.153 INFO (qtp2034610694-30013) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:46:59.326 INFO (qtp2034610694-29925) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_1.replica_n21:QUERY./select.requests&key=solr.core.my_collection.shard2_1.replica_n21:UPDATE./update.requests&key=solr.core.my_collection.shard2_1.replica_n21:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:46:59.329 INFO (qtp2034610694-29903) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=1
2020-12-03 12:47:59.588 INFO (qtp2034610694-29925) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_1.replica_n21:QUERY./select.requests&key=solr.core.my_collection.shard2_1.replica_n21:UPDATE./update.requests&key=solr.core.my_collection.shard2_1.replica_n21:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:47:59.590 INFO (qtp2034610694-29903) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:48:58.875 INFO (qtp2034610694-30010) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system
params={wt=javabin&version=2&_=1606996615357} status=0 QTime=6
2020-12-03 12:48:58.883 INFO (qtp2034610694-29927) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={prefix=CONTAINER.fs,org.eclipse.jetty.server.handler.DefaultHandler.get-requests,INDEX.sizeInBytes,SEARCHER.searcher.numDocs,SEARCHER.searcher.deletedDocs,SEARCHER.searcher.warmupTime&wt=javabin&version=2&_=1606996616949}
status=0 QTime=3
2020-12-03 12:48:59.757 INFO (qtp2034610694-29925) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_1.replica_n21:QUERY./select.requests&key=solr.core.my_collection.shard2_1.replica_n21:UPDATE./update.requests&key=solr.core.my_collection.shard2_1.replica_n21:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:48:59.759 INFO (qtp2034610694-29862) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:49:06.900 INFO (qtp2034610694-29925) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={prefix=CONTAINER.fs,org.eclipse.jetty.server.handler.DefaultHandler.get-requests,INDEX.sizeInBytes,SEARCHER.searcher.numDocs,SEARCHER.searcher.deletedDocs,SEARCHER.searcher.warmupTime&wt=javabin&version=2&_=1606999746517}
status=0 QTime=4
2020-12-03 12:49:06.902 INFO (qtp2034610694-29998) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/system
params={wt=javabin&version=2&_=1606999746317} status=0 QTime=6
2020-12-03 12:49:59.929 INFO (qtp2034610694-29998) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard2_1.replica_n21:QUERY./select.requests&key=solr.core.my_collection.shard2_1.replica_n21:UPDATE./update.requests&key=solr.core.my_collection.shard2_1.replica_n21:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:49:59.930 INFO (qtp2034610694-29862) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
@#@###### => solr_133
2020-12-03 12:15:51.274 INFO (qtp1252204540-29592) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:16:51.459 INFO (qtp1252204540-29644) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard3_0.replica_n15:QUERY./select.requests&key=solr.core.my_collection.shard3_0.replica_n15:UPDATE./update.requests&key=solr.core.my_collection.shard3_0.replica_n15:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:16:51.463 INFO (qtp1252204540-29592) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:17:51.694 INFO (qtp1252204540-29644) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard3_0.replica_n15:QUERY./select.requests&key=solr.core.my_collection.shard3_0.replica_n15:UPDATE./update.requests&key=solr.core.my_collection.shard3_0.replica_n15:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:17:51.697 INFO (qtp1252204540-10752) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=1
2020-12-03 12:18:51.880 INFO (qtp1252204540-29644) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard3_0.replica_n15:QUERY./select.requests&key=solr.core.my_collection.shard3_0.replica_n15:UPDATE./update.requests&key=solr.core.my_collection.shard3_0.replica_n15:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:18:51.881 INFO (qtp1252204540-10752) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:19:17.525 INFO (qtp1252204540-10752) [c:my_collection
s:shard3_0 r:core_node17 x:my_collection_shard3_0_replica_n15]
o.a.s.u.DirectUpdateHandler2 start
commit{_version_=1685059490309734400,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
2020-12-03 12:19:17.525 INFO (qtp1252204540-10752) [c:my_collection
s:shard3_0 r:core_node17 x:my_collection_shard3_0_replica_n15]
o.a.s.u.SolrIndexWriter Calling setCommitData with
IW:org.apache.solr.update.SolrIndexWriter@6d8d2b0d
commitCommandVersion:1685059490309734400
2020-12-03 12:19:17.734 INFO (qtp1252204540-10752) [c:my_collection
s:shard3_0 r:core_node17 x:my_collection_shard3_0_replica_n15]
o.a.s.s.SolrIndexSearcher Opening
[Searcher@252a086a[my_collection_shard3_0_replica_n15] main]
2020-12-03 12:19:17.736 INFO (qtp1252204540-10752) [c:my_collection
s:shard3_0 r:core_node17 x:my_collection_shard3_0_replica_n15]
o.a.s.u.DirectUpdateHandler2 end_commit_flush
2020-12-03 12:19:52.085 INFO (qtp1252204540-29644) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard3_0.replica_n15:QUERY./select.requests&key=solr.core.my_collection.shard3_0.replica_n15:UPDATE./update.requests&key=solr.core.my_collection.shard3_0.replica_n15:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:19:52.087 INFO (qtp1252204540-29611) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:20:25.201 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_AF :5263837 missing keys [WWKeNEOp,
P9hl8bGD, _r6A0GUX, Htqj5VOu, tP8VO0QL, 2O2V8UOY, VxHndrnP, N_gHFlI_,
54yONW2l, dsx_kph1]
2020-12-03 12:20:46.452 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_AB_level :660515 missing keys [23647754,
10529959, 23263400, 11299581, 19841955, 19203485, 2934559, 18221208,
19273969, 22470852]
2020-12-03 12:20:46.503 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_manualboost_ACs :3022 missing keys
[blXh9_dw, 03nlYmrp, uhvA7VfX, Hupc0M1Y, fqvfSzuB, ardQkLJ6, L_T7S2tB,
OMjJj4q4, lKBa6juj, rE2TnBHh]
2020-12-03 12:20:48.553 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_malayalam
2020-12-03 12:20:49.270 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_malayalam :52011 missing keys [cZn2f6Xs,
4Z6lW4Li, 5RJuPUK4, flW9TLCW, A22JCYnw, e6TYULIH, C5n2jwht, a-SVfjW9,
8kQo--Vj, iaqOLujH]
2020-12-03 12:20:49.270 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_malayalam reloaded
2020-12-03 12:20:49.270 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_hindi
2020-12-03 12:20:52.281 INFO (qtp1252204540-29644) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard3_0.replica_n15:QUERY./select.requests&key=solr.core.my_collection.shard3_0.replica_n15:UPDATE./update.requests&key=solr.core.my_collection.shard3_0.replica_n15:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:20:52.284 INFO (qtp1252204540-29611) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=1
2020-12-03 12:20:53.427 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_hindi :314507 missing keys [peUFCam7,
ynaCbJxu, Z7zDcZFn, Tv_G7PFw, L-Lqsq6X, 9ly_5w8m, jPKgJcWj, aL-ArcIE,
Nb7Equ17, XFYVtslF]
2020-12-03 12:20:53.427 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_hindi reloaded
2020-12-03 12:20:53.427 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_bengali
2020-12-03 12:20:54.848 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_bengali :109007 missing keys [yJpx_SpM,
STjLw6br, 0b06UNFk, xIWvtmkD, krextkJu, UhnWkIs1, S6IhfFgC, Ae9Jz0Ah,
RVPzenj_, GZu-zly4]
2020-12-03 12:20:54.848 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_bengali reloaded
2020-12-03 12:20:54.848 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_urdu
2020-12-03 12:20:55.288 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_urdu :33358 missing keys [ocPqghAA,
3E2mc6yo, 6TatU1en, Qjs9S8mY, 49s7qbRF, kQCvEN9Y, y7pyiqx-, m5V4L80Z,
bnA-25w7, E7BZ91he]
2020-12-03 12:20:55.288 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_urdu reloaded
2020-12-03 12:20:55.288 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_AE_primary
2020-12-03 12:21:52.619 INFO (qtp1252204540-29723) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard3_0.replica_n15:QUERY./select.requests&key=solr.core.my_collection.shard3_0.replica_n15:UPDATE./update.requests&key=solr.core.my_collection.shard3_0.replica_n15:INDEX.sizeInBytes}
status=0 QTime=2
2020-12-03 12:21:52.621 INFO (qtp1252204540-29644) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:22:53.116 INFO (qtp1252204540-29723) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard3_0.replica_n15:QUERY./select.requests&key=solr.core.my_collection.shard3_0.replica_n15:UPDATE./update.requests&key=solr.core.my_collection.shard3_0.replica_n15:INDEX.sizeInBytes}
status=0 QTime=2
2020-12-03 12:22:53.119 INFO (qtp1252204540-29644) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=1
2020-12-03 12:23:02.896 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_AE_primary :10079123 missing keys
[DfGJXwII, L7AaaN_9, 1IgF5TwC, sint5dtC, MOzvgwfX, JP7SqZsj, 1pjDJpgX,
zYikBLhg, Mzk1ZmRh, QWNSeS25]
2020-12-03 12:23:02.896 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_AE_primary reloaded
2020-12-03 12:23:02.896 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_punjabi
2020-12-03 12:23:04.445 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_punjabi :118045 missing keys [tzAwxvtj,
7tpfikGT, XNFeyrdr, J7t4TbgR, 0B-BndhI, aEPNxZIY, jKNnSkuj, lu9JJBnL,
XMWoK3VG, sufV1nuR]
2020-12-03 12:23:04.445 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_punjabi reloaded
2020-12-03 12:23:04.445 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_marathi
2020-12-03 12:23:05.115 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_marathi :49494 missing keys [ZuHbnZrs,
O4tzIeqB, 4Ah64rJ5, 9M6B1pMh, ba93lzRA, a8te1FzS, Sl_VpyeP, 8FqvXXSk,
Mq5G1H7o, HfGP8AD_]
2020-12-03 12:23:05.115 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_marathi reloaded
2020-12-03 12:23:05.115 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_english
2020-12-03 12:23:31.725 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_english :2068288 missing keys [qX0rzqI_,
0Y_wd_2f, avPGWOXX, pHSsQA8T, LBZl3cbT, U5GynSQN, cgDvU6uP, gwonomiZ,
tGj-s5DN, f573XQ9m]
2020-12-03 12:23:31.725 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_english reloaded
2020-12-03 12:23:31.725 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_dedup_AC_level
2020-12-03 12:23:53.466 INFO (qtp1252204540-29723) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard3_0.replica_n15:QUERY./select.requests&key=solr.core.my_collection.shard3_0.replica_n15:UPDATE./update.requests&key=solr.core.my_collection.shard3_0.replica_n15:INDEX.sizeInBytes}
status=0 QTime=2
2020-12-03 12:23:53.468 INFO (qtp1252204540-29592) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:24:53.890 INFO (qtp1252204540-29723) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard3_0.replica_n15:QUERY./select.requests&key=solr.core.my_collection.shard3_0.replica_n15:UPDATE./update.requests&key=solr.core.my_collection.shard3_0.replica_n15:INDEX.sizeInBytes}
status=0 QTime=2
2020-12-03 12:24:53.892 INFO (qtp1252204540-29592) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:25:54.214 INFO (qtp1252204540-29723) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard3_0.replica_n15:QUERY./select.requests&key=solr.core.my_collection.shard3_0.replica_n15:UPDATE./update.requests&key=solr.core.my_collection.shard3_0.replica_n15:INDEX.sizeInBytes}
status=0 QTime=2
2020-12-03 12:25:54.215 INFO (qtp1252204540-29644) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:26:54.900 INFO (qtp1252204540-29739) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard3_0.replica_n15:QUERY./select.requests&key=solr.core.my_collection.shard3_0.replica_n15:UPDATE./update.requests&key=solr.core.my_collection.shard3_0.replica_n15:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:26:54.901 INFO (qtp1252204540-29723) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:27:39.954 INFO (qtp1252204540-29723) [c:my_collection
s:shard3_0 r:core_node17 x:my_collection_shard3_0_replica_n15]
o.a.s.c.S.Request [my_collection_shard3_0_replica_n15] webapp=/solr
path=/select
params={mm=100%25&df=AC&distrib=false&fl=pid&fl=score&shards.purpose=16388&start=0&fsv=true&shard.url=http://solr_133:8389/solr/my_collection_shard3_0_replica_n15/&_forwardedCount=1&rows=10&version=2&q=pid:"avPGWOXX"&defType=edismax&omitHeader=false&NOW=1606998459943&isShard=true&boost=0&sow=true&timeAllowed=1000&wt=javabin&_=1606996632415}
hits=0 status=0 QTime=3
2020-12-03 12:27:41.222 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_dedup_AC_level :18473608 missing keys
[umI3MWuQ, JvUe6G-_, 8clNfIsa, sOWMUQwa, DN2BAnDu, M-D-9pAR, CHn_juqL,
hDJR6XaL, 6Xk6FNWH, ZTc5ODBm]
2020-12-03 12:27:41.222 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_dedup_AC_level reloaded
2020-12-03 12:27:41.222 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_AF
2020-12-03 12:27:55.204 INFO (qtp1252204540-29739) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard3_0.replica_n15:QUERY./select.requests&key=solr.core.my_collection.shard3_0.replica_n15:UPDATE./update.requests&key=solr.core.my_collection.shard3_0.replica_n15:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:27:55.206 INFO (qtp1252204540-29723) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:28:50.742 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_AF :5263837 missing keys [WWKeNEOp,
P9hl8bGD, _r6A0GUX, Htqj5VOu, tP8VO0QL, 2O2V8UOY, VxHndrnP, N_gHFlI_,
54yONW2l, dsx_kph1]
2020-12-03 12:28:50.742 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_AF reloaded
2020-12-03 12:28:50.742 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_bhojpuri
2020-12-03 12:28:51.777 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_bhojpuri :81077 missing keys [21JkD4TV,
pSIWYJuE, PQlcukgI, cSn0v29w, vMSo9khA, inBIv57N, FBZa5Jyi, Q5XzXs-4,
b-ep7NPS, 4L10e47O]
2020-12-03 12:28:51.777 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_bhojpuri reloaded
2020-12-03 12:28:51.777 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_kannada
2020-12-03 12:28:52.613 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_kannada :62865 missing keys [vMrm556n,
2ZmXX6SV, BImn1IMM, AsFLJ3Yn, c-yhRYDT, qOJerBJq, 7iICrIty, UMdY9_yd,
GLTOLU6m, J4lm2n8W]
2020-12-03 12:28:52.613 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_kannada reloaded
2020-12-03 12:28:52.613 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_gujarati
2020-12-03 12:28:53.480 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_gujarati :64845 missing keys [z2fNs6SH,
tVe_aBfK, GsUc51io, EeGK1WIO, 6H2kniND, zN92bAP2, nd69KcB9, GFh2YpQG,
H_Z6Wg5B, 5aO7JJLF]
2020-12-03 12:28:53.480 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_gujarati reloaded
2020-12-03 12:28:53.480 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_odia
2020-12-03 12:28:53.657 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_odia :12512 missing keys [LVsQn_cL,
xTB2Duf9, 3CGn8vI9, JBHr_d8J, iB-GKjwD, d3B8AVwg, SN0J0wDV, VBACWd9f,
d1aer4Mr, hiVdpYUl]
2020-12-03 12:28:53.657 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_odia reloaded
2020-12-03 12:28:53.657 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_tamil
2020-12-03 12:28:54.870 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_tamil :89668 missing keys [drpalXsJ,
Hccjv831, hJXlUq7Y, VQ65X2Jv, jCOoIObQ, UgRr_rA4, hj2tmPIx, GIiZO90Q,
ixAqa05O, t2QMsATn]
2020-12-03 12:28:54.870 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_tamil reloaded
2020-12-03 12:28:54.870 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_haryanvi
2020-12-03 12:28:55.024 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_haryanvi :10036 missing keys [fLqY7IfH,
SHafvTkd, Kizpgxr1, S7qpm8hO, _YSKGq5x, 7nmN9XB1, eoN7t6XQ, P4xJ6mZ4,
vXxCHq6Y, CmrcQ2vX]
2020-12-03 12:28:55.024 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_haryanvi reloaded
2020-12-03 12:28:55.024 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_rajasthani
2020-12-03 12:28:55.358 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_rajasthani :24408 missing keys [lkA_r3o7,
Q2qOgk5L, n2lJdK3D, V2gr4MU3, g5MLk0P0, R9sMiFZZ, jeNdTUKy, eShXwDlL,
bi3q0sRi, mu1WYLQ-]
2020-12-03 12:28:55.358 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_rajasthani reloaded
2020-12-03 12:28:55.358 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_AC_ADs
2020-12-03 12:28:55.629 INFO (qtp1252204540-29739) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard3_0.replica_n15:QUERY./select.requests&key=solr.core.my_collection.shard3_0.replica_n15:UPDATE./update.requests&key=solr.core.my_collection.shard3_0.replica_n15:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:28:55.630 INFO (qtp1252204540-29723) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:28:56.108 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_AC_ADs :57713 missing keys [pc7Tr5jO,
7IY_lge_, MNLWeqBq, F-ji7fTD, 5dS1u4Lk, 3Lh3vyZc, mQrlqOOh, 7eVi_9t8,
HAB12hE9, pXR8RWHg]
2020-12-03 12:28:56.108 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_AC_ADs reloaded
2020-12-03 12:28:56.108 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_telugu
2020-12-03 12:28:57.009 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_telugu :68598 missing keys [kYgt06qn,
q6Z_ik8h, AT9Hesvv, f63iQ1NV, qLs554tH, xfzlU5dT, kL9aVqI9, m3R-UYnC,
2KrJysAj, heKVGbUJ]
2020-12-03 12:28:57.009 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_telugu reloaded
2020-12-03 12:28:57.009 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_manualboost_ACs
2020-12-03 12:28:57.059 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_manualboost_ACs :3022 missing keys
[blXh9_dw, 03nlYmrp, uhvA7VfX, Hupc0M1Y, fqvfSzuB, ardQkLJ6, L_T7S2tB,
OMjJj4q4, lKBa6juj, rE2TnBHh]
2020-12-03 12:28:57.059 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_manualboost_ACs reloaded
2020-12-03 12:28:57.059 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_AA
2020-12-03 12:28:57.200 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_AA :10344 missing keys [sfSKzJ2I,
3IX30VNb, hXZLZtJa, wNBMEGde, Xhks2_wg, gCB9jM9t, zcRsVJSX, L-DZdfHe,
v4yNfaic, eIyVP4Yd]
2020-12-03 12:28:57.200 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_AA reloaded
2020-12-03 12:28:57.200 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_AB_level
2020-12-03 12:29:17.531 WARN (qtp1252204540-29606) [ ]
o.e.j.s.HttpChannelState org.eclipse.jetty.io.EofException: Reset
cancel_stream_error
2020-12-03 12:29:18.333 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_AB_level :660515 missing keys [23647754,
10529959, 23263400, 11299581, 19841955, 19203485, 2934559, 18221208,
19273969, 22470852]
2020-12-03 12:29:18.334 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_AB_level reloaded
2020-12-03 12:29:18.334 INFO
(searcherExecutor-117-thread-1-processing-n:solr_133:8389_solr
x:my_collection_shard3_0_replica_n15 c:my_collection s:shard3_0
r:core_node17) [c:my_collection s:shard3_0 r:core_node17
x:my_collection_shard3_0_replica_n15] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_AE
2020-12-03 12:29:55.796 INFO (qtp1252204540-29739) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard3_0.replica_n15:QUERY./select.requests&key=solr.core.my_collection.shard3_0.replica_n15:UPDATE./update.requests&key=solr.core.my_collection.shard3_0.replica_n15:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:29:55.797 INFO (qtp1252204540-29730) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:30:01.789 INFO (qtp1252204540-29730) [c:my_collection
s:shard3_0 r:core_node17 x:my_collection_shard3_0_replica_n15]
o.a.s.u.DirectUpdateHandler2 start
commit{_version_=1685060165869502464,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
2020-12-03 12:30:01.789 INFO (qtp1252204540-29730) [c:my_collection
s:shard3_0 r:core_node17 x:my_collection_shard3_0_replica_n15]
o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
2020-12-03 12:30:01.793 INFO (qtp1252204540-29730) [c:my_collection
s:shard3_0 r:core_node17 x:my_collection_shard3_0_replica_n15]
o.a.s.u.DirectUpdateHandler2 end_commit_flush
2020-12-03 12:30:01.793 INFO (qtp1252204540-29730) [c:my_collection
s:shard3_0 r:core_node17 x:my_collection_shard3_0_replica_n15]
o.a.s.u.p.LogUpdateProcessorFactory [my_collection_shard3_0_replica_n15]
webapp=/solr path=/update
params={update.distrib=TOLEADER&waitSearcher=true&openSearcher=false&commit=true&softCommit=false&distrib.from=http://solr_254:8389/solr/my_collection_shard2_0_replica_n11/&commit_end_point=leaders&wt=javabin&version=2&expungeDeletes=false}{commit=}
0 5
2020-12-03 12:30:55.982 INFO (qtp1252204540-29739) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard3_0.replica_n15:QUERY./select.requests&key=solr.core.my_collection.shard3_0.replica_n15:UPDATE./update.requests&key=solr.core.my_collection.shard3_0.replica_n15:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:30:55.985 INFO (qtp1252204540-29730) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=1
@#@###### => solr_198
2020-12-03 12:15:51.278 INFO (qtp131343682-29931) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard3_1.replica_n23:UPDATE./update.requests&key=solr.core.my_collection.shard3_1.replica_n23:QUERY./select.requests&key=solr.core.my_collection.shard3_1.replica_n23:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:15:51.281 INFO (qtp131343682-29809) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=1
2020-12-03 12:16:51.467 INFO (qtp131343682-4774) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard3_1.replica_n23:UPDATE./update.requests&key=solr.core.my_collection.shard3_1.replica_n23:QUERY./select.requests&key=solr.core.my_collection.shard3_1.replica_n23:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:16:51.468 INFO (qtp131343682-29931) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:17:51.701 INFO (qtp131343682-4774) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard3_1.replica_n23:UPDATE./update.requests&key=solr.core.my_collection.shard3_1.replica_n23:QUERY./select.requests&key=solr.core.my_collection.shard3_1.replica_n23:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:17:51.703 INFO (qtp131343682-29775) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:18:51.885 INFO (qtp131343682-4774) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard3_1.replica_n23:UPDATE./update.requests&key=solr.core.my_collection.shard3_1.replica_n23:QUERY./select.requests&key=solr.core.my_collection.shard3_1.replica_n23:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:18:51.889 INFO (qtp131343682-29775) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:19:17.525 INFO (qtp131343682-29775) [c:my_collection
s:shard3_1 r:core_node24 x:my_collection_shard3_1_replica_n23]
o.a.s.u.DirectUpdateHandler2 start
commit{_version_=1685059490309734400,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
2020-12-03 12:19:17.525 INFO (qtp131343682-29775) [c:my_collection
s:shard3_1 r:core_node24 x:my_collection_shard3_1_replica_n23]
o.a.s.u.SolrIndexWriter Calling setCommitData with
IW:org.apache.solr.update.SolrIndexWriter@9d7d7f9
commitCommandVersion:1685059490309734400
2020-12-03 12:19:17.841 INFO (qtp131343682-29775) [c:my_collection
s:shard3_1 r:core_node24 x:my_collection_shard3_1_replica_n23]
o.a.s.s.SolrIndexSearcher Opening
[Searcher@5de0bf7e[my_collection_shard3_1_replica_n23] main]
2020-12-03 12:19:17.845 INFO (qtp131343682-29775) [c:my_collection
s:shard3_1 r:core_node24 x:my_collection_shard3_1_replica_n23]
o.a.s.u.DirectUpdateHandler2 end_commit_flush
2020-12-03 12:19:52.091 INFO (qtp131343682-4774) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard3_1.replica_n23:UPDATE./update.requests&key=solr.core.my_collection.shard3_1.replica_n23:QUERY./select.requests&key=solr.core.my_collection.shard3_1.replica_n23:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:19:52.094 INFO (qtp131343682-29931) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=1
2020-12-03 12:20:52.287 INFO (qtp131343682-4774) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard3_1.replica_n23:UPDATE./update.requests&key=solr.core.my_collection.shard3_1.replica_n23:QUERY./select.requests&key=solr.core.my_collection.shard3_1.replica_n23:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:20:52.289 INFO (qtp131343682-29931) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:20:55.212 INFO
(searcherExecutor-38-thread-1-processing-n:solr_198:8389_solr
x:my_collection_shard3_1_replica_n23 c:my_collection s:shard3_1
r:core_node24) [c:my_collection s:shard3_1 r:core_node24
x:my_collection_shard3_1_replica_n23] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_AF :5248622 missing keys [WWKeNEOp,
P9hl8bGD, _r6A0GUX, Htqj5VOu, NOTOQMim, tP8VO0QL, 2O2V8UOY, VxHndrnP,
fIqqfPkp, N_gHFlI_]
2020-12-03 12:21:23.891 INFO
(searcherExecutor-38-thread-1-processing-n:solr_198:8389_solr
x:my_collection_shard3_1_replica_n23 c:my_collection s:shard3_1
r:core_node24) [c:my_collection s:shard3_1 r:core_node24
x:my_collection_shard3_1_replica_n23] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_AB_level :661479 missing keys [23647754,
23263400, 20939560, 19841955, 14108851, 19203485, 2934559, 18385364,
18221208, 19273969]
2020-12-03 12:21:23.959 INFO
(searcherExecutor-38-thread-1-processing-n:solr_198:8389_solr
x:my_collection_shard3_1_replica_n23 c:my_collection s:shard3_1
r:core_node24) [c:my_collection s:shard3_1 r:core_node24
x:my_collection_shard3_1_replica_n23] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_manualboost_ACs :3008 missing keys
[blXh9_dw, 03nlYmrp, uhvA7VfX, Hupc0M1Y, fqvfSzuB, LoUgZFaE, BFV775k9,
L_T7S2tB, OMjJj4q4, lKBa6juj]
2020-12-03 12:21:25.964 INFO
(searcherExecutor-38-thread-1-processing-n:solr_198:8389_solr
x:my_collection_shard3_1_replica_n23 c:my_collection s:shard3_1
r:core_node24) [c:my_collection s:shard3_1 r:core_node24
x:my_collection_shard3_1_replica_n23] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_malayalam
2020-12-03 12:21:26.924 INFO
(searcherExecutor-38-thread-1-processing-n:solr_198:8389_solr
x:my_collection_shard3_1_replica_n23 c:my_collection s:shard3_1
r:core_node24) [c:my_collection s:shard3_1 r:core_node24
x:my_collection_shard3_1_replica_n23] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_malayalam :52128 missing keys [cZn2f6Xs,
4Z6lW4Li, flW9TLCW, GNMHIYVb, A22JCYnw, ugLDMeC1, pYxX0twz, C5n2jwht,
EDbM64uJ, a-SVfjW9]
2020-12-03 12:21:26.924 INFO
(searcherExecutor-38-thread-1-processing-n:solr_198:8389_solr
x:my_collection_shard3_1_replica_n23 c:my_collection s:shard3_1
r:core_node24) [c:my_collection s:shard3_1 r:core_node24
x:my_collection_shard3_1_replica_n23] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_malayalam reloaded
2020-12-03 12:21:26.924 INFO
(searcherExecutor-38-thread-1-processing-n:solr_198:8389_solr
x:my_collection_shard3_1_replica_n23 c:my_collection s:shard3_1
r:core_node24) [c:my_collection s:shard3_1 r:core_node24
x:my_collection_shard3_1_replica_n23] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_hindi
2020-12-03 12:21:33.065 INFO
(searcherExecutor-38-thread-1-processing-n:solr_198:8389_solr
x:my_collection_shard3_1_replica_n23 c:my_collection s:shard3_1
r:core_node24) [c:my_collection s:shard3_1 r:core_node24
x:my_collection_shard3_1_replica_n23] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_hindi :313940 missing keys [ynaCbJxu,
Z7zDcZFn, Tv_G7PFw, L-Lqsq6X, jPKgJcWj, aL-ArcIE, XFYVtslF, qOrQRrra,
hpsM3g77, Ql174i-_]
2020-12-03 12:21:33.065 INFO
(searcherExecutor-38-thread-1-processing-n:solr_198:8389_solr
x:my_collection_shard3_1_replica_n23 c:my_collection s:shard3_1
r:core_node24) [c:my_collection s:shard3_1 r:core_node24
x:my_collection_shard3_1_replica_n23] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_hindi reloaded
2020-12-03 12:21:33.065 INFO
(searcherExecutor-38-thread-1-processing-n:solr_198:8389_solr
x:my_collection_shard3_1_replica_n23 c:my_collection s:shard3_1
r:core_node24) [c:my_collection s:shard3_1 r:core_node24
x:my_collection_shard3_1_replica_n23] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_bengali
2020-12-03 12:21:35.173 INFO
(searcherExecutor-38-thread-1-processing-n:solr_198:8389_solr
x:my_collection_shard3_1_replica_n23 c:my_collection s:shard3_1
r:core_node24) [c:my_collection s:shard3_1 r:core_node24
x:my_collection_shard3_1_replica_n23] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_bengali :108958 missing keys [STjLw6br,
0b06UNFk, xIWvtmkD, krextkJu, UhnWkIs1, S6IhfFgC, Ae9Jz0Ah, RVPzenj_,
MbQCj-IA, GZu-zly4]
2020-12-03 12:21:35.173 INFO
(searcherExecutor-38-thread-1-processing-n:solr_198:8389_solr
x:my_collection_shard3_1_replica_n23 c:my_collection s:shard3_1
r:core_node24) [c:my_collection s:shard3_1 r:core_node24
x:my_collection_shard3_1_replica_n23] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_bengali reloaded
2020-12-03 12:21:35.173 INFO
(searcherExecutor-38-thread-1-processing-n:solr_198:8389_solr
x:my_collection_shard3_1_replica_n23 c:my_collection s:shard3_1
r:core_node24) [c:my_collection s:shard3_1 r:core_node24
x:my_collection_shard3_1_replica_n23] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_urdu
2020-12-03 12:21:35.807 INFO
(searcherExecutor-38-thread-1-processing-n:solr_198:8389_solr
x:my_collection_shard3_1_replica_n23 c:my_collection s:shard3_1
r:core_node24) [c:my_collection s:shard3_1 r:core_node24
x:my_collection_shard3_1_replica_n23] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_urdu :33394 missing keys [ocPqghAA,
3E2mc6yo, 6TatU1en, Qjs9S8mY, 49s7qbRF, kQCvEN9Y, y7pyiqx-, tG4nP8RA,
m5V4L80Z, bnA-25w7]
2020-12-03 12:21:35.807 INFO
(searcherExecutor-38-thread-1-processing-n:solr_198:8389_solr
x:my_collection_shard3_1_replica_n23 c:my_collection s:shard3_1
r:core_node24) [c:my_collection s:shard3_1 r:core_node24
x:my_collection_shard3_1_replica_n23] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_urdu reloaded
2020-12-03 12:21:35.807 INFO
(searcherExecutor-38-thread-1-processing-n:solr_198:8389_solr
x:my_collection_shard3_1_replica_n23 c:my_collection s:shard3_1
r:core_node24) [c:my_collection s:shard3_1 r:core_node24
x:my_collection_shard3_1_replica_n23] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_AE_primary
2020-12-03 12:21:52.625 INFO (qtp131343682-29809) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard3_1.replica_n23:UPDATE./update.requests&key=solr.core.my_collection.shard3_1.replica_n23:QUERY./select.requests&key=solr.core.my_collection.shard3_1.replica_n23:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:21:52.628 INFO (qtp131343682-4774) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=1
2020-12-03 12:22:53.123 INFO (qtp131343682-29809) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard3_1.replica_n23:UPDATE./update.requests&key=solr.core.my_collection.shard3_1.replica_n23:QUERY./select.requests&key=solr.core.my_collection.shard3_1.replica_n23:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:22:53.125 INFO (qtp131343682-4774) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:23:53.472 INFO (qtp131343682-29809) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard3_1.replica_n23:UPDATE./update.requests&key=solr.core.my_collection.shard3_1.replica_n23:QUERY./select.requests&key=solr.core.my_collection.shard3_1.replica_n23:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:23:53.475 INFO (qtp131343682-29928) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=2
2020-12-03 12:24:39.443 INFO
(searcherExecutor-38-thread-1-processing-n:solr_198:8389_solr
x:my_collection_shard3_1_replica_n23 c:my_collection s:shard3_1
r:core_node24) [c:my_collection s:shard3_1 r:core_node24
x:my_collection_shard3_1_replica_n23] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_AE_primary :10066093 missing keys
[DfGJXwII, L7AaaN_9, sint5dtC, MOzvgwfX, 5dzFIeup, 1pjDJpgX, zYikBLhg,
Mzk1ZmRh, QWNSeS25, Q109fsDb]
2020-12-03 12:24:39.443 INFO
(searcherExecutor-38-thread-1-processing-n:solr_198:8389_solr
x:my_collection_shard3_1_replica_n23 c:my_collection s:shard3_1
r:core_node24) [c:my_collection s:shard3_1 r:core_node24
x:my_collection_shard3_1_replica_n23] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_AE_primary reloaded
2020-12-03 12:24:39.443 INFO
(searcherExecutor-38-thread-1-processing-n:solr_198:8389_solr
x:my_collection_shard3_1_replica_n23 c:my_collection s:shard3_1
r:core_node24) [c:my_collection s:shard3_1 r:core_node24
x:my_collection_shard3_1_replica_n23] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_punjabi
2020-12-03 12:24:41.813 INFO
(searcherExecutor-38-thread-1-processing-n:solr_198:8389_solr
x:my_collection_shard3_1_replica_n23 c:my_collection s:shard3_1
r:core_node24) [c:my_collection s:shard3_1 r:core_node24
x:my_collection_shard3_1_replica_n23] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_punjabi :117986 missing keys [mo6fMaVh,
tzAwxvtj, 7tpfikGT, XNFeyrdr, tkPFUx_I, J7t4TbgR, 0B-BndhI, aEPNxZIY,
jKNnSkuj, lu9JJBnL]
2020-12-03 12:24:41.814 INFO
(searcherExecutor-38-thread-1-processing-n:solr_198:8389_solr
x:my_collection_shard3_1_replica_n23 c:my_collection s:shard3_1
r:core_node24) [c:my_collection s:shard3_1 r:core_node24
x:my_collection_shard3_1_replica_n23] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_punjabi reloaded
2020-12-03 12:24:41.814 INFO
(searcherExecutor-38-thread-1-processing-n:solr_198:8389_solr
x:my_collection_shard3_1_replica_n23 c:my_collection s:shard3_1
r:core_node24) [c:my_collection s:shard3_1 r:core_node24
x:my_collection_shard3_1_replica_n23] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_marathi
2020-12-03 12:24:42.841 INFO
(searcherExecutor-38-thread-1-processing-n:solr_198:8389_solr
x:my_collection_shard3_1_replica_n23 c:my_collection s:shard3_1
r:core_node24) [c:my_collection s:shard3_1 r:core_node24
x:my_collection_shard3_1_replica_n23] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_marathi :49729 missing keys [ZuHbnZrs,
4Ah64rJ5, 9M6B1pMh, SmQJINlD, ba93lzRA, a8te1FzS, Sl_VpyeP, h5KnrFk7,
8FqvXXSk, Mq5G1H7o]
2020-12-03 12:24:42.841 INFO
(searcherExecutor-38-thread-1-processing-n:solr_198:8389_solr
x:my_collection_shard3_1_replica_n23 c:my_collection s:shard3_1
r:core_node24) [c:my_collection s:shard3_1 r:core_node24
x:my_collection_shard3_1_replica_n23] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_marathi reloaded
2020-12-03 12:24:42.841 INFO
(searcherExecutor-38-thread-1-processing-n:solr_198:8389_solr
x:my_collection_shard3_1_replica_n23 c:my_collection s:shard3_1
r:core_node24) [c:my_collection s:shard3_1 r:core_node24
x:my_collection_shard3_1_replica_n23] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_english
2020-12-03 12:24:53.896 INFO (qtp131343682-29809) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard3_1.replica_n23:UPDATE./update.requests&key=solr.core.my_collection.shard3_1.replica_n23:QUERY./select.requests&key=solr.core.my_collection.shard3_1.replica_n23:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:24:53.897 INFO (qtp131343682-29928) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:25:21.929 INFO
(searcherExecutor-38-thread-1-processing-n:solr_198:8389_solr
x:my_collection_shard3_1_replica_n23 c:my_collection s:shard3_1
r:core_node24) [c:my_collection s:shard3_1 r:core_node24
x:my_collection_shard3_1_replica_n23] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_english :2063957 missing keys [qX0rzqI_,
0Y_wd_2f, avPGWOXX, pHSsQA8T, LBZl3cbT, muPE6G_Y, bE8hgzKe, gwonomiZ,
f573XQ9m, ao8BEhG8]
2020-12-03 12:25:21.929 INFO
(searcherExecutor-38-thread-1-processing-n:solr_198:8389_solr
x:my_collection_shard3_1_replica_n23 c:my_collection s:shard3_1
r:core_node24) [c:my_collection s:shard3_1 r:core_node24
x:my_collection_shard3_1_replica_n23] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_english reloaded
2020-12-03 12:25:21.929 INFO
(searcherExecutor-38-thread-1-processing-n:solr_198:8389_solr
x:my_collection_shard3_1_replica_n23 c:my_collection s:shard3_1
r:core_node24) [c:my_collection s:shard3_1 r:core_node24
x:my_collection_shard3_1_replica_n23] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_dedup_AC_level
2020-12-03 12:25:54.219 INFO (qtp131343682-29809) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard3_1.replica_n23:UPDATE./update.requests&key=solr.core.my_collection.shard3_1.replica_n23:QUERY./select.requests&key=solr.core.my_collection.shard3_1.replica_n23:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:25:54.222 INFO (qtp131343682-4774) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:26:54.907 INFO (qtp131343682-29946) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard3_1.replica_n23:UPDATE./update.requests&key=solr.core.my_collection.shard3_1.replica_n23:QUERY./select.requests&key=solr.core.my_collection.shard3_1.replica_n23:INDEX.sizeInBytes}
status=0 QTime=2
2020-12-03 12:26:54.910 INFO (qtp131343682-29809) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=2
2020-12-03 12:27:39.954 INFO (qtp131343682-29809) [c:my_collection
s:shard3_1 r:core_node24 x:my_collection_shard3_1_replica_n23]
o.a.s.c.S.Request [my_collection_shard3_1_replica_n23] webapp=/solr
path=/select
params={mm=100%25&df=AC&distrib=false&fl=pid&fl=score&shards.purpose=16388&start=0&fsv=true&shard.url=http://solr_198:8389/solr/my_collection_shard3_1_replica_n23/&_forwardedCount=1&rows=10&version=2&q=pid:"avPGWOXX"&defType=edismax&omitHeader=false&NOW=1606998459943&isShard=true&boost=0&sow=true&timeAllowed=1000&wt=javabin&_=1606996632415}
hits=0 status=0 QTime=3
2020-12-03 12:27:55.211 INFO (qtp131343682-29946) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard3_1.replica_n23:UPDATE./update.requests&key=solr.core.my_collection.shard3_1.replica_n23:QUERY./select.requests&key=solr.core.my_collection.shard3_1.replica_n23:INDEX.sizeInBytes}
status=0 QTime=2
2020-12-03 12:27:55.213 INFO (qtp131343682-29809) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:28:55.635 INFO (qtp131343682-29946) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard3_1.replica_n23:UPDATE./update.requests&key=solr.core.my_collection.shard3_1.replica_n23:QUERY./select.requests&key=solr.core.my_collection.shard3_1.replica_n23:INDEX.sizeInBytes}
status=0 QTime=2
2020-12-03 12:28:55.640 INFO (qtp131343682-29809) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:29:17.530 WARN (qtp131343682-29937) [ ]
o.e.j.s.HttpChannelState org.eclipse.jetty.io.EofException: Reset
cancel_stream_error
2020-12-03 12:29:55.803 INFO (qtp131343682-29946) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard3_1.replica_n23:UPDATE./update.requests&key=solr.core.my_collection.shard3_1.replica_n23:QUERY./select.requests&key=solr.core.my_collection.shard3_1.replica_n23:INDEX.sizeInBytes}
status=0 QTime=2
2020-12-03 12:29:55.805 INFO (qtp131343682-29595) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:30:01.790 INFO (qtp131343682-29595) [c:my_collection
s:shard3_1 r:core_node24 x:my_collection_shard3_1_replica_n23]
o.a.s.u.DirectUpdateHandler2 start
commit{_version_=1685060165870551040,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
2020-12-03 12:30:01.790 INFO (qtp131343682-29595) [c:my_collection
s:shard3_1 r:core_node24 x:my_collection_shard3_1_replica_n23]
o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
2020-12-03 12:30:01.793 INFO (qtp131343682-29595) [c:my_collection
s:shard3_1 r:core_node24 x:my_collection_shard3_1_replica_n23]
o.a.s.u.DirectUpdateHandler2 end_commit_flush
2020-12-03 12:30:01.793 INFO (qtp131343682-29595) [c:my_collection
s:shard3_1 r:core_node24 x:my_collection_shard3_1_replica_n23]
o.a.s.u.p.LogUpdateProcessorFactory [my_collection_shard3_1_replica_n23]
webapp=/solr path=/update
params={update.distrib=TOLEADER&waitSearcher=true&openSearcher=false&commit=true&softCommit=false&distrib.from=http://solr_254:8389/solr/my_collection_shard2_0_replica_n11/&commit_end_point=leaders&wt=javabin&version=2&expungeDeletes=false}{commit=}
0 5
2020-12-03 12:30:55.991 INFO (qtp131343682-29946) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard3_1.replica_n23:UPDATE./update.requests&key=solr.core.my_collection.shard3_1.replica_n23:QUERY./select.requests&key=solr.core.my_collection.shard3_1.replica_n23:INDEX.sizeInBytes}
status=0 QTime=2
@#@###### => solr_199
2020-12-03 12:15:51.286 INFO (qtp1854053968-31528) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard1_0.replica_n7:UPDATE./update.requests&key=solr.core.my_collection.shard1_0.replica_n7:INDEX.sizeInBytes&key=solr.core.my_collection.shard1_0.replica_n7:QUERY./select.requests}
status=0 QTime=1
2020-12-03 12:15:51.288 INFO (qtp1854053968-31624) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=1
2020-12-03 12:16:51.472 INFO (qtp1854053968-31670) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard1_0.replica_n7:UPDATE./update.requests&key=solr.core.my_collection.shard1_0.replica_n7:INDEX.sizeInBytes&key=solr.core.my_collection.shard1_0.replica_n7:QUERY./select.requests}
status=0 QTime=1
2020-12-03 12:16:51.474 INFO (qtp1854053968-31528) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:17:51.708 INFO (qtp1854053968-31670) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard1_0.replica_n7:UPDATE./update.requests&key=solr.core.my_collection.shard1_0.replica_n7:INDEX.sizeInBytes&key=solr.core.my_collection.shard1_0.replica_n7:QUERY./select.requests}
status=0 QTime=1
2020-12-03 12:17:51.711 INFO (qtp1854053968-31808) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=2
2020-12-03 12:18:51.893 INFO (qtp1854053968-31670) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard1_0.replica_n7:UPDATE./update.requests&key=solr.core.my_collection.shard1_0.replica_n7:INDEX.sizeInBytes&key=solr.core.my_collection.shard1_0.replica_n7:QUERY./select.requests}
status=0 QTime=1
2020-12-03 12:18:51.895 INFO (qtp1854053968-31808) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:19:17.525 INFO (qtp1854053968-31808) [c:my_collection
s:shard1_0 r:core_node9 x:my_collection_shard1_0_replica_n7]
o.a.s.u.DirectUpdateHandler2 start
commit{_version_=1685059490309734400,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
2020-12-03 12:19:17.525 INFO (qtp1854053968-31808) [c:my_collection
s:shard1_0 r:core_node9 x:my_collection_shard1_0_replica_n7]
o.a.s.u.SolrIndexWriter Calling setCommitData with
IW:org.apache.solr.update.SolrIndexWriter@65cec364
commitCommandVersion:1685059490309734400
2020-12-03 12:19:17.758 INFO (qtp1854053968-31808) [c:my_collection
s:shard1_0 r:core_node9 x:my_collection_shard1_0_replica_n7]
o.a.s.s.SolrIndexSearcher Opening
[Searcher@1d38a44e[my_collection_shard1_0_replica_n7] main]
2020-12-03 12:19:17.763 INFO (qtp1854053968-31808) [c:my_collection
s:shard1_0 r:core_node9 x:my_collection_shard1_0_replica_n7]
o.a.s.u.DirectUpdateHandler2 end_commit_flush
2020-12-03 12:19:52.099 INFO (qtp1854053968-31670) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard1_0.replica_n7:UPDATE./update.requests&key=solr.core.my_collection.shard1_0.replica_n7:INDEX.sizeInBytes&key=solr.core.my_collection.shard1_0.replica_n7:QUERY./select.requests}
status=0 QTime=2
2020-12-03 12:19:52.101 INFO (qtp1854053968-31627) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:20:52.295 INFO (qtp1854053968-31670) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard1_0.replica_n7:UPDATE./update.requests&key=solr.core.my_collection.shard1_0.replica_n7:INDEX.sizeInBytes&key=solr.core.my_collection.shard1_0.replica_n7:QUERY./select.requests}
status=0 QTime=2
2020-12-03 12:20:52.298 INFO (qtp1854053968-31627) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=2
2020-12-03 12:21:14.115 INFO
(searcherExecutor-117-thread-1-processing-n:solr_199:8389_solr
x:my_collection_shard1_0_replica_n7 c:my_collection s:shard1_0 r:core_node9)
[c:my_collection s:shard1_0 r:core_node9
x:my_collection_shard1_0_replica_n7] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_AF :5202404 missing keys [WWKeNEOp,
P9hl8bGD, _r6A0GUX, NOTOQMim, tP8VO0QL, 2O2V8UOY, VxHndrnP, fIqqfPkp,
N_gHFlI_, 54yONW2l]
2020-12-03 12:21:50.144 INFO
(searcherExecutor-117-thread-1-processing-n:solr_199:8389_solr
x:my_collection_shard1_0_replica_n7 c:my_collection s:shard1_0 r:core_node9)
[c:my_collection s:shard1_0 r:core_node9
x:my_collection_shard1_0_replica_n7] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_AB_level :661930 missing keys [23647754,
10529959, 20939560, 19841955, 14108851, 22958257, 19203485, 2934559,
18385364, 18221208]
2020-12-03 12:21:50.227 INFO
(searcherExecutor-117-thread-1-processing-n:solr_199:8389_solr
x:my_collection_shard1_0_replica_n7 c:my_collection s:shard1_0 r:core_node9)
[c:my_collection s:shard1_0 r:core_node9
x:my_collection_shard1_0_replica_n7] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_manualboost_ACs :3002 missing keys
[blXh9_dw, 03nlYmrp, uhvA7VfX, Hupc0M1Y, fqvfSzuB, ardQkLJ6, LoUgZFaE,
BFV775k9, L_T7S2tB, OMjJj4q4]
2020-12-03 12:21:52.633 INFO (qtp1854053968-31624) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard1_0.replica_n7:UPDATE./update.requests&key=solr.core.my_collection.shard1_0.replica_n7:INDEX.sizeInBytes&key=solr.core.my_collection.shard1_0.replica_n7:QUERY./select.requests}
status=0 QTime=1
2020-12-03 12:21:52.634 INFO (qtp1854053968-31419) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:21:52.697 INFO
(searcherExecutor-117-thread-1-processing-n:solr_199:8389_solr
x:my_collection_shard1_0_replica_n7 c:my_collection s:shard1_0 r:core_node9)
[c:my_collection s:shard1_0 r:core_node9
x:my_collection_shard1_0_replica_n7] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_malayalam
2020-12-03 12:21:53.850 INFO
(searcherExecutor-117-thread-1-processing-n:solr_199:8389_solr
x:my_collection_shard1_0_replica_n7 c:my_collection s:shard1_0 r:core_node9)
[c:my_collection s:shard1_0 r:core_node9
x:my_collection_shard1_0_replica_n7] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_malayalam :52067 missing keys [cZn2f6Xs,
4Z6lW4Li, 5RJuPUK4, flW9TLCW, GNMHIYVb, A22JCYnw, ugLDMeC1, e6TYULIH,
pYxX0twz, C5n2jwht]
2020-12-03 12:21:53.850 INFO
(searcherExecutor-117-thread-1-processing-n:solr_199:8389_solr
x:my_collection_shard1_0_replica_n7 c:my_collection s:shard1_0 r:core_node9)
[c:my_collection s:shard1_0 r:core_node9
x:my_collection_shard1_0_replica_n7] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_malayalam reloaded
2020-12-03 12:21:53.850 INFO
(searcherExecutor-117-thread-1-processing-n:solr_199:8389_solr
x:my_collection_shard1_0_replica_n7 c:my_collection s:shard1_0 r:core_node9)
[c:my_collection s:shard1_0 r:core_node9
x:my_collection_shard1_0_replica_n7] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_hindi
2020-12-03 12:22:00.990 INFO
(searcherExecutor-117-thread-1-processing-n:solr_199:8389_solr
x:my_collection_shard1_0_replica_n7 c:my_collection s:shard1_0 r:core_node9)
[c:my_collection s:shard1_0 r:core_node9
x:my_collection_shard1_0_replica_n7] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_hindi :314622 missing keys [peUFCam7,
Z7zDcZFn, Tv_G7PFw, L-Lqsq6X, 9ly_5w8m, aL-ArcIE, Nb7Equ17, XFYVtslF,
hpsM3g77, Ql174i-_]
2020-12-03 12:22:00.990 INFO
(searcherExecutor-117-thread-1-processing-n:solr_199:8389_solr
x:my_collection_shard1_0_replica_n7 c:my_collection s:shard1_0 r:core_node9)
[c:my_collection s:shard1_0 r:core_node9
x:my_collection_shard1_0_replica_n7] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_hindi reloaded
2020-12-03 12:22:00.990 INFO
(searcherExecutor-117-thread-1-processing-n:solr_199:8389_solr
x:my_collection_shard1_0_replica_n7 c:my_collection s:shard1_0 r:core_node9)
[c:my_collection s:shard1_0 r:core_node9
x:my_collection_shard1_0_replica_n7] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_bengali
2020-12-03 12:22:03.583 INFO
(searcherExecutor-117-thread-1-processing-n:solr_199:8389_solr
x:my_collection_shard1_0_replica_n7 c:my_collection s:shard1_0 r:core_node9)
[c:my_collection s:shard1_0 r:core_node9
x:my_collection_shard1_0_replica_n7] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_bengali :108826 missing keys [yJpx_SpM,
STjLw6br, xIWvtmkD, krextkJu, UhnWkIs1, Ae9Jz0Ah, RVPzenj_, MbQCj-IA,
WmuhbPZ7, O7MD10uY]
2020-12-03 12:22:03.583 INFO
(searcherExecutor-117-thread-1-processing-n:solr_199:8389_solr
x:my_collection_shard1_0_replica_n7 c:my_collection s:shard1_0 r:core_node9)
[c:my_collection s:shard1_0 r:core_node9
x:my_collection_shard1_0_replica_n7] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_bengali reloaded
2020-12-03 12:22:03.583 INFO
(searcherExecutor-117-thread-1-processing-n:solr_199:8389_solr
x:my_collection_shard1_0_replica_n7 c:my_collection s:shard1_0 r:core_node9)
[c:my_collection s:shard1_0 r:core_node9
x:my_collection_shard1_0_replica_n7] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_urdu
2020-12-03 12:22:04.390 INFO
(searcherExecutor-117-thread-1-processing-n:solr_199:8389_solr
x:my_collection_shard1_0_replica_n7 c:my_collection s:shard1_0 r:core_node9)
[c:my_collection s:shard1_0 r:core_node9
x:my_collection_shard1_0_replica_n7] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_urdu :33260 missing keys [3E2mc6yo,
6TatU1en, Qjs9S8mY, 49s7qbRF, kQCvEN9Y, tG4nP8RA, m5V4L80Z, E7BZ91he,
vFKCaMmX, LLfBfArD]
2020-12-03 12:22:04.390 INFO
(searcherExecutor-117-thread-1-processing-n:solr_199:8389_solr
x:my_collection_shard1_0_replica_n7 c:my_collection s:shard1_0 r:core_node9)
[c:my_collection s:shard1_0 r:core_node9
x:my_collection_shard1_0_replica_n7] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_urdu reloaded
2020-12-03 12:22:04.390 INFO
(searcherExecutor-117-thread-1-processing-n:solr_199:8389_solr
x:my_collection_shard1_0_replica_n7 c:my_collection s:shard1_0 r:core_node9)
[c:my_collection s:shard1_0 r:core_node9
x:my_collection_shard1_0_replica_n7] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_AE_primary
2020-12-03 12:22:53.130 INFO (qtp1854053968-31624) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard1_0.replica_n7:UPDATE./update.requests&key=solr.core.my_collection.shard1_0.replica_n7:INDEX.sizeInBytes&key=solr.core.my_collection.shard1_0.replica_n7:QUERY./select.requests}
status=0 QTime=1
2020-12-03 12:22:53.132 INFO (qtp1854053968-31419) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:23:53.479 INFO (qtp1854053968-31624) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard1_0.replica_n7:UPDATE./update.requests&key=solr.core.my_collection.shard1_0.replica_n7:INDEX.sizeInBytes&key=solr.core.my_collection.shard1_0.replica_n7:QUERY./select.requests}
status=0 QTime=1
2020-12-03 12:23:53.482 INFO (qtp1854053968-31528) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=1
2020-12-03 12:24:53.902 INFO (qtp1854053968-31624) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard1_0.replica_n7:UPDATE./update.requests&key=solr.core.my_collection.shard1_0.replica_n7:INDEX.sizeInBytes&key=solr.core.my_collection.shard1_0.replica_n7:QUERY./select.requests}
status=0 QTime=1
2020-12-03 12:24:53.903 INFO (qtp1854053968-31528) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:25:44.725 INFO
(searcherExecutor-117-thread-1-processing-n:solr_199:8389_solr
x:my_collection_shard1_0_replica_n7 c:my_collection s:shard1_0 r:core_node9)
[c:my_collection s:shard1_0 r:core_node9
x:my_collection_shard1_0_replica_n7] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_AE_primary :9880483 missing keys
[DfGJXwII, 1IgF5TwC, sint5dtC, MOzvgwfX, 5dzFIeup, JP7SqZsj, 1pjDJpgX,
zYikBLhg, Mzk1ZmRh, QWNSeS25]
2020-12-03 12:25:44.725 INFO
(searcherExecutor-117-thread-1-processing-n:solr_199:8389_solr
x:my_collection_shard1_0_replica_n7 c:my_collection s:shard1_0 r:core_node9)
[c:my_collection s:shard1_0 r:core_node9
x:my_collection_shard1_0_replica_n7] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_AE_primary reloaded
2020-12-03 12:25:44.725 INFO
(searcherExecutor-117-thread-1-processing-n:solr_199:8389_solr
x:my_collection_shard1_0_replica_n7 c:my_collection s:shard1_0 r:core_node9)
[c:my_collection s:shard1_0 r:core_node9
x:my_collection_shard1_0_replica_n7] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_punjabi
2020-12-03 12:25:47.372 INFO
(searcherExecutor-117-thread-1-processing-n:solr_199:8389_solr
x:my_collection_shard1_0_replica_n7 c:my_collection s:shard1_0 r:core_node9)
[c:my_collection s:shard1_0 r:core_node9
x:my_collection_shard1_0_replica_n7] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_punjabi :118115 missing keys [mo6fMaVh,
tkPFUx_I, J7t4TbgR, 0B-BndhI, lu9JJBnL, XMWoK3VG, sufV1nuR, LgV87KKz,
uz22rZjK, I2Y5nIQa]
2020-12-03 12:25:47.372 INFO
(searcherExecutor-117-thread-1-processing-n:solr_199:8389_solr
x:my_collection_shard1_0_replica_n7 c:my_collection s:shard1_0 r:core_node9)
[c:my_collection s:shard1_0 r:core_node9
x:my_collection_shard1_0_replica_n7] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_punjabi reloaded
2020-12-03 12:25:47.372 INFO
(searcherExecutor-117-thread-1-processing-n:solr_199:8389_solr
x:my_collection_shard1_0_replica_n7 c:my_collection s:shard1_0 r:core_node9)
[c:my_collection s:shard1_0 r:core_node9
x:my_collection_shard1_0_replica_n7] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_marathi
2020-12-03 12:25:48.518 INFO
(searcherExecutor-117-thread-1-processing-n:solr_199:8389_solr
x:my_collection_shard1_0_replica_n7 c:my_collection s:shard1_0 r:core_node9)
[c:my_collection s:shard1_0 r:core_node9
x:my_collection_shard1_0_replica_n7] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_marathi :49691 missing keys [ZuHbnZrs,
O4tzIeqB, 4Ah64rJ5, SmQJINlD, ba93lzRA, Sl_VpyeP, h5KnrFk7, 8FqvXXSk,
7e0F61HV, rg_1qQ2f]
2020-12-03 12:25:48.518 INFO
(searcherExecutor-117-thread-1-processing-n:solr_199:8389_solr
x:my_collection_shard1_0_replica_n7 c:my_collection s:shard1_0 r:core_node9)
[c:my_collection s:shard1_0 r:core_node9
x:my_collection_shard1_0_replica_n7] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_marathi reloaded
2020-12-03 12:25:48.518 INFO
(searcherExecutor-117-thread-1-processing-n:solr_199:8389_solr
x:my_collection_shard1_0_replica_n7 c:my_collection s:shard1_0 r:core_node9)
[c:my_collection s:shard1_0 r:core_node9
x:my_collection_shard1_0_replica_n7] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_english
2020-12-03 12:25:54.226 INFO (qtp1854053968-31624) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard1_0.replica_n7:UPDATE./update.requests&key=solr.core.my_collection.shard1_0.replica_n7:INDEX.sizeInBytes&key=solr.core.my_collection.shard1_0.replica_n7:QUERY./select.requests}
status=0 QTime=1
2020-12-03 12:25:54.229 INFO (qtp1854053968-31419) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=2
2020-12-03 12:26:34.330 INFO
(searcherExecutor-117-thread-1-processing-n:solr_199:8389_solr
x:my_collection_shard1_0_replica_n7 c:my_collection s:shard1_0 r:core_node9)
[c:my_collection s:shard1_0 r:core_node9
x:my_collection_shard1_0_replica_n7] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_english :2052942 missing keys [0Y_wd_2f,
avPGWOXX, pHSsQA8T, U5GynSQN, muPE6G_Y, cgDvU6uP, bE8hgzKe, gwonomiZ,
tGj-s5DN, f573XQ9m]
2020-12-03 12:26:34.330 INFO
(searcherExecutor-117-thread-1-processing-n:solr_199:8389_solr
x:my_collection_shard1_0_replica_n7 c:my_collection s:shard1_0 r:core_node9)
[c:my_collection s:shard1_0 r:core_node9
x:my_collection_shard1_0_replica_n7] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_english reloaded
2020-12-03 12:26:34.330 INFO
(searcherExecutor-117-thread-1-processing-n:solr_199:8389_solr
x:my_collection_shard1_0_replica_n7 c:my_collection s:shard1_0 r:core_node9)
[c:my_collection s:shard1_0 r:core_node9
x:my_collection_shard1_0_replica_n7] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_dedup_AC_level
2020-12-03 12:26:54.916 INFO (qtp1854053968-31677) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard1_0.replica_n7:UPDATE./update.requests&key=solr.core.my_collection.shard1_0.replica_n7:INDEX.sizeInBytes&key=solr.core.my_collection.shard1_0.replica_n7:QUERY./select.requests}
status=0 QTime=2
2020-12-03 12:26:54.923 INFO (qtp1854053968-31624) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:27:39.954 INFO (qtp1854053968-31624) [c:my_collection
s:shard1_0 r:core_node9 x:my_collection_shard1_0_replica_n7]
o.a.s.c.S.Request [my_collection_shard1_0_replica_n7] webapp=/solr
path=/select
params={mm=100%25&df=AC&distrib=false&fl=pid&fl=score&shards.purpose=16388&start=0&fsv=true&shard.url=http://solr_199:8389/solr/my_collection_shard1_0_replica_n7/&_forwardedCount=1&rows=10&version=2&q=pid:"avPGWOXX"&defType=edismax&omitHeader=false&NOW=1606998459943&isShard=true&boost=0&sow=true&timeAllowed=1000&wt=javabin&_=1606996632415}
hits=0 status=0 QTime=3
2020-12-03 12:27:55.217 INFO (qtp1854053968-31677) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard1_0.replica_n7:UPDATE./update.requests&key=solr.core.my_collection.shard1_0.replica_n7:INDEX.sizeInBytes&key=solr.core.my_collection.shard1_0.replica_n7:QUERY./select.requests}
status=0 QTime=2
2020-12-03 12:27:55.219 INFO (qtp1854053968-31624) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:28:55.644 INFO (qtp1854053968-31677) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard1_0.replica_n7:UPDATE./update.requests&key=solr.core.my_collection.shard1_0.replica_n7:INDEX.sizeInBytes&key=solr.core.my_collection.shard1_0.replica_n7:QUERY./select.requests}
status=0 QTime=1
2020-12-03 12:28:55.646 INFO (qtp1854053968-31624) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:29:17.531 WARN (qtp1854053968-31501) [ ]
o.e.j.s.HttpChannelState org.eclipse.jetty.io.EofException: Reset
cancel_stream_error
2020-12-03 12:29:55.809 INFO (qtp1854053968-31677) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard1_0.replica_n7:UPDATE./update.requests&key=solr.core.my_collection.shard1_0.replica_n7:INDEX.sizeInBytes&key=solr.core.my_collection.shard1_0.replica_n7:QUERY./select.requests}
status=0 QTime=1
2020-12-03 12:29:55.813 INFO (qtp1854053968-31670) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=2
2020-12-03 12:30:01.788 INFO (qtp1854053968-31670) [c:my_collection
s:shard1_0 r:core_node9 x:my_collection_shard1_0_replica_n7]
o.a.s.u.DirectUpdateHandler2 start
commit{_version_=1685060165868453888,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
2020-12-03 12:30:01.788 INFO (qtp1854053968-31670) [c:my_collection
s:shard1_0 r:core_node9 x:my_collection_shard1_0_replica_n7]
o.a.s.u.SolrIndexWriter Calling setCommitData with
IW:org.apache.solr.update.SolrIndexWriter@65cec364
commitCommandVersion:1685060165868453888
2020-12-03 12:30:01.842 INFO (qtp1854053968-31670) [c:my_collection
s:shard1_0 r:core_node9 x:my_collection_shard1_0_replica_n7]
o.a.s.s.SolrIndexSearcher Opening
[Searcher@452f8a47[my_collection_shard1_0_replica_n7] realtime]
2020-12-03 12:30:01.842 INFO (qtp1854053968-31670) [c:my_collection
s:shard1_0 r:core_node9 x:my_collection_shard1_0_replica_n7]
o.a.s.u.DirectUpdateHandler2 end_commit_flush
2020-12-03 12:30:01.842 INFO (qtp1854053968-31670) [c:my_collection
s:shard1_0 r:core_node9 x:my_collection_shard1_0_replica_n7]
o.a.s.u.p.LogUpdateProcessorFactory [my_collection_shard1_0_replica_n7]
webapp=/solr path=/update
params={update.distrib=TOLEADER&waitSearcher=true&openSearcher=false&commit=true&softCommit=false&distrib.from=http://solr_254:8389/solr/my_collection_shard2_0_replica_n11/&commit_end_point=leaders&wt=javabin&version=2&expungeDeletes=false}{commit=}
0 55
2020-12-03 12:30:55.998 INFO (qtp1854053968-31677) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard1_0.replica_n7:UPDATE./update.requests&key=solr.core.my_collection.shard1_0.replica_n7:INDEX.sizeInBytes&key=solr.core.my_collection.shard1_0.replica_n7:QUERY./select.requests}
status=0 QTime=1
2020-12-03 12:30:56.000 INFO (qtp1854053968-31670) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
@#@###### => solr_200
2020-12-03 12:15:51.260 INFO (qtp1267082248-29686) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard1_1.replica_n19:QUERY./select.requests&key=solr.core.my_collection.shard1_1.replica_n19:UPDATE./update.requests&key=solr.core.my_collection.shard1_1.replica_n19:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:15:51.262 INFO (qtp1267082248-29900) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=1
2020-12-03 12:16:51.447 INFO (qtp1267082248-29750) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard1_1.replica_n19:QUERY./select.requests&key=solr.core.my_collection.shard1_1.replica_n19:UPDATE./update.requests&key=solr.core.my_collection.shard1_1.replica_n19:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:16:51.449 INFO (qtp1267082248-29686) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:17:51.683 INFO (qtp1267082248-29750) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard1_1.replica_n19:QUERY./select.requests&key=solr.core.my_collection.shard1_1.replica_n19:UPDATE./update.requests&key=solr.core.my_collection.shard1_1.replica_n19:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:17:51.685 INFO (qtp1267082248-29910) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:18:51.867 INFO (qtp1267082248-29750) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard1_1.replica_n19:QUERY./select.requests&key=solr.core.my_collection.shard1_1.replica_n19:UPDATE./update.requests&key=solr.core.my_collection.shard1_1.replica_n19:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:18:51.870 INFO (qtp1267082248-29910) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=1
2020-12-03 12:19:17.525 INFO (qtp1267082248-29910) [c:my_collection
s:shard1_1 r:core_node20 x:my_collection_shard1_1_replica_n19]
o.a.s.u.DirectUpdateHandler2 start
commit{_version_=1685059490309734400,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
2020-12-03 12:19:17.525 INFO (qtp1267082248-29910) [c:my_collection
s:shard1_1 r:core_node20 x:my_collection_shard1_1_replica_n19]
o.a.s.u.SolrIndexWriter Calling setCommitData with
IW:org.apache.solr.update.SolrIndexWriter@5c4629d8
commitCommandVersion:1685059490309734400
2020-12-03 12:19:17.770 INFO (qtp1267082248-29910) [c:my_collection
s:shard1_1 r:core_node20 x:my_collection_shard1_1_replica_n19]
o.a.s.s.SolrIndexSearcher Opening
[Searcher@6dec76a6[my_collection_shard1_1_replica_n19] main]
2020-12-03 12:19:17.775 INFO (qtp1267082248-29910) [c:my_collection
s:shard1_1 r:core_node20 x:my_collection_shard1_1_replica_n19]
o.a.s.u.DirectUpdateHandler2 end_commit_flush
2020-12-03 12:19:52.073 INFO (qtp1267082248-29750) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard1_1.replica_n19:QUERY./select.requests&key=solr.core.my_collection.shard1_1.replica_n19:UPDATE./update.requests&key=solr.core.my_collection.shard1_1.replica_n19:INDEX.sizeInBytes}
status=0 QTime=2
2020-12-03 12:19:52.075 INFO (qtp1267082248-29845) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:20:52.268 INFO (qtp1267082248-29750) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard1_1.replica_n19:QUERY./select.requests&key=solr.core.my_collection.shard1_1.replica_n19:UPDATE./update.requests&key=solr.core.my_collection.shard1_1.replica_n19:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:20:52.271 INFO (qtp1267082248-29845) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=1
2020-12-03 12:21:27.465 INFO
(searcherExecutor-38-thread-1-processing-n:solr_200:8389_solr
x:my_collection_shard1_1_replica_n19 c:my_collection s:shard1_1
r:core_node20) [c:my_collection s:shard1_1 r:core_node20
x:my_collection_shard1_1_replica_n19] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_AF :5214648 missing keys [WWKeNEOp,
P9hl8bGD, _r6A0GUX, Htqj5VOu, NOTOQMim, 2O2V8UOY, VxHndrnP, fIqqfPkp,
N_gHFlI_, 54yONW2l]
2020-12-03 12:21:52.604 INFO (qtp1267082248-29900) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard1_1.replica_n19:QUERY./select.requests&key=solr.core.my_collection.shard1_1.replica_n19:UPDATE./update.requests&key=solr.core.my_collection.shard1_1.replica_n19:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:21:52.606 INFO (qtp1267082248-29750) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:22:06.608 INFO
(searcherExecutor-38-thread-1-processing-n:solr_200:8389_solr
x:my_collection_shard1_1_replica_n19 c:my_collection s:shard1_1
r:core_node20) [c:my_collection s:shard1_1 r:core_node20
x:my_collection_shard1_1_replica_n19] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_AB_level :661947 missing keys [23647754,
10529959, 23263400, 20939560, 11299581, 19841955, 14108851, 22958257,
2934559, 18385364]
2020-12-03 12:22:06.693 INFO
(searcherExecutor-38-thread-1-processing-n:solr_200:8389_solr
x:my_collection_shard1_1_replica_n19 c:my_collection s:shard1_1
r:core_node20) [c:my_collection s:shard1_1 r:core_node20
x:my_collection_shard1_1_replica_n19] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_manualboost_ACs :2986 missing keys
[blXh9_dw, 03nlYmrp, Hupc0M1Y, fqvfSzuB, ardQkLJ6, LoUgZFaE, BFV775k9,
lKBa6juj, rE2TnBHh, UNTsGnao]
2020-12-03 12:22:09.339 INFO
(searcherExecutor-38-thread-1-processing-n:solr_200:8389_solr
x:my_collection_shard1_1_replica_n19 c:my_collection s:shard1_1
r:core_node20) [c:my_collection s:shard1_1 r:core_node20
x:my_collection_shard1_1_replica_n19] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_malayalam
2020-12-03 12:22:10.701 INFO
(searcherExecutor-38-thread-1-processing-n:solr_200:8389_solr
x:my_collection_shard1_1_replica_n19 c:my_collection s:shard1_1
r:core_node20) [c:my_collection s:shard1_1 r:core_node20
x:my_collection_shard1_1_replica_n19] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_malayalam :51924 missing keys [cZn2f6Xs,
4Z6lW4Li, 5RJuPUK4, flW9TLCW, GNMHIYVb, A22JCYnw, ugLDMeC1, e6TYULIH,
pYxX0twz, C5n2jwht]
2020-12-03 12:22:10.701 INFO
(searcherExecutor-38-thread-1-processing-n:solr_200:8389_solr
x:my_collection_shard1_1_replica_n19 c:my_collection s:shard1_1
r:core_node20) [c:my_collection s:shard1_1 r:core_node20
x:my_collection_shard1_1_replica_n19] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_malayalam reloaded
2020-12-03 12:22:10.701 INFO
(searcherExecutor-38-thread-1-processing-n:solr_200:8389_solr
x:my_collection_shard1_1_replica_n19 c:my_collection s:shard1_1
r:core_node20) [c:my_collection s:shard1_1 r:core_node20
x:my_collection_shard1_1_replica_n19] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_hindi
2020-12-03 12:22:18.639 INFO
(searcherExecutor-38-thread-1-processing-n:solr_200:8389_solr
x:my_collection_shard1_1_replica_n19 c:my_collection s:shard1_1
r:core_node20) [c:my_collection s:shard1_1 r:core_node20
x:my_collection_shard1_1_replica_n19] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_hindi :314669 missing keys [peUFCam7,
ynaCbJxu, Tv_G7PFw, 9ly_5w8m, jPKgJcWj, aL-ArcIE, Nb7Equ17, qOrQRrra,
hpsM3g77, Ql174i-_]
2020-12-03 12:22:18.640 INFO
(searcherExecutor-38-thread-1-processing-n:solr_200:8389_solr
x:my_collection_shard1_1_replica_n19 c:my_collection s:shard1_1
r:core_node20) [c:my_collection s:shard1_1 r:core_node20
x:my_collection_shard1_1_replica_n19] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_hindi reloaded
2020-12-03 12:22:18.640 INFO
(searcherExecutor-38-thread-1-processing-n:solr_200:8389_solr
x:my_collection_shard1_1_replica_n19 c:my_collection s:shard1_1
r:core_node20) [c:my_collection s:shard1_1 r:core_node20
x:my_collection_shard1_1_replica_n19] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_bengali
2020-12-03 12:22:21.474 INFO
(searcherExecutor-38-thread-1-processing-n:solr_200:8389_solr
x:my_collection_shard1_1_replica_n19 c:my_collection s:shard1_1
r:core_node20) [c:my_collection s:shard1_1 r:core_node20
x:my_collection_shard1_1_replica_n19] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_bengali :109207 missing keys [yJpx_SpM,
STjLw6br, 0b06UNFk, xIWvtmkD, UhnWkIs1, S6IhfFgC, Ae9Jz0Ah, RVPzenj_,
MbQCj-IA, GZu-zly4]
2020-12-03 12:22:21.474 INFO
(searcherExecutor-38-thread-1-processing-n:solr_200:8389_solr
x:my_collection_shard1_1_replica_n19 c:my_collection s:shard1_1
r:core_node20) [c:my_collection s:shard1_1 r:core_node20
x:my_collection_shard1_1_replica_n19] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_bengali reloaded
2020-12-03 12:22:21.474 INFO
(searcherExecutor-38-thread-1-processing-n:solr_200:8389_solr
x:my_collection_shard1_1_replica_n19 c:my_collection s:shard1_1
r:core_node20) [c:my_collection s:shard1_1 r:core_node20
x:my_collection_shard1_1_replica_n19] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_urdu
2020-12-03 12:22:22.320 INFO
(searcherExecutor-38-thread-1-processing-n:solr_200:8389_solr
x:my_collection_shard1_1_replica_n19 c:my_collection s:shard1_1
r:core_node20) [c:my_collection s:shard1_1 r:core_node20
x:my_collection_shard1_1_replica_n19] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_urdu :33224 missing keys [ocPqghAA,
6TatU1en, Qjs9S8mY, 49s7qbRF, kQCvEN9Y, y7pyiqx-, tG4nP8RA, m5V4L80Z,
bnA-25w7, E7BZ91he]
2020-12-03 12:22:22.320 INFO
(searcherExecutor-38-thread-1-processing-n:solr_200:8389_solr
x:my_collection_shard1_1_replica_n19 c:my_collection s:shard1_1
r:core_node20) [c:my_collection s:shard1_1 r:core_node20
x:my_collection_shard1_1_replica_n19] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_urdu reloaded
2020-12-03 12:22:22.320 INFO
(searcherExecutor-38-thread-1-processing-n:solr_200:8389_solr
x:my_collection_shard1_1_replica_n19 c:my_collection s:shard1_1
r:core_node20) [c:my_collection s:shard1_1 r:core_node20
x:my_collection_shard1_1_replica_n19] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_AE_primary
2020-12-03 12:22:53.104 INFO (qtp1267082248-29900) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard1_1.replica_n19:QUERY./select.requests&key=solr.core.my_collection.shard1_1.replica_n19:UPDATE./update.requests&key=solr.core.my_collection.shard1_1.replica_n19:INDEX.sizeInBytes}
status=0 QTime=2
2020-12-03 12:22:53.106 INFO (qtp1267082248-29750) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:23:53.453 INFO (qtp1267082248-29900) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard1_1.replica_n19:QUERY./select.requests&key=solr.core.my_collection.shard1_1.replica_n19:UPDATE./update.requests&key=solr.core.my_collection.shard1_1.replica_n19:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:23:53.455 INFO (qtp1267082248-29842) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:24:53.877 INFO (qtp1267082248-29900) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard1_1.replica_n19:QUERY./select.requests&key=solr.core.my_collection.shard1_1.replica_n19:UPDATE./update.requests&key=solr.core.my_collection.shard1_1.replica_n19:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:24:53.879 INFO (qtp1267082248-29842) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:25:54.199 INFO (qtp1267082248-29900) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard1_1.replica_n19:QUERY./select.requests&key=solr.core.my_collection.shard1_1.replica_n19:UPDATE./update.requests&key=solr.core.my_collection.shard1_1.replica_n19:INDEX.sizeInBytes}
status=0 QTime=3
2020-12-03 12:25:54.203 INFO (qtp1267082248-29750) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=1
2020-12-03 12:26:19.371 INFO
(searcherExecutor-38-thread-1-processing-n:solr_200:8389_solr
x:my_collection_shard1_1_replica_n19 c:my_collection s:shard1_1
r:core_node20) [c:my_collection s:shard1_1 r:core_node20
x:my_collection_shard1_1_replica_n19] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_AE_primary :9890495 missing keys
[DfGJXwII, L7AaaN_9, 1IgF5TwC, sint5dtC, MOzvgwfX, 5dzFIeup, JP7SqZsj,
1pjDJpgX, zYikBLhg, Mzk1ZmRh]
2020-12-03 12:26:19.371 INFO
(searcherExecutor-38-thread-1-processing-n:solr_200:8389_solr
x:my_collection_shard1_1_replica_n19 c:my_collection s:shard1_1
r:core_node20) [c:my_collection s:shard1_1 r:core_node20
x:my_collection_shard1_1_replica_n19] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_AE_primary reloaded
2020-12-03 12:26:19.371 INFO
(searcherExecutor-38-thread-1-processing-n:solr_200:8389_solr
x:my_collection_shard1_1_replica_n19 c:my_collection s:shard1_1
r:core_node20) [c:my_collection s:shard1_1 r:core_node20
x:my_collection_shard1_1_replica_n19] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_punjabi
2020-12-03 12:26:22.016 INFO
(searcherExecutor-38-thread-1-processing-n:solr_200:8389_solr
x:my_collection_shard1_1_replica_n19 c:my_collection s:shard1_1
r:core_node20) [c:my_collection s:shard1_1 r:core_node20
x:my_collection_shard1_1_replica_n19] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_punjabi :118183 missing keys [mo6fMaVh,
tzAwxvtj, 7tpfikGT, XNFeyrdr, tkPFUx_I, J7t4TbgR, 0B-BndhI, aEPNxZIY,
jKNnSkuj, lu9JJBnL]
2020-12-03 12:26:22.016 INFO
(searcherExecutor-38-thread-1-processing-n:solr_200:8389_solr
x:my_collection_shard1_1_replica_n19 c:my_collection s:shard1_1
r:core_node20) [c:my_collection s:shard1_1 r:core_node20
x:my_collection_shard1_1_replica_n19] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_punjabi reloaded
2020-12-03 12:26:22.016 INFO
(searcherExecutor-38-thread-1-processing-n:solr_200:8389_solr
x:my_collection_shard1_1_replica_n19 c:my_collection s:shard1_1
r:core_node20) [c:my_collection s:shard1_1 r:core_node20
x:my_collection_shard1_1_replica_n19] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_marathi
2020-12-03 12:26:23.151 INFO
(searcherExecutor-38-thread-1-processing-n:solr_200:8389_solr
x:my_collection_shard1_1_replica_n19 c:my_collection s:shard1_1
r:core_node20) [c:my_collection s:shard1_1 r:core_node20
x:my_collection_shard1_1_replica_n19] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_marathi :49647 missing keys [ZuHbnZrs,
O4tzIeqB, 4Ah64rJ5, 9M6B1pMh, SmQJINlD, ba93lzRA, a8te1FzS, Sl_VpyeP,
h5KnrFk7, 8FqvXXSk]
2020-12-03 12:26:23.151 INFO
(searcherExecutor-38-thread-1-processing-n:solr_200:8389_solr
x:my_collection_shard1_1_replica_n19 c:my_collection s:shard1_1
r:core_node20) [c:my_collection s:shard1_1 r:core_node20
x:my_collection_shard1_1_replica_n19] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_marathi reloaded
2020-12-03 12:26:23.151 INFO
(searcherExecutor-38-thread-1-processing-n:solr_200:8389_solr
x:my_collection_shard1_1_replica_n19 c:my_collection s:shard1_1
r:core_node20) [c:my_collection s:shard1_1 r:core_node20
x:my_collection_shard1_1_replica_n19] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_english
2020-12-03 12:26:54.888 INFO (qtp1267082248-29872) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard1_1.replica_n19:QUERY./select.requests&key=solr.core.my_collection.shard1_1.replica_n19:UPDATE./update.requests&key=solr.core.my_collection.shard1_1.replica_n19:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:26:54.889 INFO (qtp1267082248-29900) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:27:08.894 INFO
(searcherExecutor-38-thread-1-processing-n:solr_200:8389_solr
x:my_collection_shard1_1_replica_n19 c:my_collection s:shard1_1
r:core_node20) [c:my_collection s:shard1_1 r:core_node20
x:my_collection_shard1_1_replica_n19] o.a.s.s.f.FileFloatSource Loaded
external value source external_eff_english :2055958 missing keys [qX0rzqI_,
0Y_wd_2f, avPGWOXX, pHSsQA8T, LBZl3cbT, U5GynSQN, muPE6G_Y, cgDvU6uP,
bE8hgzKe, gwonomiZ]
2020-12-03 12:27:08.894 INFO
(searcherExecutor-38-thread-1-processing-n:solr_200:8389_solr
x:my_collection_shard1_1_replica_n19 c:my_collection s:shard1_1
r:core_node20) [c:my_collection s:shard1_1 r:core_node20
x:my_collection_shard1_1_replica_n19] o.a.s.s.f.FileFloatSource
FileFloatSource cache for field eff_english reloaded
2020-12-03 12:27:08.894 INFO
(searcherExecutor-38-thread-1-processing-n:solr_200:8389_solr
x:my_collection_shard1_1_replica_n19 c:my_collection s:shard1_1
r:core_node20) [c:my_collection s:shard1_1 r:core_node20
x:my_collection_shard1_1_replica_n19] o.a.s.s.f.FileFloatSource Refreshing
FileFloatSource cache for field eff_dedup_AC_level
2020-12-03 12:27:39.955 INFO (qtp1267082248-29900) [c:my_collection
s:shard1_1 r:core_node20 x:my_collection_shard1_1_replica_n19]
o.a.s.c.S.Request [my_collection_shard1_1_replica_n19] webapp=/solr
path=/select
params={mm=100%25&df=AC&distrib=false&fl=pid&fl=score&shards.purpose=16388&start=0&fsv=true&shard.url=http://solr_200:8389/solr/my_collection_shard1_1_replica_n19/&_forwardedCount=1&rows=10&version=2&q=pid:"avPGWOXX"&defType=edismax&omitHeader=false&NOW=1606998459943&isShard=true&boost=0&sow=true&timeAllowed=1000&wt=javabin&_=1606996632415}
hits=0 status=0 QTime=4
2020-12-03 12:27:55.190 INFO (qtp1267082248-29872) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard1_1.replica_n19:QUERY./select.requests&key=solr.core.my_collection.shard1_1.replica_n19:UPDATE./update.requests&key=solr.core.my_collection.shard1_1.replica_n19:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:27:55.193 INFO (qtp1267082248-29900) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=1
2020-12-03 12:28:55.617 INFO (qtp1267082248-29872) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard1_1.replica_n19:QUERY./select.requests&key=solr.core.my_collection.shard1_1.replica_n19:UPDATE./update.requests&key=solr.core.my_collection.shard1_1.replica_n19:INDEX.sizeInBytes}
status=0 QTime=2
2020-12-03 12:28:55.618 INFO (qtp1267082248-29900) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:29:17.530 WARN (qtp1267082248-29716) [ ]
o.e.j.s.HttpChannelState org.eclipse.jetty.io.EofException: Reset
cancel_stream_error
2020-12-03 12:29:55.783 INFO (qtp1267082248-29872) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard1_1.replica_n19:QUERY./select.requests&key=solr.core.my_collection.shard1_1.replica_n19:UPDATE./update.requests&key=solr.core.my_collection.shard1_1.replica_n19:INDEX.sizeInBytes}
status=0 QTime=1
2020-12-03 12:29:55.785 INFO (qtp1267082248-29686) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.jvm:os.processCpuLoad&key=solr.node:CONTAINER.fs.coreRoot.usableSpace&key=solr.jvm:os.systemLoadAverage&key=solr.jvm:memory.heap.used}
status=0 QTime=0
2020-12-03 12:30:01.792 INFO (qtp1267082248-29686) [c:my_collection
s:shard1_1 r:core_node20 x:my_collection_shard1_1_replica_n19]
o.a.s.u.DirectUpdateHandler2 start
commit{_version_=1685060165872648192,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
2020-12-03 12:30:01.792 INFO (qtp1267082248-29686) [c:my_collection
s:shard1_1 r:core_node20 x:my_collection_shard1_1_replica_n19]
o.a.s.u.SolrIndexWriter Calling setCommitData with
IW:org.apache.solr.update.SolrIndexWriter@5c4629d8
commitCommandVersion:1685060165872648192
2020-12-03 12:30:01.837 INFO (qtp1267082248-29686) [c:my_collection
s:shard1_1 r:core_node20 x:my_collection_shard1_1_replica_n19]
o.a.s.s.SolrIndexSearcher Opening
[Searcher@1e2364e0[my_collection_shard1_1_replica_n19] realtime]
2020-12-03 12:30:01.838 INFO (qtp1267082248-29686) [c:my_collection
s:shard1_1 r:core_node20 x:my_collection_shard1_1_replica_n19]
o.a.s.u.DirectUpdateHandler2 end_commit_flush
2020-12-03 12:30:01.838 INFO (qtp1267082248-29686) [c:my_collection
s:shard1_1 r:core_node20 x:my_collection_shard1_1_replica_n19]
o.a.s.u.p.LogUpdateProcessorFactory [my_collection_shard1_1_replica_n19]
webapp=/solr path=/update
params={update.distrib=TOLEADER&waitSearcher=true&openSearcher=false&commit=true&softCommit=false&distrib.from=http://solr_254:8389/solr/my_collection_shard2_0_replica_n11/&commit_end_point=leaders&wt=javabin&version=2&expungeDeletes=false}{commit=}
0 51
2020-12-03 12:30:55.971 INFO (qtp1267082248-29872) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics
params={wt=javabin&version=2&key=solr.core.my_collection.shard1_1.replica_n19:QUERY./select.requests&key=solr.core.my_collection.shard1_1.replica_n19:UPDATE./update.requests&key=solr.core.my_collection.shard1_1.replica_n19:INDEX.sizeInBytes}
status=0 QTime=1
--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Commits (with openSearcher = true) are too slow in solr 8
Posted by Erick Erickson <er...@gmail.com>.
I completely agree with Shawn. I’d emphasize that your heap is that large
probably to accommodate badly mis-configured caches.
Why it’s different in 5.4 I don’t quite know, but 10-12
minutes is unacceptable anyway.
My guess is that you made your heaps that large as a consequence of
having low hit rates. If you were using bare NOW in fq clauses,
perhaps you were getting very low hit rates as a result and expanded
the cache size, see:
https://dzone.com/articles/solr-date-math-now-and-filter
At any rate, I _strongly_ recommend that you drop your filterCache
to the default size of 512, and drop your autowarmCount to something
very small, say 16. Ditto for queryResultCache. The documentCache
to maybe 10,000 (autowarm is a no-op for documentCache). Then
drop your heap to something closer to 16G. Then test, tune, test. Do
NOT assume bigger caches are the answer until you have evidence.
Keep reducing your heap size until you start to see GC problems (on
a test system obviously) to get your lower limit. Then add some
back for your production to give you some breathing room.
Finally, see Uwe’s blog:
https://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
to get a sense of why the size on disk is not necessarily a good
indicator of the heap requirements.
Best,
Erick
> On Nov 4, 2020, at 2:40 AM, Shawn Heisey <ap...@elyograg.org> wrote:
>
> On 11/3/2020 11:46 PM, raj.yadav wrote:
>> We have two parallel system one is solr 8.5.2 and other one is solr 5.4
>> In solr_5.4 commit time with opensearcher true is 10 to 12 minutes while in
>> solr_8 it's around 25 minutes.
>
> Commits on a properly configured and sized system should take a few seconds, not minutes. 10 to 12 minutes for a commit is an enormous red flag.
>
>> This is our current caching policy of solr_8
>> <filterCache class="solr.CaffeineCache"
>> size="32768"
>> initialSize="6000"
>> autowarmCount="6000"/>
>
> This is probably the culprit. Do you know how many entries the filterCache actually ends up with? What you've said with this config is "every time I open a new searcher, I'm going to execute up to 6000 queries against the new index." If each query takes one second, running 6000 of them is going to take 100 minutes. I have seen these queries take a lot longer than one second.
>
> Also, each entry in the filterCache can be enormous, depending on the number of docs in the index. Let's say that you have five million documents in your core. With five million documents, each entry in the filterCache is going to be 625000 bytes. That means you need 20GB of heap memory for a full filterCache of 32768 entries -- 20GB of memory above and beyond everything else that Solr requires. Your message doesn't say how many documents you have, it only says the index is 11GB. From that, it is not possible for me to figure out how many documents you have.
>
>> While debugging this we came across this page.
>> https://cwiki.apache.org/confluence/display/SOLR/SolrPerformanceProblems#SolrPerformanceProblems-Slowcommits
>
> I wrote that wiki page.
>
>> Here one of the reasons for slow commit is mentioned as:
>> */`Heap size issues. Problems from the heap being too big will tend to be
>> infrequent, while problems from the heap being too small will tend to happen
>> consistently.`/*
>> Can anyone please help me understand the above point?
>
> If your heap is a lot bigger than it needs to be, then what you'll see is slow garbage collections, but it won't happen very often. If the heap is too small, then there will be garbage collections that happen REALLY often, leaving few system resources for actually running the program. This applies to ANY Java program, not just Solr.
>
>> System config:
>> disk size: 250 GB
>> cpu: (8 vcpus, 64 GiB memory)
>> Index size: 11 GB
>> JVM heap size: 30 GB
>
> That heap seems to be a lot larger than it needs to be. I have run systems with over 100GB of index, with tens of millions of documents, on an 8GB heap. My filterCache on each core had a max size of 64, with an autowarmCount of four ... and commits STILL would take 10 to 15 seconds, which I consider to be very slow. Most of that time was spent executing those four queries in order to autowarm the filterCache.
>
> What I would recommend you start with is reducing the size of the filterCache. Try a size of 128 and an autowarmCount of 8, see what you get for a hit rate on the cache. Adjust from there as necessary. And I would reduce the heap size for Solr as well -- your heap requirements should drop dramatically with a reduced filterCache.
>
> Thanks,
> Shawn
Re: Commits (with openSearcher = true) are too slow in solr 8
Posted by Shawn Heisey <ap...@elyograg.org>.
On 11/3/2020 11:46 PM, raj.yadav wrote:
> We have two parallel system one is solr 8.5.2 and other one is solr 5.4
> In solr_5.4 commit time with opensearcher true is 10 to 12 minutes while in
> solr_8 it's around 25 minutes.
Commits on a properly configured and sized system should take a few
seconds, not minutes. 10 to 12 minutes for a commit is an enormous red
flag.
> This is our current caching policy of solr_8
>
> <filterCache class="solr.CaffeineCache"
> size="32768"
> initialSize="6000"
> autowarmCount="6000"/>
This is probably the culprit. Do you know how many entries the
filterCache actually ends up with? What you've said with this config is
"every time I open a new searcher, I'm going to execute up to 6000
queries against the new index." If each query takes one second, running
6000 of them is going to take 100 minutes. I have seen these queries
take a lot longer than one second.
Also, each entry in the filterCache can be enormous, depending on the
number of docs in the index. Let's say that you have five million
documents in your core. With five million documents, each entry in the
filterCache is going to be 625000 bytes. That means you need 20GB of
heap memory for a full filterCache of 32768 entries -- 20GB of memory
above and beyond everything else that Solr requires. Your message
doesn't say how many documents you have, it only says the index is 11GB.
From that, it is not possible for me to figure out how many documents
you have.
> While debugging this we came across this page.
> https://cwiki.apache.org/confluence/display/SOLR/SolrPerformanceProblems#SolrPerformanceProblems-Slowcommits
I wrote that wiki page.
> Here one of the reasons for slow commit is mentioned as:
> */`Heap size issues. Problems from the heap being too big will tend to be
> infrequent, while problems from the heap being too small will tend to happen
> consistently.`/*
>
> Can anyone please help me understand the above point?
If your heap is a lot bigger than it needs to be, then what you'll see
is slow garbage collections, but it won't happen very often. If the
heap is too small, then there will be garbage collections that happen
REALLY often, leaving few system resources for actually running the
program. This applies to ANY Java program, not just Solr.
> System config:
> disk size: 250 GB
> cpu: (8 vcpus, 64 GiB memory)
> Index size: 11 GB
> JVM heap size: 30 GB
That heap seems to be a lot larger than it needs to be. I have run
systems with over 100GB of index, with tens of millions of documents, on
an 8GB heap. My filterCache on each core had a max size of 64, with an
autowarmCount of four ... and commits STILL would take 10 to 15 seconds,
which I consider to be very slow. Most of that time was spent executing
those four queries in order to autowarm the filterCache.
What I would recommend you start with is reducing the size of the
filterCache. Try a size of 128 and an autowarmCount of 8, see what you
get for a hit rate on the cache. Adjust from there as necessary. And I
would reduce the heap size for Solr as well -- your heap requirements
should drop dramatically with a reduced filterCache.
Thanks,
Shawn