You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Emir Arnautovic <em...@sematext.com> on 2015/12/01 13:45:54 UTC
Re: ZooKeeper nodes die taking down Solr Cluster?
Hi Frank,
Can you please confirm that Solr nodes are aware of entire ZK ensemble?
Can you give more info how it is deployed - ZK on separate servers? What
is load on Solr when it happens? Do you see any errors in Solr logs?
Thanks,
Emir
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
On 30.11.2015 15:42, Kelly, Frank wrote:
> I am somewhat new to SolrCloud and ZooKeeper.
>
> We are deploying ZK and SolrCloud on AWS.
> We are noticing an issue where the one of the three nodes in the ZooKeeper ensemble "drops out" of the ensemble (although the Java process continues to run fine and nothing obviously bad in the ZooKeeper log files).
> And (perhaps its a coincidence) but the Solr cluster then seems to be impacted where Solr nodes appear as "gone" even though their Java process is still running.
>
> When we restart ZooKeeper nodes the Solr Cluster does not recover.
>
> I have ZooKeeper logs and Solr logs - just wondering what else I should capture before posting to this mailing list
>
> Thanks!
>
> -Frank
>
>
> Frank Kelly
> Principal Software Engineer
> Predictive Analytics Team (SCBE/HAC/CDA)
>
>
>
>
> Email: f<ma...@here.com>
> Website: http://www.here.com<http://www.here.com/>
>
>
>
>
> 5 Wayside Rd, Burlington, MA 01803, USA
> Here, a Nokia business
>
>
>
>
>
>
Re: ZooKeeper nodes die taking down Solr Cluster?
Posted by Shawn Heisey <ap...@elyograg.org>.
On 12/1/2015 8:21 AM, Kelly, Frank wrote:
> java.lang.OutOfMemoryError: Java heap space
> 597593838 INFO
> (zkCallback-4-thread-1103-processing-n:52.91.90.134:8983_solr) [ ]
> o.a.s.s.ZkIndexSchemaReader A schema change: WatchedEvent
> state:SyncConnected type:NodeDataChanged
> path:/configs/mycollection/managed-schema, has occurred - updating schema
> from ZooKeeper ...
>
>
> So it looks like it ran out of memory . . . Strange but I thought my
> collections were pretty small.
> Any idea why a replace-field-type call might cause an OutOfMemoryException?
The default heap size on Solr 5.x is 512MB. This is extremely small, it
doesn't take much index data for this amount of memory to be exceeded.
Another message you sent to the list mentions "-m 1g" when starting Solr
... even a gigabyte of RAM might be very small, depending on exactly how
Solr is configured, what is indexed into Solr, and how it is being queried.
Unless you can take steps to make Solr use less memory, you're going to
need to increase the max heap.
Thanks,
Shawn
Re: ZooKeeper nodes die taking down Solr Cluster?
Posted by Emir Arnautovic <em...@sematext.com>.
Hi Frank,
Seems like OOM is reason why Solr failed. What is size of your
collection and heap size. You could do thread and heap dump and see what
is taking memory. Not sure how threads are named but hope that
thread-1103 does not mean there are 1103 threads reading ZK.
Regards,
Emir
On 01.12.2015 16:21, Kelly, Frank wrote:
> Thanks Emir - responses inline below
>
>> Can you please confirm that Solr nodes are aware of entire ZK ensemble?
> Can you explain how I could find that out - I looked into the logs and the
> Admin UI and didn¹t see
> A way to examine if the Solr nodes saw the entire ensemble
Did you provide list of ZK on startup?
>> Can you give more info how it is deployed - ZK on separate servers?
> 3 Zookeeper nodes (separate servers)
> 3 Shards and 3 replicas (again each node on separate servers)
> Deployed in same AWS region and same AZ
>
>> What is load on Solr when it happens?
> Very minimal
> Although it seems to occur around or after the time I do a Curl command to
> update a field or field type in a collection.
> See curl request below
>
>> Do you see any errors in Solr logs?
>
> I see the following in the log file of one of the servers that dropped out
> of the cluster
>
> 597534321 INFO
> (zkCallback-4-thread-1094-processing-n:52.91.90.134:8983_solr) [ ]
> o.a.s.s.ZkIndexSchemaReader A schema change: WatchedEvent
> state:SyncConnected type:NodeDataChanged
> path:/configs/mycollection/managed-schema, has occurred - updating schema
> from ZooKeeper ...
> 597533348 INFO
> (zkCallback-4-thread-1079-processing-n:52.91.90.134:8983_solr) [ ]
> o.a.s.s.ZkIndexSchemaReader Retrieved schema version 89 from ZooKeeper
> 597533348 INFO
> (zkCallback-4-thread-1080-processing-n:52.91.90.134:8983_solr) [ ]
> o.a.s.s.ZkIndexSchemaReader Retrieved schema version 89 from ZooKeeper
> 597574579 INFO
> (zkCallback-4-thread-1080-processing-n:52.91.90.134:8983_solr) [ ]
> o.a.s.s.IndexSchema Reading Solr Schema from
> /configs/mycollection/managed-schema
> 597533348 INFO
> (zkCallback-4-thread-1076-processing-n:52.91.90.134:8983_solr) [ ]
> o.a.s.s.ZkIndexSchemaReader Retrieved schema version 89 from ZooKeeper
> 597574580 INFO
> (zkCallback-4-thread-1076-processing-n:52.91.90.134:8983_solr) [ ]
> o.a.s.s.IndexSchema Reading Solr Schema from
> /configs/mycollection/managed-schema
> 597533348 INFO
> (zkCallback-4-thread-1083-processing-n:52.91.90.134:8983_solr) [ ]
> o.a.s.s.ZkIndexSchemaReader Retrieved schema version 89 from ZooKeeper
> 597574580 INFO
> (zkCallback-4-thread-1083-processing-n:52.91.90.134:8983_solr) [ ]
> o.a.s.s.IndexSchema Reading Solr Schema from
> /configs/mycollection/managed-schema
> 597533348 INFO
> (zkCallback-4-thread-1074-processing-n:52.91.90.134:8983_solr) [ ]
> o.a.s.s.ZkIndexSchemaReader Retrieved schema version 89 from ZooKeeper
> 597533348 INFO
> (zkCallback-4-thread-1087-processing-n:52.91.90.134:8983_solr) [ ]
> o.a.s.s.ZkIndexSchemaReader Retrieved schema version 89 from ZooKeeper
> 597576496 INFO
> (zkCallback-4-thread-1087-processing-n:52.91.90.134:8983_solr) [ ]
> o.a.s.s.IndexSchema Reading Solr Schema from
> /configs/mycollection/managed-schema
> 597533348 INFO
> (zkCallback-4-thread-1077-processing-n:52.91.90.134:8983_solr) [ ]
> o.a.s.s.ZkIndexSchemaReader Retrieved schema version 89 from ZooKeeper
> 597576497 INFO
> (zkCallback-4-thread-1077-processing-n:52.91.90.134:8983_solr) [ ]
> o.a.s.s.IndexSchema Reading Solr Schema from
> /configs/mycollection/managed-schema
> 597533348 INFO
> (zkCallback-4-thread-1089-processing-n:52.91.90.134:8983_solr) [ ]
> o.a.s.s.ZkIndexSchemaReader Retrieved schema version 89 from ZooKeeper
> 597576497 INFO
> (zkCallback-4-thread-1089-processing-n:52.91.90.134:8983_solr) [ ]
> o.a.s.s.IndexSchema Reading Solr Schema from
> /configs/mycollection/managed-schema
> 597533348 INFO
> (zkCallback-4-thread-1078-processing-n:52.91.90.134:8983_solr) [ ]
> o.a.s.s.ZkIndexSchemaReader Retrieved schema version 89 from ZooKeeper
> 597577450 INFO
> (zkCallback-4-thread-1078-processing-n:52.91.90.134:8983_solr) [ ]
> o.a.s.s.IndexSchema Reading Solr Schema from
> /configs/mycollection/managed-schema
> 597533345 INFO
> (zkCallback-4-thread-1093-processing-n:52.91.90.134:8983_solr) [ ]
> o.a.s.s.ZkIndexSchemaReader A schema change: WatchedEvent
> state:SyncConnected type:NodeDataChanged
> path:/configs/mycollection/managed-schema, has occurred - updating schema
> from ZooKeeper ...
> 597533342 INFO
> (zkCallback-4-thread-1081-processing-n:52.91.90.134:8983_solr) [ ]
> o.a.s.s.ZkIndexSchemaReader Retrieved schema version 89 from ZooKeeper
> 597578973 INFO
> (zkCallback-4-thread-1081-processing-n:52.91.90.134:8983_solr) [ ]
> o.a.s.s.IndexSchema Reading Solr Schema from
> /configs/mycollection/managed-schema
> 597533342 INFO
> (zkCallback-4-thread-1082-processing-n:52.91.90.134:8983_solr) [ ]
> o.a.s.s.ZkIndexSchemaReader Retrieved schema version 89 from ZooKeeper
> 597578973 INFO
> (zkCallback-4-thread-1082-processing-n:52.91.90.134:8983_solr) [ ]
> o.a.s.s.IndexSchema Reading Solr Schema from
> /configs/mycollection/managed-schema
> 597533342 INFO
> (zkCallback-4-thread-1084-processing-n:52.91.90.134:8983_solr) [ ]
> o.a.s.s.ZkIndexSchemaReader Retrieved schema version 89 from ZooKeeper
> 597644246 INFO
> (zkCallback-4-thread-1073-processing-n:52.91.90.134:8983_solr) [ ]
> o.a.s.s.ZkIndexSchemaReader A schema change: WatchedEvent
> state:SyncConnected type:NodeDataChanged
> path:/configs/mycollection/managed-schema, has occurred - updating schema
> from ZooKeeper ...
> 597674162 WARN
> (zkCallback-4-thread-1073-processing-n:52.91.90.134:8983_solr) [ ]
> o.a.s.s.ZkIndexSchemaReader ZooKeeper watch triggered, but Solr cannot
> talk to ZK
> 597628388 ERROR
> (zkCallback-4-thread-881-processing-n:52.91.90.134:8983_solr-EventThread)
> [ ] o.a.z.ClientCnxn Error while calling watcher
> java.lang.OutOfMemoryError: Java heap space
> 597593838 INFO
> (zkCallback-4-thread-1103-processing-n:52.91.90.134:8983_solr) [ ]
> o.a.s.s.ZkIndexSchemaReader A schema change: WatchedEvent
> state:SyncConnected type:NodeDataChanged
> path:/configs/mycollection/managed-schema, has occurred - updating schema
> from ZooKeeper ...
>
>
> So it looks like it ran out of memory . . . Strange but I thought my
> collections were pretty small.
> Any idea why a replace-field-type call might cause an OutOfMemoryException?
>
> -Frank
>
> Example Curl request that causes Solr nodes to appear as ³Gone²
>
> curl -X POST -H 'Content-type:application/json' --data-binary '{
> "replace-field-type" : {
> "name":"text_ws",
> "class":"solr.TextField",
> "positionIncrementGap":"100",
> "indexAnalyzer" : {
> "tokenizer":{ "class":"solr.StandardTokenizerFactory" },
> "filters":[
> { "class":"solr.StopFilterFactory", "ignoreCase":true,
> "words":"stopwords.txt" },
> { "class":"solr.LowerCaseFilterFactory" },
> { "class":"solr.EdgeNGramFilterFactory",
> "minGramSize":"2", "maxGramSize":"15" }
> ]
> },
> "queryAnalyzer" : {
> "tokenizer":{ "class":"solr.StandardTokenizerFactory" },
> "filters":[
> { "class":"solr.StopFilterFactory", "ignoreCase":true,
> "words":"stopwords.txt" },
> { "class":"solr.SynonymFilterFactory",
> "synonyms":"synonyms.txt", "ignoreCase":true, "expand":true },
> { "class":"solr.LowerCaseFilterFactory" }
> ]
> }
> }
> }' http://aa.bb.cc.dd:8983/solr/my_collection_name/schema
>
>
>
>
>
>
>
>
>
>
> On 12/1/15, 7:45 AM, "Emir Arnautovic" <em...@sematext.com>
> wrote:
>
>> Hi Frank,
>> Can you please confirm that Solr nodes are aware of entire ZK ensemble?
>> Can you give more info how it is deployed - ZK on separate servers? What
>> is load on Solr when it happens? Do you see any errors in Solr logs?
>>
>> Thanks,
>> Emir
>>
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
Re: ZooKeeper nodes die taking down Solr Cluster?
Posted by "Kelly, Frank" <fr...@here.com>.
Thanks Emir - responses inline below
>Can you please confirm that Solr nodes are aware of entire ZK ensemble?
Can you explain how I could find that out - I looked into the logs and the
Admin UI and didn¹t see
A way to examine if the Solr nodes saw the entire ensemble
>Can you give more info how it is deployed - ZK on separate servers?
3 Zookeeper nodes (separate servers)
3 Shards and 3 replicas (again each node on separate servers)
Deployed in same AWS region and same AZ
>What is load on Solr when it happens?
Very minimal
Although it seems to occur around or after the time I do a Curl command to
update a field or field type in a collection.
See curl request below
>Do you see any errors in Solr logs?
I see the following in the log file of one of the servers that dropped out
of the cluster
597534321 INFO
(zkCallback-4-thread-1094-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader A schema change: WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/configs/mycollection/managed-schema, has occurred - updating schema
from ZooKeeper ...
597533348 INFO
(zkCallback-4-thread-1079-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader Retrieved schema version 89 from ZooKeeper
597533348 INFO
(zkCallback-4-thread-1080-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader Retrieved schema version 89 from ZooKeeper
597574579 INFO
(zkCallback-4-thread-1080-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.IndexSchema Reading Solr Schema from
/configs/mycollection/managed-schema
597533348 INFO
(zkCallback-4-thread-1076-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader Retrieved schema version 89 from ZooKeeper
597574580 INFO
(zkCallback-4-thread-1076-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.IndexSchema Reading Solr Schema from
/configs/mycollection/managed-schema
597533348 INFO
(zkCallback-4-thread-1083-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader Retrieved schema version 89 from ZooKeeper
597574580 INFO
(zkCallback-4-thread-1083-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.IndexSchema Reading Solr Schema from
/configs/mycollection/managed-schema
597533348 INFO
(zkCallback-4-thread-1074-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader Retrieved schema version 89 from ZooKeeper
597533348 INFO
(zkCallback-4-thread-1087-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader Retrieved schema version 89 from ZooKeeper
597576496 INFO
(zkCallback-4-thread-1087-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.IndexSchema Reading Solr Schema from
/configs/mycollection/managed-schema
597533348 INFO
(zkCallback-4-thread-1077-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader Retrieved schema version 89 from ZooKeeper
597576497 INFO
(zkCallback-4-thread-1077-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.IndexSchema Reading Solr Schema from
/configs/mycollection/managed-schema
597533348 INFO
(zkCallback-4-thread-1089-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader Retrieved schema version 89 from ZooKeeper
597576497 INFO
(zkCallback-4-thread-1089-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.IndexSchema Reading Solr Schema from
/configs/mycollection/managed-schema
597533348 INFO
(zkCallback-4-thread-1078-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader Retrieved schema version 89 from ZooKeeper
597577450 INFO
(zkCallback-4-thread-1078-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.IndexSchema Reading Solr Schema from
/configs/mycollection/managed-schema
597533345 INFO
(zkCallback-4-thread-1093-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader A schema change: WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/configs/mycollection/managed-schema, has occurred - updating schema
from ZooKeeper ...
597533342 INFO
(zkCallback-4-thread-1081-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader Retrieved schema version 89 from ZooKeeper
597578973 INFO
(zkCallback-4-thread-1081-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.IndexSchema Reading Solr Schema from
/configs/mycollection/managed-schema
597533342 INFO
(zkCallback-4-thread-1082-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader Retrieved schema version 89 from ZooKeeper
597578973 INFO
(zkCallback-4-thread-1082-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.IndexSchema Reading Solr Schema from
/configs/mycollection/managed-schema
597533342 INFO
(zkCallback-4-thread-1084-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader Retrieved schema version 89 from ZooKeeper
597644246 INFO
(zkCallback-4-thread-1073-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader A schema change: WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/configs/mycollection/managed-schema, has occurred - updating schema
from ZooKeeper ...
597674162 WARN
(zkCallback-4-thread-1073-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader ZooKeeper watch triggered, but Solr cannot
talk to ZK
597628388 ERROR
(zkCallback-4-thread-881-processing-n:52.91.90.134:8983_solr-EventThread)
[ ] o.a.z.ClientCnxn Error while calling watcher
java.lang.OutOfMemoryError: Java heap space
597593838 INFO
(zkCallback-4-thread-1103-processing-n:52.91.90.134:8983_solr) [ ]
o.a.s.s.ZkIndexSchemaReader A schema change: WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/configs/mycollection/managed-schema, has occurred - updating schema
from ZooKeeper ...
So it looks like it ran out of memory . . . Strange but I thought my
collections were pretty small.
Any idea why a replace-field-type call might cause an OutOfMemoryException?
-Frank
Example Curl request that causes Solr nodes to appear as ³Gone²
curl -X POST -H 'Content-type:application/json' --data-binary '{
"replace-field-type" : {
"name":"text_ws",
"class":"solr.TextField",
"positionIncrementGap":"100",
"indexAnalyzer" : {
"tokenizer":{ "class":"solr.StandardTokenizerFactory" },
"filters":[
{ "class":"solr.StopFilterFactory", "ignoreCase":true,
"words":"stopwords.txt" },
{ "class":"solr.LowerCaseFilterFactory" },
{ "class":"solr.EdgeNGramFilterFactory",
"minGramSize":"2", "maxGramSize":"15" }
]
},
"queryAnalyzer" : {
"tokenizer":{ "class":"solr.StandardTokenizerFactory" },
"filters":[
{ "class":"solr.StopFilterFactory", "ignoreCase":true,
"words":"stopwords.txt" },
{ "class":"solr.SynonymFilterFactory",
"synonyms":"synonyms.txt", "ignoreCase":true, "expand":true },
{ "class":"solr.LowerCaseFilterFactory" }
]
}
}
}' http://aa.bb.cc.dd:8983/solr/my_collection_name/schema
On 12/1/15, 7:45 AM, "Emir Arnautovic" <em...@sematext.com>
wrote:
>Hi Frank,
>Can you please confirm that Solr nodes are aware of entire ZK ensemble?
>Can you give more info how it is deployed - ZK on separate servers? What
>is load on Solr when it happens? Do you see any errors in Solr logs?
>
>Thanks,
>Emir
>