You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "Zarski, Jacek" <Za...@DNB.com.INVALID> on 2018/07/30 20:06:07 UTC
Zookeeper / Solr interaction
Hi,
We have the following environment setup for zookeeper/solrcould
3 zookeeper ensemble
2 Solr cloud servers
I am writing you to further inquire about the interaction of solr and zookeeper. In particular relating to transactions in the transaction logs. I have a script running that logs the amount of transactions. I am matching this log with snapshot timing and new log creation.
After a problem arose in our PROD environment, I have tracked it to an unrecommended configuration where logs and data was kept on the same drive. Since then we have configured separate drives for logs and data in that environment. The behavior that caused the problem was when a snapshot was happening, a solr instance reported that it was unable to establish a ZK leader. Following that failure, during recovery, 4 more snapshots happened in short succession(10 minutes) on all 3 zk servers causing the whole environment to be unresponsive until restart for 1.5 hours.
I am currently working to recreate the problem and gather more information on the cause and impact of snapshots. I have configured a DEV environment with the same number of servers. I have changed the zk configuration to again have the logs and data in the same drive and directory. I am seeing that snapshots cause a degredation in performance due to IO block but I would like more information on transactions and snapshots to confirm this behavior and our suspicions.
Here are the scenarios I would like more information about:
1. When the solr server is restarted, I see a huge influx of transactions on the zookeeper transaction log. What is the solr behavior that is causing this and is this normal?
2. There is scenarios where snapshots are being created without reaching "snapCount" (snapCount=100000) transactions. I have documented snapshots at 17k and 45k transactions. In what scenarios would a snapshot be created other than reaching "snapCount" transactions?
3. Since zk won't respond before writing to the transaction log... at Snapshot time(IO block) is it possible for the solr server to wait for a response from zk causing all other writes to be buffered resulting in a full heap and therefore an out of memory failure on the solr node?
a. Now referencing question #1... When a solr node recovers, the influx of transactions plus the continuing writes seems to be enough to trigger another snapshot resulting in further downtime. Is this case plausible?
Thanks,
Jacek
Re: Zookeeper / Solr interaction
Posted by Erick Erickson <er...@gmail.com>.
Ok, your OOM errors are most likely due to
trying to stuff too many replicas into too little memory.
You have 100 collections, 8 shards per collection and
1 replica per shard. So if my math is right, you have
800 replicas total, 400 replicas per Solr instance.
6G of memory is very little for that many replicas, and
any significant number of docs in these collections will
make matters worse. Anything to do with sorting,
faceting or grouping that works on fields that do not
use docValues will make it even worse yet.
All that said, I've certainly seen many more replicas
than that operate with a 3-node external ZK ensemble so
I doubt it's the case you're overloading ZK. There are
some additional multipliers to the number of ZK
events having to do with the "overseer".
My best guess is that your focus on ZK events is a red
herring, and that if you distributed your replicas over more
Solr nodes and/or were able to allocate significantly more
memory to each node the problem would go away.
Note that the usual recommendation is that you allocate no
more than 50% of your available physical memory to
the Java heap.
Best,
Erick
On Tue, Jul 31, 2018 at 8:06 AM, Zarski, Jacek <Za...@dnb.com.invalid> wrote:
> Thanks for responding! That's some good info. Here are the answers to the questions you had...
>
> Solr has 6gb of heap
> We have 1 replica per shard at 8 shards per collection
> We currently have approximately 100 collections
> Zookeeper is an external ensemble each with their own server
>
> -----Original Message-----
> From: Erick Erickson <er...@gmail.com>
> Sent: Monday, July 30, 2018 7:56 PM
> To: solr-user <so...@lucene.apache.org>
> Subject: Re: Zookeeper / Solr interaction
>
> 1> Yeah, the interactions with ZK are quite chatty. Basically each
> replica may have several changes of state with ZooKeeper.
> Down->recovering->active. How many replicas do you have on a node?
>
> 2> Unfortunately I don't have much info on this point.
>
> 3> I would not expect OOMs on the Solr node while waiting for ZK to
> respond. How much heap are you allocating to Solr? How many replicas do you have?
>
> a> Plausible yes, but that many transactions seems quite high. How
> many replicas do you have on your Solr instance? One scenario here is that you have thousands of replicas. Is your ZK ensemble an external one? And are they running on separate hardware? Because this many transactions for only two Solr instances seems quite high so I'm curious about a few more details of your setup, how many collections, shards and replicas are we talking here?
>
> Best,
> Erick
>
>
> On Mon, Jul 30, 2018 at 1:10 PM, Zarski, Jacek <Za...@dnb.com.invalid> wrote:
>> Some information I forgot to include:
>> Solr version : 7.2.1
>> Zk version : 3.4.10
>>
>> -----Original Message-----
>> From: Zarski, Jacek <Za...@DNB.com.INVALID>
>> Sent: Monday, July 30, 2018 4:06 PM
>> To: solr-user@lucene.apache.org
>> Subject: Zookeeper / Solr interaction
>>
>> Hi,
>>
>> We have the following environment setup for zookeeper/solrcould
>>
>> 3 zookeeper ensemble
>> 2 Solr cloud servers
>>
>> I am writing you to further inquire about the interaction of solr and zookeeper. In particular relating to transactions in the transaction logs. I have a script running that logs the amount of transactions. I am matching this log with snapshot timing and new log creation.
>>
>> After a problem arose in our PROD environment, I have tracked it to an unrecommended configuration where logs and data was kept on the same drive. Since then we have configured separate drives for logs and data in that environment. The behavior that caused the problem was when a snapshot was happening, a solr instance reported that it was unable to establish a ZK leader. Following that failure, during recovery, 4 more snapshots happened in short succession(10 minutes) on all 3 zk servers causing the whole environment to be unresponsive until restart for 1.5 hours.
>>
>> I am currently working to recreate the problem and gather more information on the cause and impact of snapshots. I have configured a DEV environment with the same number of servers. I have changed the zk configuration to again have the logs and data in the same drive and directory. I am seeing that snapshots cause a degredation in performance due to IO block but I would like more information on transactions and snapshots to confirm this behavior and our suspicions.
>>
>> Here are the scenarios I would like more information about:
>>
>> 1. When the solr server is restarted, I see a huge influx of transactions on the zookeeper transaction log. What is the solr behavior that is causing this and is this normal?
>>
>> 2. There is scenarios where snapshots are being created without reaching "snapCount" (snapCount=100000) transactions. I have documented snapshots at 17k and 45k transactions. In what scenarios would a snapshot be created other than reaching "snapCount" transactions?
>>
>> 3. Since zk won't respond before writing to the transaction log... at Snapshot time(IO block) is it possible for the solr server to wait for a response from zk causing all other writes to be buffered resulting in a full heap and therefore an out of memory failure on the solr node?
>>
>> a. Now referencing question #1... When a solr node recovers, the influx of transactions plus the continuing writes seems to be enough to trigger another snapshot resulting in further downtime. Is this case plausible?
>>
>> Thanks,
>> Jacek
RE: Zookeeper / Solr interaction
Posted by "Zarski, Jacek" <Za...@DNB.com.INVALID>.
Thanks for responding! That's some good info. Here are the answers to the questions you had...
Solr has 6gb of heap
We have 1 replica per shard at 8 shards per collection
We currently have approximately 100 collections
Zookeeper is an external ensemble each with their own server
-----Original Message-----
From: Erick Erickson <er...@gmail.com>
Sent: Monday, July 30, 2018 7:56 PM
To: solr-user <so...@lucene.apache.org>
Subject: Re: Zookeeper / Solr interaction
1> Yeah, the interactions with ZK are quite chatty. Basically each
replica may have several changes of state with ZooKeeper.
Down->recovering->active. How many replicas do you have on a node?
2> Unfortunately I don't have much info on this point.
3> I would not expect OOMs on the Solr node while waiting for ZK to
respond. How much heap are you allocating to Solr? How many replicas do you have?
a> Plausible yes, but that many transactions seems quite high. How
many replicas do you have on your Solr instance? One scenario here is that you have thousands of replicas. Is your ZK ensemble an external one? And are they running on separate hardware? Because this many transactions for only two Solr instances seems quite high so I'm curious about a few more details of your setup, how many collections, shards and replicas are we talking here?
Best,
Erick
On Mon, Jul 30, 2018 at 1:10 PM, Zarski, Jacek <Za...@dnb.com.invalid> wrote:
> Some information I forgot to include:
> Solr version : 7.2.1
> Zk version : 3.4.10
>
> -----Original Message-----
> From: Zarski, Jacek <Za...@DNB.com.INVALID>
> Sent: Monday, July 30, 2018 4:06 PM
> To: solr-user@lucene.apache.org
> Subject: Zookeeper / Solr interaction
>
> Hi,
>
> We have the following environment setup for zookeeper/solrcould
>
> 3 zookeeper ensemble
> 2 Solr cloud servers
>
> I am writing you to further inquire about the interaction of solr and zookeeper. In particular relating to transactions in the transaction logs. I have a script running that logs the amount of transactions. I am matching this log with snapshot timing and new log creation.
>
> After a problem arose in our PROD environment, I have tracked it to an unrecommended configuration where logs and data was kept on the same drive. Since then we have configured separate drives for logs and data in that environment. The behavior that caused the problem was when a snapshot was happening, a solr instance reported that it was unable to establish a ZK leader. Following that failure, during recovery, 4 more snapshots happened in short succession(10 minutes) on all 3 zk servers causing the whole environment to be unresponsive until restart for 1.5 hours.
>
> I am currently working to recreate the problem and gather more information on the cause and impact of snapshots. I have configured a DEV environment with the same number of servers. I have changed the zk configuration to again have the logs and data in the same drive and directory. I am seeing that snapshots cause a degredation in performance due to IO block but I would like more information on transactions and snapshots to confirm this behavior and our suspicions.
>
> Here are the scenarios I would like more information about:
>
> 1. When the solr server is restarted, I see a huge influx of transactions on the zookeeper transaction log. What is the solr behavior that is causing this and is this normal?
>
> 2. There is scenarios where snapshots are being created without reaching "snapCount" (snapCount=100000) transactions. I have documented snapshots at 17k and 45k transactions. In what scenarios would a snapshot be created other than reaching "snapCount" transactions?
>
> 3. Since zk won't respond before writing to the transaction log... at Snapshot time(IO block) is it possible for the solr server to wait for a response from zk causing all other writes to be buffered resulting in a full heap and therefore an out of memory failure on the solr node?
>
> a. Now referencing question #1... When a solr node recovers, the influx of transactions plus the continuing writes seems to be enough to trigger another snapshot resulting in further downtime. Is this case plausible?
>
> Thanks,
> Jacek
Re: Zookeeper / Solr interaction
Posted by Erick Erickson <er...@gmail.com>.
1> Yeah, the interactions with ZK are quite chatty. Basically each
replica may have several changes of state with ZooKeeper.
Down->recovering->active. How many replicas do you have on a node?
2> Unfortunately I don't have much info on this point.
3> I would not expect OOMs on the Solr node while waiting for ZK to
respond. How much heap are you allocating to Solr? How many replicas
do you have?
a> Plausible yes, but that many transactions seems quite high. How
many replicas do you have on your Solr instance? One scenario here is
that you have thousands of replicas. Is your ZK ensemble an external
one? And are they running on separate hardware? Because this many
transactions for only two Solr instances seems quite high so I'm
curious about a few more details of your setup, how many collections,
shards and replicas are we talking here?
Best,
Erick
On Mon, Jul 30, 2018 at 1:10 PM, Zarski, Jacek <Za...@dnb.com.invalid> wrote:
> Some information I forgot to include:
> Solr version : 7.2.1
> Zk version : 3.4.10
>
> -----Original Message-----
> From: Zarski, Jacek <Za...@DNB.com.INVALID>
> Sent: Monday, July 30, 2018 4:06 PM
> To: solr-user@lucene.apache.org
> Subject: Zookeeper / Solr interaction
>
> Hi,
>
> We have the following environment setup for zookeeper/solrcould
>
> 3 zookeeper ensemble
> 2 Solr cloud servers
>
> I am writing you to further inquire about the interaction of solr and zookeeper. In particular relating to transactions in the transaction logs. I have a script running that logs the amount of transactions. I am matching this log with snapshot timing and new log creation.
>
> After a problem arose in our PROD environment, I have tracked it to an unrecommended configuration where logs and data was kept on the same drive. Since then we have configured separate drives for logs and data in that environment. The behavior that caused the problem was when a snapshot was happening, a solr instance reported that it was unable to establish a ZK leader. Following that failure, during recovery, 4 more snapshots happened in short succession(10 minutes) on all 3 zk servers causing the whole environment to be unresponsive until restart for 1.5 hours.
>
> I am currently working to recreate the problem and gather more information on the cause and impact of snapshots. I have configured a DEV environment with the same number of servers. I have changed the zk configuration to again have the logs and data in the same drive and directory. I am seeing that snapshots cause a degredation in performance due to IO block but I would like more information on transactions and snapshots to confirm this behavior and our suspicions.
>
> Here are the scenarios I would like more information about:
>
> 1. When the solr server is restarted, I see a huge influx of transactions on the zookeeper transaction log. What is the solr behavior that is causing this and is this normal?
>
> 2. There is scenarios where snapshots are being created without reaching "snapCount" (snapCount=100000) transactions. I have documented snapshots at 17k and 45k transactions. In what scenarios would a snapshot be created other than reaching "snapCount" transactions?
>
> 3. Since zk won't respond before writing to the transaction log... at Snapshot time(IO block) is it possible for the solr server to wait for a response from zk causing all other writes to be buffered resulting in a full heap and therefore an out of memory failure on the solr node?
>
> a. Now referencing question #1... When a solr node recovers, the influx of transactions plus the continuing writes seems to be enough to trigger another snapshot resulting in further downtime. Is this case plausible?
>
> Thanks,
> Jacek
RE: Zookeeper / Solr interaction
Posted by "Zarski, Jacek" <Za...@DNB.com.INVALID>.
Some information I forgot to include:
Solr version : 7.2.1
Zk version : 3.4.10
-----Original Message-----
From: Zarski, Jacek <Za...@DNB.com.INVALID>
Sent: Monday, July 30, 2018 4:06 PM
To: solr-user@lucene.apache.org
Subject: Zookeeper / Solr interaction
Hi,
We have the following environment setup for zookeeper/solrcould
3 zookeeper ensemble
2 Solr cloud servers
I am writing you to further inquire about the interaction of solr and zookeeper. In particular relating to transactions in the transaction logs. I have a script running that logs the amount of transactions. I am matching this log with snapshot timing and new log creation.
After a problem arose in our PROD environment, I have tracked it to an unrecommended configuration where logs and data was kept on the same drive. Since then we have configured separate drives for logs and data in that environment. The behavior that caused the problem was when a snapshot was happening, a solr instance reported that it was unable to establish a ZK leader. Following that failure, during recovery, 4 more snapshots happened in short succession(10 minutes) on all 3 zk servers causing the whole environment to be unresponsive until restart for 1.5 hours.
I am currently working to recreate the problem and gather more information on the cause and impact of snapshots. I have configured a DEV environment with the same number of servers. I have changed the zk configuration to again have the logs and data in the same drive and directory. I am seeing that snapshots cause a degredation in performance due to IO block but I would like more information on transactions and snapshots to confirm this behavior and our suspicions.
Here are the scenarios I would like more information about:
1. When the solr server is restarted, I see a huge influx of transactions on the zookeeper transaction log. What is the solr behavior that is causing this and is this normal?
2. There is scenarios where snapshots are being created without reaching "snapCount" (snapCount=100000) transactions. I have documented snapshots at 17k and 45k transactions. In what scenarios would a snapshot be created other than reaching "snapCount" transactions?
3. Since zk won't respond before writing to the transaction log... at Snapshot time(IO block) is it possible for the solr server to wait for a response from zk causing all other writes to be buffered resulting in a full heap and therefore an out of memory failure on the solr node?
a. Now referencing question #1... When a solr node recovers, the influx of transactions plus the continuing writes seems to be enough to trigger another snapshot resulting in further downtime. Is this case plausible?
Thanks,
Jacek