You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "Oakley, Craig (NIH/NLM/NCBI) [C]" <cr...@nih.gov.INVALID> on 2020/07/01 19:42:25 UTC
RE: CDCR stress-test issues
For the record, it is not just Solr7.4 which has the problem. When I start afresh with Solr8.5.2, both symptoms persist.
With Solr8.5.2, tlogs accumulate endlessly at the non-Leader nodes of the Source SolrCloud and are never released regardless of maxNumLogsToKeep setting
And with Solr8.5.2, if four scripts run simultaneously for a few minutes, each script running a loop each iteration of which adds batches of 6 records to the Source SolrCloud, a couple dozen records wind up on the Source without ever arriving at the Target SolrCloud (although the Target does have records which were added after the missing records).
Does anyone yet have any suggestion how to get CDCR to work properly?
-----Original Message-----
From: Oakley, Craig (NIH/NLM/NCBI) [C] <cr...@nih.gov.INVALID>
Sent: Wednesday, June 24, 2020 9:46 AM
To: solr-user@lucene.apache.org
Subject: CDCR stress-test issues
In attempting to stress-test CDCR (running Solr 7.4), I am running into a couple of issues.
One is that the tlog files keep accumulating for some nodes in the CDCR system, particularly for the non-Leader nodes in the Source SolrCloud. No quantity of hard commits seem to cause any of these tlog files to be released. This can become a problem upon reboot if there are hundreds of thousands of tlog files, and Solr fails to start (complaining that there are too many open files).
The tlogs had been accumulating on all the nodes of the CDCR set of SolrClouds until I added these two lines to the solrconfig.xml file (for testing purposes, using numbers much lower than in the examples):
<int name="numRecordsToKeep">5</int>
<int name="maxNumLogsToKeep">2</int>
Since then, it is mostly the non-Leader nodes of the Source SolrCloud which accumulates tlog files (the Target SolrCloud does seem to have a tendency to clean up the tlog files, as does the Leader of the Source SolrCloud). If I use ADDREPLICAPROP and REBALANCELEADERS to change which node is the Leader, and if I then start adding more data, the tlogs on the new Leader sometimes will go away, but then the old Leader begins accumulating tlog files. I am dubious whether frequent reassignment of Leadership would be a practical solution.
I also have several times attempted to simulate a production environment by running several loops simultaneously, each of which inserts multiple records on each iteration of the loop. Several times, I end up with a dozen records on (both replicas of) the Source which never make it to (either replica of) the Target. The Target has thousands of records which were inserted before the missing records, and thousands of records which were inserted after the missing records (and all these records, the replicated and the missing, were inserted by curl commands which only differed in sequential numbers incorporated into the values being inserted).
I also have a question regarding SOLR-13141: the 11/Feb/19 comment says that the fix for Solr 7.3 had a problem; and the header says "Affects Version/s: 7.5, 7.6": does that indicate that Solr 7.4 is not affected?
Are there any suggestions?
Thanks
RE: CDCR stress-test issues
Posted by "Oakley, Craig (NIH/NLM/NCBI) [C]" <cr...@nih.gov.INVALID>.
Yes, I saw that yesterday.
I guess that I was not the only one who noticed the unreliability after all.
-----Original Message-----
From: Ishan Chattopadhyaya <ic...@gmail.com>
Sent: Friday, July 17, 2020 1:17 AM
To: solr-user <so...@lucene.apache.org>
Subject: Re: CDCR stress-test issues
FYI, CDCR support, as it exists in Solr today, has been deprecated in 8.6.
It suffers from serious design flaws and it allows such things to happen
that you observe. While there may be workarounds, it is advisable to not
rely on CDCR in production.
Thanks,
Ishan
On Thu, 2 Jul, 2020, 1:12 am Oakley, Craig (NIH/NLM/NCBI) [C],
<cr...@nih.gov.invalid> wrote:
> For the record, it is not just Solr7.4 which has the problem. When I start
> afresh with Solr8.5.2, both symptoms persist.
>
> With Solr8.5.2, tlogs accumulate endlessly at the non-Leader nodes of the
> Source SolrCloud and are never released regardless of maxNumLogsToKeep
> setting
>
> And with Solr8.5.2, if four scripts run simultaneously for a few minutes,
> each script running a loop each iteration of which adds batches of 6
> records to the Source SolrCloud, a couple dozen records wind up on the
> Source without ever arriving at the Target SolrCloud (although the Target
> does have records which were added after the missing records).
>
> Does anyone yet have any suggestion how to get CDCR to work properly?
>
>
> -----Original Message-----
> From: Oakley, Craig (NIH/NLM/NCBI) [C] <cr...@nih.gov.INVALID>
> Sent: Wednesday, June 24, 2020 9:46 AM
> To: solr-user@lucene.apache.org
> Subject: CDCR stress-test issues
>
> In attempting to stress-test CDCR (running Solr 7.4), I am running into a
> couple of issues.
>
> One is that the tlog files keep accumulating for some nodes in the CDCR
> system, particularly for the non-Leader nodes in the Source SolrCloud. No
> quantity of hard commits seem to cause any of these tlog files to be
> released. This can become a problem upon reboot if there are hundreds of
> thousands of tlog files, and Solr fails to start (complaining that there
> are too many open files).
>
> The tlogs had been accumulating on all the nodes of the CDCR set of
> SolrClouds until I added these two lines to the solrconfig.xml file (for
> testing purposes, using numbers much lower than in the examples):
> <int name="numRecordsToKeep">5</int>
> <int name="maxNumLogsToKeep">2</int>
> Since then, it is mostly the non-Leader nodes of the Source SolrCloud
> which accumulates tlog files (the Target SolrCloud does seem to have a
> tendency to clean up the tlog files, as does the Leader of the Source
> SolrCloud). If I use ADDREPLICAPROP and REBALANCELEADERS to change which
> node is the Leader, and if I then start adding more data, the tlogs on the
> new Leader sometimes will go away, but then the old Leader begins
> accumulating tlog files. I am dubious whether frequent reassignment of
> Leadership would be a practical solution.
>
> I also have several times attempted to simulate a production environment
> by running several loops simultaneously, each of which inserts multiple
> records on each iteration of the loop. Several times, I end up with a dozen
> records on (both replicas of) the Source which never make it to (either
> replica of) the Target. The Target has thousands of records which were
> inserted before the missing records, and thousands of records which were
> inserted after the missing records (and all these records, the replicated
> and the missing, were inserted by curl commands which only differed in
> sequential numbers incorporated into the values being inserted).
>
> I also have a question regarding SOLR-13141: the 11/Feb/19 comment says
> that the fix for Solr 7.3 had a problem; and the header says "Affects
> Version/s: 7.5, 7.6": does that indicate that Solr 7.4 is not affected?
>
> Are there any suggestions?
>
> Thanks
>
Re: CDCR stress-test issues
Posted by Ishan Chattopadhyaya <ic...@gmail.com>.
FYI, CDCR support, as it exists in Solr today, has been deprecated in 8.6.
It suffers from serious design flaws and it allows such things to happen
that you observe. While there may be workarounds, it is advisable to not
rely on CDCR in production.
Thanks,
Ishan
On Thu, 2 Jul, 2020, 1:12 am Oakley, Craig (NIH/NLM/NCBI) [C],
<cr...@nih.gov.invalid> wrote:
> For the record, it is not just Solr7.4 which has the problem. When I start
> afresh with Solr8.5.2, both symptoms persist.
>
> With Solr8.5.2, tlogs accumulate endlessly at the non-Leader nodes of the
> Source SolrCloud and are never released regardless of maxNumLogsToKeep
> setting
>
> And with Solr8.5.2, if four scripts run simultaneously for a few minutes,
> each script running a loop each iteration of which adds batches of 6
> records to the Source SolrCloud, a couple dozen records wind up on the
> Source without ever arriving at the Target SolrCloud (although the Target
> does have records which were added after the missing records).
>
> Does anyone yet have any suggestion how to get CDCR to work properly?
>
>
> -----Original Message-----
> From: Oakley, Craig (NIH/NLM/NCBI) [C] <cr...@nih.gov.INVALID>
> Sent: Wednesday, June 24, 2020 9:46 AM
> To: solr-user@lucene.apache.org
> Subject: CDCR stress-test issues
>
> In attempting to stress-test CDCR (running Solr 7.4), I am running into a
> couple of issues.
>
> One is that the tlog files keep accumulating for some nodes in the CDCR
> system, particularly for the non-Leader nodes in the Source SolrCloud. No
> quantity of hard commits seem to cause any of these tlog files to be
> released. This can become a problem upon reboot if there are hundreds of
> thousands of tlog files, and Solr fails to start (complaining that there
> are too many open files).
>
> The tlogs had been accumulating on all the nodes of the CDCR set of
> SolrClouds until I added these two lines to the solrconfig.xml file (for
> testing purposes, using numbers much lower than in the examples):
> <int name="numRecordsToKeep">5</int>
> <int name="maxNumLogsToKeep">2</int>
> Since then, it is mostly the non-Leader nodes of the Source SolrCloud
> which accumulates tlog files (the Target SolrCloud does seem to have a
> tendency to clean up the tlog files, as does the Leader of the Source
> SolrCloud). If I use ADDREPLICAPROP and REBALANCELEADERS to change which
> node is the Leader, and if I then start adding more data, the tlogs on the
> new Leader sometimes will go away, but then the old Leader begins
> accumulating tlog files. I am dubious whether frequent reassignment of
> Leadership would be a practical solution.
>
> I also have several times attempted to simulate a production environment
> by running several loops simultaneously, each of which inserts multiple
> records on each iteration of the loop. Several times, I end up with a dozen
> records on (both replicas of) the Source which never make it to (either
> replica of) the Target. The Target has thousands of records which were
> inserted before the missing records, and thousands of records which were
> inserted after the missing records (and all these records, the replicated
> and the missing, were inserted by curl commands which only differed in
> sequential numbers incorporated into the values being inserted).
>
> I also have a question regarding SOLR-13141: the 11/Feb/19 comment says
> that the fix for Solr 7.3 had a problem; and the header says "Affects
> Version/s: 7.5, 7.6": does that indicate that Solr 7.4 is not affected?
>
> Are there any suggestions?
>
> Thanks
>