You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "Xie, Sean" <Se...@finra.org> on 2017/07/08 04:35:51 UTC
CDCR - how to deal with the transaction log files
Once enabled CDCR, update log stores an unlimited number of entries. This is causing the tlog folder getting bigger and bigger, as well as the open files are growing. How can one reduce the number of open files and also to reduce the tlog files? If it’s not taken care properly, sooner or later the log files size and open file count will exceed the limits.
Thanks
Sean
Confidentiality Notice:: This email, including attachments, may include non-public, proprietary, confidential or legally privileged information. If you are not an intended recipient or an authorized agent of an intended recipient, you are hereby notified that any dissemination, distribution or copying of the information contained in or transmitted with this e-mail is unauthorized and strictly prohibited. If you have received this email in error, please notify the sender by replying to this message and permanently delete this e-mail, its attachments, and any copies of it immediately. You should not retain, copy or use this e-mail or any attachment for any purpose, nor disclose all or any part of the contents to any other person. Thank you.
Re: CDCR - how to deal with the transaction log files
Posted by Erick Erickson <er...@gmail.com>.
This should not be the case if you are actively sending updates to the
target cluster. The tlog is used to store unsent updates, so if the
connection is broken for some time, the target cluster will have a
chance to catch up.
If you don't have the remote DC online and do not intend to bring it
online soon, you should turn CDCR off.
Best,
Erick
On Fri, Jul 7, 2017 at 9:35 PM, Xie, Sean <Se...@finra.org> wrote:
> Once enabled CDCR, update log stores an unlimited number of entries. This is causing the tlog folder getting bigger and bigger, as well as the open files are growing. How can one reduce the number of open files and also to reduce the tlog files? If it’s not taken care properly, sooner or later the log files size and open file count will exceed the limits.
>
> Thanks
> Sean
>
>
> Confidentiality Notice:: This email, including attachments, may include non-public, proprietary, confidential or legally privileged information. If you are not an intended recipient or an authorized agent of an intended recipient, you are hereby notified that any dissemination, distribution or copying of the information contained in or transmitted with this e-mail is unauthorized and strictly prohibited. If you have received this email in error, please notify the sender by replying to this message and permanently delete this e-mail, its attachments, and any copies of it immediately. You should not retain, copy or use this e-mail or any attachment for any purpose, nor disclose all or any part of the contents to any other person. Thank you.
Re: CDCR - how to deal with the transaction log files
Posted by Susheel Kumar <su...@gmail.com>.
I just voted for https://issues.apache.org/jira/browse/SOLR-11069 to get it
resolved, as we are discussing to start using CDCR soon.
On Fri, Jul 14, 2017 at 5:21 PM, Varun Thacker <va...@vthacker.in> wrote:
> https://issues.apache.org/jira/browse/SOLR-11069 is tracking why is
> LASTPROCESSEDVERSION=-1
> on the source cluster always
>
> On Fri, Jul 14, 2017 at 11:46 AM, jmyatt <jm...@wayfair.com> wrote:
>
> > Thanks for the suggestion - tried that today and still no luck. Time to
> > write a script to naively / blindly delete old logs and run that in cron.
> > *sigh*
> >
> >
> >
> > --
> > View this message in context: http://lucene.472066.n3.
> > nabble.com/CDCR-how-to-deal-with-the-transaction-log-
> > files-tp4345062p4346138.html
> > Sent from the Solr - User mailing list archive at Nabble.com.
> >
>
Re: CDCR - how to deal with the transaction log files
Posted by Varun Thacker <va...@vthacker.in>.
https://issues.apache.org/jira/browse/SOLR-11069 is tracking why is
LASTPROCESSEDVERSION=-1
on the source cluster always
On Fri, Jul 14, 2017 at 11:46 AM, jmyatt <jm...@wayfair.com> wrote:
> Thanks for the suggestion - tried that today and still no luck. Time to
> write a script to naively / blindly delete old logs and run that in cron.
> *sigh*
>
>
>
> --
> View this message in context: http://lucene.472066.n3.
> nabble.com/CDCR-how-to-deal-with-the-transaction-log-
> files-tp4345062p4346138.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: CDCR - how to deal with the transaction log files
Posted by jmyatt <jm...@wayfair.com>.
Thanks for the suggestion - tried that today and still no luck. Time to
write a script to naively / blindly delete old logs and run that in cron.
*sigh*
--
View this message in context: http://lucene.472066.n3.nabble.com/CDCR-how-to-deal-with-the-transaction-log-files-tp4345062p4346138.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: CDCR - how to deal with the transaction log files
Posted by "Xie, Sean" <Se...@finra.org>.
Try run second data import or any other indexing jobs after the replication of the first data import is completed.
My observation is during the replication period (when there is docs in queue), tlog clean up will not triggered. So when queue is 0, and submit second batch and monitor the queue and tlogs again.
-- Thank you
Sean
From: jmyatt <jm...@wayfair.com>>
Date: Wednesday, Jul 12, 2017, 6:58 PM
To: solr-user@lucene.apache.org <so...@lucene.apache.org>>
Subject: [EXTERNAL] Re: CDCR - how to deal with the transaction log files
glad to hear you found your solution! I have been combing over this post and
others on this discussion board many times and have tried so many tweaks to
configuration, order of steps, etc, all with absolutely no success in
getting the Source cluster tlogs to delete. So incredibly frustrating. If
anyone has other pearls of wisdom I'd love some advice. Quick hits on what
I've tried:
- solrconfig exactly like Sean's (target and source respectively) expect no
autoSoftCommit
- I am also calling cdcr?action=DISABLEBUFFER (on source as well as on
target) explicitly before starting since the config setting of
defaultState=disabled doesn't seem to work
- when I create the collection on source first, I get the warning "The log
reader for target collection {collection name} is not initialised". When I
reverse the order (create the collection on target first), no such warning
- tlogs replicate as expected, hard commits on both target and source cause
tlogs to rollover, etc - all of that works as expected
- action=QUEUES on source reflects the queueSize accurately. Also *always*
shows updateLogSynchronizer state as "stopped"
- action=LASTPROCESSEDVERSION on both source and target always seems correct
(I don't see the -1 that Sean mentioned).
- I'm creating new collections every time and running full data imports that
take 5-10 minutes. Again, all data replication, log rollover, and autocommit
activity seems to work as expected, and logs on target are deleted. It's
just those pesky source tlogs I can't get to delete.
--
View this message in context: http://lucene.472066.n3.nabble.com/CDCR-how-to-deal-with-the-transaction-log-files-tp4345062p4345715.html
Sent from the Solr - User mailing list archive at Nabble.com.
Confidentiality Notice:: This email, including attachments, may include non-public, proprietary, confidential or legally privileged information. If you are not an intended recipient or an authorized agent of an intended recipient, you are hereby notified that any dissemination, distribution or copying of the information contained in or transmitted with this e-mail is unauthorized and strictly prohibited. If you have received this email in error, please notify the sender by replying to this message and permanently delete this e-mail, its attachments, and any copies of it immediately. You should not retain, copy or use this e-mail or any attachment for any purpose, nor disclose all or any part of the contents to any other person. Thank you.
RE: CDCR - how to deal with the transaction log files
Posted by "Xie, Sean" <Se...@finra.org>.
You don't need to start cdcr on target cluster. Other steps are exactly what I did. After disable buffer on both target and source, the tlog files are purged according to the specs.
-- Thank you
Sean
From: Patrick Hoeffel <pa...@polarisalpha.com>>
Date: Friday, Jul 28, 2017, 4:01 PM
To: solr-user@lucene.apache.org <so...@lucene.apache.org>>
Cc: jmyatt@wayfair.com <jm...@wayfair.com>>
Subject: [EXTERNAL] RE: CDCR - how to deal with the transaction log files
Amrit,
Problem solved! My biggest mistake was in my SOURCE-side configuration. The zkHost field needed the entire zkHost string, including the CHROOT indicator. I suppose that should have been obvious to me, but the examples only showed the IP Address of the target ZK, and I made a poor assumption.
<!-- CORRECT CDCR CONFIG SECTION -->
<requestHandler name="/cdcr" class="solr.CdcrRequestHandler">
<lst name="replica">
<str name="zkHost">10.161.0.7:2181,10.161.0.6:2181,10.161.0.5:2181/chroot/solr</str>
<str name="source">ks_v1</str>
<str name="target">ks_v1</str>
</lst>
<!-- MY ORIGINAL INCORRECT CDCR CONFIG SECTION -->
<requestHandler name="/cdcr" class="solr.CdcrRequestHandler">
<lst name="replica">
<str name="zkHost">10.161.0.7:2181</str> <=== Problem was here.
<str name="source">ks_v1</str>
<str name="target">ks_v1</str>
</lst>
After that, I just made sure I did this:
1. Stop all Solr nodes at both SOURCE and TARGET.
2. $ rm -rf $SOLR_HOME/server/solr/collection_name/data/tlog/*.*
3. On the TARGET:
a. $ collection/cdcr?action=DISABLEBUFFER
b. $ collection/cdcr?action=START
4. On the Source:
a. $ collection/cdcr?action=DISABLEBUFFER
b. $ collection/cdcr?action=START
At this point any existing data in the SOURCE collection started flowing into the TARGET collection, and it has remained congruent ever since.
Thanks,
Patrick Hoeffel
Senior Software Engineer
(Direct) 719-452-7371
(Mobile) 719-210-3706
patrick.hoeffel@polarisalpha.com
PolarisAlpha.com
-----Original Message-----
From: Amrit Sarkar [mailto:sarkaramrit2@gmail.com]
Sent: Friday, July 21, 2017 7:21 AM
To: solr-user@lucene.apache.org
Cc: jmyatt@wayfair.com
Subject: Re: CDCR - how to deal with the transaction log files
Patrick,
Yes! You created default UpdateLog which got written to a disk and then you changed it to CdcrUpdateLog in configs. I find no reason it would create a proper COLLECTIONCHECKPOINT on target tlog.
One thing you can try before creating / starting from scratch is restarting source cluster nodes, the leaders of shard will try to create the same COLLECTIONCHECKPOINT, which may or may not be successful.
Amrit Sarkar
Search Engineer
Lucidworks, Inc.
415-589-9269
www.lucidworks.com<http://www.lucidworks.com>
Twitter http://twitter.com/lucidworks
LinkedIn: https://www.linkedin.com/in/sarkaramrit2
On Fri, Jul 21, 2017 at 11:09 AM, Patrick Hoeffel < patrick.hoeffel@polarisalpha.com> wrote:
> I'm working on my first setup of CDCR, and I'm seeing the same "The
> log reader for target collection {collection name} is not initialised"
> as you saw.
>
> It looks like you're creating collections on a regular basis, but for
> me, I create it one time and never again. I've been creating the
> collection first from defaults and then applying the CDCR-aware
> solrconfig changes afterward. It sounds like maybe I need to create
> the configset in ZK first, then create the collections, first on the
> Target and then on the Source, and I should be good?
>
> Thanks,
>
> Patrick Hoeffel
> Senior Software Engineer
> (Direct) 719-452-7371
> (Mobile) 719-210-3706
> patrick.hoeffel@polarisalpha.com
> PolarisAlpha.com
>
>
> -----Original Message-----
> From: jmyatt [mailto:jmyatt@wayfair.com]
> Sent: Wednesday, July 12, 2017 4:49 PM
> To: solr-user@lucene.apache.org
> Subject: Re: CDCR - how to deal with the transaction log files
>
> glad to hear you found your solution! I have been combing over this
> post and others on this discussion board many times and have tried so
> many tweaks to configuration, order of steps, etc, all with absolutely
> no success in getting the Source cluster tlogs to delete. So
> incredibly frustrating. If anyone has other pearls of wisdom I'd love some advice.
> Quick hits on what I've tried:
>
> - solrconfig exactly like Sean's (target and source respectively)
> expect no autoSoftCommit
> - I am also calling cdcr?action=DISABLEBUFFER (on source as well as on
> target) explicitly before starting since the config setting of
> defaultState=disabled doesn't seem to work
> - when I create the collection on source first, I get the warning "The
> log reader for target collection {collection name} is not
> initialised". When I reverse the order (create the collection on
> target first), no such warning
> - tlogs replicate as expected, hard commits on both target and source
> cause tlogs to rollover, etc - all of that works as expected
> - action=QUEUES on source reflects the queueSize accurately. Also
> *always* shows updateLogSynchronizer state as "stopped"
> - action=LASTPROCESSEDVERSION on both source and target always seems
> correct (I don't see the -1 that Sean mentioned).
> - I'm creating new collections every time and running full data
> imports that take 5-10 minutes. Again, all data replication, log
> rollover, and autocommit activity seems to work as expected, and logs
> on target are deleted. It's just those pesky source tlogs I can't get to delete.
>
>
>
> --
> View this message in context: http://lucene.472066.n3.
> nabble.com/CDCR-how-to-deal-with-the-transaction-log-
> files-tp4345062p4345715.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Confidentiality Notice:: This email, including attachments, may include non-public, proprietary, confidential or legally privileged information. If you are not an intended recipient or an authorized agent of an intended recipient, you are hereby notified that any dissemination, distribution or copying of the information contained in or transmitted with this e-mail is unauthorized and strictly prohibited. If you have received this email in error, please notify the sender by replying to this message and permanently delete this e-mail, its attachments, and any copies of it immediately. You should not retain, copy or use this e-mail or any attachment for any purpose, nor disclose all or any part of the contents to any other person. Thank you.
RE: CDCR - how to deal with the transaction log files
Posted by Patrick Hoeffel <pa...@polarisalpha.com>.
Amrit,
Problem solved! My biggest mistake was in my SOURCE-side configuration. The zkHost field needed the entire zkHost string, including the CHROOT indicator. I suppose that should have been obvious to me, but the examples only showed the IP Address of the target ZK, and I made a poor assumption.
<!-- CORRECT CDCR CONFIG SECTION -->
<requestHandler name="/cdcr" class="solr.CdcrRequestHandler">
<lst name="replica">
<str name="zkHost">10.161.0.7:2181,10.161.0.6:2181,10.161.0.5:2181/chroot/solr</str>
<str name="source">ks_v1</str>
<str name="target">ks_v1</str>
</lst>
<!-- MY ORIGINAL INCORRECT CDCR CONFIG SECTION -->
<requestHandler name="/cdcr" class="solr.CdcrRequestHandler">
<lst name="replica">
<str name="zkHost">10.161.0.7:2181</str> <=== Problem was here.
<str name="source">ks_v1</str>
<str name="target">ks_v1</str>
</lst>
After that, I just made sure I did this:
1. Stop all Solr nodes at both SOURCE and TARGET.
2. $ rm -rf $SOLR_HOME/server/solr/collection_name/data/tlog/*.*
3. On the TARGET:
a. $ collection/cdcr?action=DISABLEBUFFER
b. $ collection/cdcr?action=START
4. On the Source:
a. $ collection/cdcr?action=DISABLEBUFFER
b. $ collection/cdcr?action=START
At this point any existing data in the SOURCE collection started flowing into the TARGET collection, and it has remained congruent ever since.
Thanks,
Patrick Hoeffel
Senior Software Engineer
(Direct) 719-452-7371
(Mobile) 719-210-3706
patrick.hoeffel@polarisalpha.com
PolarisAlpha.com
-----Original Message-----
From: Amrit Sarkar [mailto:sarkaramrit2@gmail.com]
Sent: Friday, July 21, 2017 7:21 AM
To: solr-user@lucene.apache.org
Cc: jmyatt@wayfair.com
Subject: Re: CDCR - how to deal with the transaction log files
Patrick,
Yes! You created default UpdateLog which got written to a disk and then you changed it to CdcrUpdateLog in configs. I find no reason it would create a proper COLLECTIONCHECKPOINT on target tlog.
One thing you can try before creating / starting from scratch is restarting source cluster nodes, the leaders of shard will try to create the same COLLECTIONCHECKPOINT, which may or may not be successful.
Amrit Sarkar
Search Engineer
Lucidworks, Inc.
415-589-9269
www.lucidworks.com
Twitter http://twitter.com/lucidworks
LinkedIn: https://www.linkedin.com/in/sarkaramrit2
On Fri, Jul 21, 2017 at 11:09 AM, Patrick Hoeffel < patrick.hoeffel@polarisalpha.com> wrote:
> I'm working on my first setup of CDCR, and I'm seeing the same "The
> log reader for target collection {collection name} is not initialised"
> as you saw.
>
> It looks like you're creating collections on a regular basis, but for
> me, I create it one time and never again. I've been creating the
> collection first from defaults and then applying the CDCR-aware
> solrconfig changes afterward. It sounds like maybe I need to create
> the configset in ZK first, then create the collections, first on the
> Target and then on the Source, and I should be good?
>
> Thanks,
>
> Patrick Hoeffel
> Senior Software Engineer
> (Direct) 719-452-7371
> (Mobile) 719-210-3706
> patrick.hoeffel@polarisalpha.com
> PolarisAlpha.com
>
>
> -----Original Message-----
> From: jmyatt [mailto:jmyatt@wayfair.com]
> Sent: Wednesday, July 12, 2017 4:49 PM
> To: solr-user@lucene.apache.org
> Subject: Re: CDCR - how to deal with the transaction log files
>
> glad to hear you found your solution! I have been combing over this
> post and others on this discussion board many times and have tried so
> many tweaks to configuration, order of steps, etc, all with absolutely
> no success in getting the Source cluster tlogs to delete. So
> incredibly frustrating. If anyone has other pearls of wisdom I'd love some advice.
> Quick hits on what I've tried:
>
> - solrconfig exactly like Sean's (target and source respectively)
> expect no autoSoftCommit
> - I am also calling cdcr?action=DISABLEBUFFER (on source as well as on
> target) explicitly before starting since the config setting of
> defaultState=disabled doesn't seem to work
> - when I create the collection on source first, I get the warning "The
> log reader for target collection {collection name} is not
> initialised". When I reverse the order (create the collection on
> target first), no such warning
> - tlogs replicate as expected, hard commits on both target and source
> cause tlogs to rollover, etc - all of that works as expected
> - action=QUEUES on source reflects the queueSize accurately. Also
> *always* shows updateLogSynchronizer state as "stopped"
> - action=LASTPROCESSEDVERSION on both source and target always seems
> correct (I don't see the -1 that Sean mentioned).
> - I'm creating new collections every time and running full data
> imports that take 5-10 minutes. Again, all data replication, log
> rollover, and autocommit activity seems to work as expected, and logs
> on target are deleted. It's just those pesky source tlogs I can't get to delete.
>
>
>
> --
> View this message in context: http://lucene.472066.n3.
> nabble.com/CDCR-how-to-deal-with-the-transaction-log-
> files-tp4345062p4345715.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: CDCR - how to deal with the transaction log files
Posted by Amrit Sarkar <sa...@gmail.com>.
Patrick,
Yes! You created default UpdateLog which got written to a disk and then you
changed it to CdcrUpdateLog in configs. I find no reason it would create a
proper COLLECTIONCHECKPOINT on target tlog.
One thing you can try before creating / starting from scratch is restarting
source cluster nodes, the leaders of shard will try to create the same
COLLECTIONCHECKPOINT, which may or may not be successful.
Amrit Sarkar
Search Engineer
Lucidworks, Inc.
415-589-9269
www.lucidworks.com
Twitter http://twitter.com/lucidworks
LinkedIn: https://www.linkedin.com/in/sarkaramrit2
On Fri, Jul 21, 2017 at 11:09 AM, Patrick Hoeffel <
patrick.hoeffel@polarisalpha.com> wrote:
> I'm working on my first setup of CDCR, and I'm seeing the same "The log
> reader for target collection {collection name} is not initialised" as you
> saw.
>
> It looks like you're creating collections on a regular basis, but for me,
> I create it one time and never again. I've been creating the collection
> first from defaults and then applying the CDCR-aware solrconfig changes
> afterward. It sounds like maybe I need to create the configset in ZK first,
> then create the collections, first on the Target and then on the Source,
> and I should be good?
>
> Thanks,
>
> Patrick Hoeffel
> Senior Software Engineer
> (Direct) 719-452-7371
> (Mobile) 719-210-3706
> patrick.hoeffel@polarisalpha.com
> PolarisAlpha.com
>
>
> -----Original Message-----
> From: jmyatt [mailto:jmyatt@wayfair.com]
> Sent: Wednesday, July 12, 2017 4:49 PM
> To: solr-user@lucene.apache.org
> Subject: Re: CDCR - how to deal with the transaction log files
>
> glad to hear you found your solution! I have been combing over this post
> and others on this discussion board many times and have tried so many
> tweaks to configuration, order of steps, etc, all with absolutely no
> success in getting the Source cluster tlogs to delete. So incredibly
> frustrating. If anyone has other pearls of wisdom I'd love some advice.
> Quick hits on what I've tried:
>
> - solrconfig exactly like Sean's (target and source respectively) expect
> no autoSoftCommit
> - I am also calling cdcr?action=DISABLEBUFFER (on source as well as on
> target) explicitly before starting since the config setting of
> defaultState=disabled doesn't seem to work
> - when I create the collection on source first, I get the warning "The log
> reader for target collection {collection name} is not initialised". When I
> reverse the order (create the collection on target first), no such warning
> - tlogs replicate as expected, hard commits on both target and source
> cause tlogs to rollover, etc - all of that works as expected
> - action=QUEUES on source reflects the queueSize accurately. Also
> *always* shows updateLogSynchronizer state as "stopped"
> - action=LASTPROCESSEDVERSION on both source and target always seems
> correct (I don't see the -1 that Sean mentioned).
> - I'm creating new collections every time and running full data imports
> that take 5-10 minutes. Again, all data replication, log rollover, and
> autocommit activity seems to work as expected, and logs on target are
> deleted. It's just those pesky source tlogs I can't get to delete.
>
>
>
> --
> View this message in context: http://lucene.472066.n3.
> nabble.com/CDCR-how-to-deal-with-the-transaction-log-
> files-tp4345062p4345715.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
RE: CDCR - how to deal with the transaction log files
Posted by Patrick Hoeffel <pa...@polarisalpha.com>.
I'm working on my first setup of CDCR, and I'm seeing the same "The log reader for target collection {collection name} is not initialised" as you saw.
It looks like you're creating collections on a regular basis, but for me, I create it one time and never again. I've been creating the collection first from defaults and then applying the CDCR-aware solrconfig changes afterward. It sounds like maybe I need to create the configset in ZK first, then create the collections, first on the Target and then on the Source, and I should be good?
Thanks,
Patrick Hoeffel
Senior Software Engineer
(Direct) 719-452-7371
(Mobile) 719-210-3706
patrick.hoeffel@polarisalpha.com
PolarisAlpha.com
-----Original Message-----
From: jmyatt [mailto:jmyatt@wayfair.com]
Sent: Wednesday, July 12, 2017 4:49 PM
To: solr-user@lucene.apache.org
Subject: Re: CDCR - how to deal with the transaction log files
glad to hear you found your solution! I have been combing over this post and others on this discussion board many times and have tried so many tweaks to configuration, order of steps, etc, all with absolutely no success in getting the Source cluster tlogs to delete. So incredibly frustrating. If anyone has other pearls of wisdom I'd love some advice. Quick hits on what I've tried:
- solrconfig exactly like Sean's (target and source respectively) expect no autoSoftCommit
- I am also calling cdcr?action=DISABLEBUFFER (on source as well as on
target) explicitly before starting since the config setting of defaultState=disabled doesn't seem to work
- when I create the collection on source first, I get the warning "The log reader for target collection {collection name} is not initialised". When I reverse the order (create the collection on target first), no such warning
- tlogs replicate as expected, hard commits on both target and source cause tlogs to rollover, etc - all of that works as expected
- action=QUEUES on source reflects the queueSize accurately. Also *always* shows updateLogSynchronizer state as "stopped"
- action=LASTPROCESSEDVERSION on both source and target always seems correct (I don't see the -1 that Sean mentioned).
- I'm creating new collections every time and running full data imports that take 5-10 minutes. Again, all data replication, log rollover, and autocommit activity seems to work as expected, and logs on target are deleted. It's just those pesky source tlogs I can't get to delete.
--
View this message in context: http://lucene.472066.n3.nabble.com/CDCR-how-to-deal-with-the-transaction-log-files-tp4345062p4345715.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: CDCR - how to deal with the transaction log files
Posted by jmyatt <jm...@wayfair.com>.
glad to hear you found your solution! I have been combing over this post and
others on this discussion board many times and have tried so many tweaks to
configuration, order of steps, etc, all with absolutely no success in
getting the Source cluster tlogs to delete. So incredibly frustrating. If
anyone has other pearls of wisdom I'd love some advice. Quick hits on what
I've tried:
- solrconfig exactly like Sean's (target and source respectively) expect no
autoSoftCommit
- I am also calling cdcr?action=DISABLEBUFFER (on source as well as on
target) explicitly before starting since the config setting of
defaultState=disabled doesn't seem to work
- when I create the collection on source first, I get the warning "The log
reader for target collection {collection name} is not initialised". When I
reverse the order (create the collection on target first), no such warning
- tlogs replicate as expected, hard commits on both target and source cause
tlogs to rollover, etc - all of that works as expected
- action=QUEUES on source reflects the queueSize accurately. Also *always*
shows updateLogSynchronizer state as "stopped"
- action=LASTPROCESSEDVERSION on both source and target always seems correct
(I don't see the -1 that Sean mentioned).
- I'm creating new collections every time and running full data imports that
take 5-10 minutes. Again, all data replication, log rollover, and autocommit
activity seems to work as expected, and logs on target are deleted. It's
just those pesky source tlogs I can't get to delete.
--
View this message in context: http://lucene.472066.n3.nabble.com/CDCR-how-to-deal-with-the-transaction-log-files-tp4345062p4345715.html
Sent from the Solr - User mailing list archive at Nabble.com.