You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Webster Homer <we...@sial.com> on 2016/12/22 15:10:16 UTC

CDCR logging is Needlessly verbose, fills up the file system fast

While testing CDCR I found that it is writing tons of log messages per
second. Example:
2016-12-21 23:24:41.652 INFO  (qtp110456297-13) [c:sial-catalog-material
s:shard1 r:core_node1 x:sial-catalog-material_shard1_replica1]
o.a.s.c.S.Request [sial-catalog-material_shard1_replica1]  webapp=/solr
path=/cdcr params={qt=/cdcr&action=BOOTSTRAP_STATUS&wt=javabin&version=2}
status=0 QTime=0
2016-12-21 23:24:41.653 INFO  (qtp110456297-18) [c:sial-catalog-material
s:shard1 r:core_node1 x:sial-catalog-material_shard1_replica1]
o.a.s.c.S.Request [sial-catalog-material_shard1_replica1]  webapp=/solr
path=/cdcr params={qt=/cdcr&action=BOOTSTRAP_STATUS&wt=javabin&version=2}
status=0 QTime=0
2016-12-21 23:24:41.655 INFO  (qtp110456297-14) [c:sial-catalog-material
s:shard1 r:core_node1 x:sial-catalog-material_shard1_replica1]
o.a.s.c.S.Request [sial-catalog-material_shard1_replica1]  webapp=/solr
path=/cdcr params={qt=/cdcr&action=BOOTSTRAP_STATUS&wt=javabin&version=2}
status=0 QTime=0
2016-12-21 23:24:41.657 INFO  (qtp110456297-17) [c:sial-catalog-material
s:shard1 r:core_node1 x:sial-catalog-material_shard1_replica1]
o.a.s.c.S.Request [sial-catalog-material_shard1_replica1]  webapp=/solr
path=/cdcr params={qt=/cdcr&action=BOOTSTRAP_STATUS&wt=javabin&version=2}
status=0 QTime=0


These should be DEBUG messages and NOT INFO messages. Is there a way to
selectively turn them off?  The above is from a Target collection, it is
even worse on the Source side.

I'd rather not change my logging level as most INFO messages are useful.

This is a very poor default logging level for these messages.

-- 


This message and any attachment are confidential and may be privileged or 
otherwise protected from disclosure. If you are not the intended recipient, 
you must not copy this message or attachment or disclose the contents to 
any other person. If you have received this transmission in error, please 
notify the sender immediately and delete the message and any attachment 
from your system. Merck KGaA, Darmstadt, Germany and any of its 
subsidiaries do not accept liability for any omissions or errors in this 
message which may arise as a result of E-Mail-transmission or for damages 
resulting from any unauthorized changes of the content of this message and 
any attachment thereto. Merck KGaA, Darmstadt, Germany and any of its 
subsidiaries do not guarantee that this message is free of viruses and does 
not accept liability for any damages caused by any virus transmitted 
therewith.

Click http://www.merckgroup.com/disclaimer to access the German, French, 
Spanish and Portuguese versions of this disclaimer.

Re: CDCR logging is Needlessly verbose, fills up the file system fast

Posted by Webster Homer <we...@sial.com>.
I know that we have never set the schedule parameter to 1 millisecond. We
have specified either 100 or 1000. I wondered why it was writing so
frequently. I suspect a bug somewhere

However, we will have multiple collections using cdcr, and in some cases
the source collection will have multiple targets.

I understand your point about the request handler, I think being able to
specify messaging levels for specific handlers is essential

On Mon, Jan 9, 2017 at 10:07 PM, Shawn Heisey <ap...@elyograg.org> wrote:

> On 12/22/2016 8:10 AM, Webster Homer wrote:
> > While testing CDCR I found that it is writing tons of log messages per
> > second. Example:
> > 2016-12-21 23:24:41.652 INFO  (qtp110456297-13) [c:sial-catalog-material
> > s:shard1 r:core_node1 x:sial-catalog-material_shard1_replica1]
> > o.a.s.c.S.Request [sial-catalog-material_shard1_replica1]  webapp=/solr
> > path=/cdcr params={qt=/cdcr&action=BOOTSTRAP_STATUS&wt=javabin&
> version=2}
> > status=0 QTime=0
> > 2016-12-21 23:24:41.653 INFO  (qtp110456297-18) [c:sial-catalog-material
> > s:shard1 r:core_node1 x:sial-catalog-material_shard1_replica1]
> > o.a.s.c.S.Request [sial-catalog-material_shard1_replica1]  webapp=/solr
> > path=/cdcr params={qt=/cdcr&action=BOOTSTRAP_STATUS&wt=javabin&
> version=2}
> > status=0 QTime=0
>
> I hadn't looked closely at the messages you were seeing in your logs
> until now.
>
> These messages are *request* logging.  This is the same code path that
> logs every query -- it's not specific to CDCR.  It's just logging all
> the requests that Solr is receiving.  If this log message were changed
> to DEBUG, then Solr would not log queries by default.  A large number of
> Solr users want that logging.
>
> I think that you could probably avoid seeing these logs by configuring
> log4j to not log things tagged
> asorg.apache.solr.core.SolrCore.Request(even though it's not a real
> class, I think log4j can still configure it) ... but then you wouldn't
> get your queries logged either.
>
> In order to not log these particular messages, but still log queries and
> other requests, the request logging code will need to have a way to
> specify that certain messages should not be logged.  This might be
> something thatcould beconfigurable at the request handler definition
> level -- put something in the requestHandler configuration (for /cdcr in
> this case) that tells it to skip logging.  That seems like a good
> feature to have.
>
> After looking at the CDCR configuration page in the reference guide, I
> might have a little more insight.  You're getting one of these logs
> every 1-2 milliseconds ... so it sounds like you have configured the
> CDCR with a schedule of one millisecond.  The default value for the
> replicator schedule is is ten milliseconds, and the update log
> synchronizer defaults to a full minute.  I'm guessing that CDCR is not
> designed to have such a low schedule value.  I would personally
> configure the replicator schedule even higher than the default --
> network latency between Internet locations is often longer than ten
> milliseconds.
>
> Thanks,
> Shawn
>
>

-- 


This message and any attachment are confidential and may be privileged or 
otherwise protected from disclosure. If you are not the intended recipient, 
you must not copy this message or attachment or disclose the contents to 
any other person. If you have received this transmission in error, please 
notify the sender immediately and delete the message and any attachment 
from your system. Merck KGaA, Darmstadt, Germany and any of its 
subsidiaries do not accept liability for any omissions or errors in this 
message which may arise as a result of E-Mail-transmission or for damages 
resulting from any unauthorized changes of the content of this message and 
any attachment thereto. Merck KGaA, Darmstadt, Germany and any of its 
subsidiaries do not guarantee that this message is free of viruses and does 
not accept liability for any damages caused by any virus transmitted 
therewith.

Click http://www.emdgroup.com/disclaimer to access the German, French, 
Spanish and Portuguese versions of this disclaimer.

Re: CDCR logging is Needlessly verbose, fills up the file system fast

Posted by Shawn Heisey <ap...@elyograg.org>.
On 12/22/2016 8:10 AM, Webster Homer wrote:
> While testing CDCR I found that it is writing tons of log messages per
> second. Example:
> 2016-12-21 23:24:41.652 INFO  (qtp110456297-13) [c:sial-catalog-material
> s:shard1 r:core_node1 x:sial-catalog-material_shard1_replica1]
> o.a.s.c.S.Request [sial-catalog-material_shard1_replica1]  webapp=/solr
> path=/cdcr params={qt=/cdcr&action=BOOTSTRAP_STATUS&wt=javabin&version=2}
> status=0 QTime=0
> 2016-12-21 23:24:41.653 INFO  (qtp110456297-18) [c:sial-catalog-material
> s:shard1 r:core_node1 x:sial-catalog-material_shard1_replica1]
> o.a.s.c.S.Request [sial-catalog-material_shard1_replica1]  webapp=/solr
> path=/cdcr params={qt=/cdcr&action=BOOTSTRAP_STATUS&wt=javabin&version=2}
> status=0 QTime=0

I hadn't looked closely at the messages you were seeing in your logs
until now.

These messages are *request* logging.  This is the same code path that
logs every query -- it's not specific to CDCR.  It's just logging all
the requests that Solr is receiving.  If this log message were changed
to DEBUG, then Solr would not log queries by default.  A large number of
Solr users want that logging.

I think that you could probably avoid seeing these logs by configuring
log4j to not log things tagged
asorg.apache.solr.core.SolrCore.Request(even though it's not a real
class, I think log4j can still configure it) ... but then you wouldn't
get your queries logged either.

In order to not log these particular messages, but still log queries and
other requests, the request logging code will need to have a way to
specify that certain messages should not be logged.  This might be
something thatcould beconfigurable at the request handler definition
level -- put something in the requestHandler configuration (for /cdcr in
this case) that tells it to skip logging.  That seems like a good
feature to have.

After looking at the CDCR configuration page in the reference guide, I
might have a little more insight.  You're getting one of these logs
every 1-2 milliseconds ... so it sounds like you have configured the
CDCR with a schedule of one millisecond.  The default value for the
replicator schedule is is ten milliseconds, and the update log
synchronizer defaults to a full minute.  I'm guessing that CDCR is not
designed to have such a low schedule value.  I would personally
configure the replicator schedule even higher than the default --
network latency between Internet locations is often longer than ten
milliseconds.

Thanks,
Shawn


Re: CDCR logging is Needlessly verbose, fills up the file system fast

Posted by Shawn Heisey <ap...@elyograg.org>.
On 1/6/2017 8:21 AM, Webster Homer wrote:
> I figured our problem with the filesystem, by default the root logger
> is configured with the CONSOLE logger, which is NOT rotated and
> eventually filled up the file system. That doesn't exonerate the CDCR
> logging problem though. The thing writes a huge amount of junk to the
> logs, information that looks more like Debug and even TRACE level
> statements. I hope that this is addressed, and soon! There doesn't
> seem to be a way to turn it off either. We want INFO level logging

The logging that goes to the console and solr.log by default is INFO. 
Even without CDCR, INFO logging is fairly verbose, but normally doesn't
chew up a whole bunch of gigabytes over a short timeframe.

CDCR is a pretty new feature, and the authors likely needed a lot of
information in the logs while they were developing it so they could
chase down bugs.  I have never used the feature and haven't looked at
the logging, but it sounds like much of it could be changed to DEBUG or
lower priority now that primary development is done.  If you haven't
already opened an issue in Jira to reduce what CDCR logs at INFO, that
would be a good idea.

To get rid of the console logging entirely, which is a good idea, edit
server/resources/log4j.properties.  Change the "log4j.rootLogger" line
to remove CONSOLE and the comma.  Optionally, you can also remove all
the other lines that contain CONSOLE.

Starting in version 6.3, you can put "-Dsolr.log.muteconsole" on the
java commandline (normally you'd do this in the solr.in.* script) to
accomplish this without changing the logging config.  I personally think
that Solr shouldn't log to the console at all unless it is running in
the foreground and the console isn't being redirected to a file.

https://issues.apache.org/jira/browse/SOLR-8186

Thanks,
Shawn


Re: CDCR logging is Needlessly verbose, fills up the file system fast

Posted by Webster Homer <we...@sial.com>.
I figured our problem with the filesystem, by default the root logger is
configured with the CONSOLE logger, which is NOT rotated and eventually
filled up the file system. That doesn't exonerate the CDCR logging problem
though. The thing writes a huge amount of junk to the logs, information
that looks more like Debug and even TRACE level statements. I hope that
this is addressed, and soon! There doesn't seem to be a way to turn it off
either. We want INFO level logging

On Tue, Jan 3, 2017 at 5:10 PM, Shawn Heisey <ap...@elyograg.org> wrote:

> On 1/3/2017 1:12 PM, Webster Homer wrote:
> > We use the default log4j.properties file which rolls the log file to
> > solr.log.1, solr.log.2 ... which isn't really the problem. What is
> > also happening is that solr.log.1 gets renamed to
> > solr_log_20170103_1110 with a timestamp as the file name. How do I
> > turn off this behavior? It is not obvious in the log4j.properties file.
>
> That rename is not done by log4j.  It is done by the bin/solr or
> bin\solr.cmd script just before Solr is started.  Unlike the solr.log.N
> files, those files are not subject to automatic deletion.
>
> My best guess is that you've got a situation where something external to
> Solr is restarting Solr frequently.  Solr's start script renames
> solr.log to solr_log_DATE_TIME each time it starts.  As far as I am
> aware, Solr does NOT have the ability to restart itself, and the CDCR
> page in the reference guide doesn't mention anything about processes
> being restarted as part of its operation.
>
> Because the logfiles that you are seeing accumulate are not handled by
> log4j, switching to log4j2 and adding compression will not help.  Moving
> to log4j2 requires a few things to be done on the development side, and
> won't be trivial.
>
> If your system requires frequent Solr restarts for some reason, then
> you're going to have to take over management of the renamed solr
> logfiles as well, or edit the start script so that it doesn't rename the
> main logfile.  SolrCloud does not deal well with frequent restarts, so
> avoid doing them.
>
> Thanks,
> Shawn
>
>

-- 


This message and any attachment are confidential and may be privileged or 
otherwise protected from disclosure. If you are not the intended recipient, 
you must not copy this message or attachment or disclose the contents to 
any other person. If you have received this transmission in error, please 
notify the sender immediately and delete the message and any attachment 
from your system. Merck KGaA, Darmstadt, Germany and any of its 
subsidiaries do not accept liability for any omissions or errors in this 
message which may arise as a result of E-Mail-transmission or for damages 
resulting from any unauthorized changes of the content of this message and 
any attachment thereto. Merck KGaA, Darmstadt, Germany and any of its 
subsidiaries do not guarantee that this message is free of viruses and does 
not accept liability for any damages caused by any virus transmitted 
therewith.

Click http://www.emdgroup.com/disclaimer to access the German, French, 
Spanish and Portuguese versions of this disclaimer.

Re: CDCR logging is Needlessly verbose, fills up the file system fast

Posted by Shawn Heisey <ap...@elyograg.org>.
On 1/3/2017 1:12 PM, Webster Homer wrote:
> We use the default log4j.properties file which rolls the log file to
> solr.log.1, solr.log.2 ... which isn't really the problem. What is
> also happening is that solr.log.1 gets renamed to
> solr_log_20170103_1110 with a timestamp as the file name. How do I
> turn off this behavior? It is not obvious in the log4j.properties file.

That rename is not done by log4j.  It is done by the bin/solr or
bin\solr.cmd script just before Solr is started.  Unlike the solr.log.N
files, those files are not subject to automatic deletion.

My best guess is that you've got a situation where something external to
Solr is restarting Solr frequently.  Solr's start script renames
solr.log to solr_log_DATE_TIME each time it starts.  As far as I am
aware, Solr does NOT have the ability to restart itself, and the CDCR
page in the reference guide doesn't mention anything about processes
being restarted as part of its operation.

Because the logfiles that you are seeing accumulate are not handled by
log4j, switching to log4j2 and adding compression will not help.  Moving
to log4j2 requires a few things to be done on the development side, and
won't be trivial.

If your system requires frequent Solr restarts for some reason, then
you're going to have to take over management of the renamed solr
logfiles as well, or edit the start script so that it doesn't rename the
main logfile.  SolrCloud does not deal well with frequent restarts, so
avoid doing them.

Thanks,
Shawn


Re: CDCR logging is Needlessly verbose, fills up the file system fast

Posted by Webster Homer <we...@sial.com>.
So since the logging messages are not from an actual class, how do I
suppress the CDCR INFO messages? I really do want normal INFO messages but
they get swamped by CDCR.

Even when CDCR issues a WARN message, it issues thousands of them.

We use the default log4j.properties file which rolls the log file to
solr.log.1, solr.log.2 ... which isn't really the problem. What is also
happening is that
solr.log.1 gets renamed to solr_log_20170103_1110 with a timestamp as the
file name. How do I turn off this behavior? It is not obvious in the
log4j.properties file.

On a side note is there any plan to move to use log4j2 which would allow
for the rolled files to be compressed?



On Tue, Jan 3, 2017 at 10:13 AM, Alan Woodward <al...@flax.co.uk> wrote:

> It’s org.apache.solr.core.SolrCore.Request - not an actual class.
>
> Alan Woodward
> www.flax.co.uk
>
>
> > On 3 Jan 2017, at 16:08, Webster Homer <we...@sial.com> wrote:
> >
> > I am working on changing the log rotation, but looking at the message:
> >
> > 2016-12-21 23:24:41.653 INFO  (qtp110456297-18) [c:sial-catalog-material
> > s:shard1 r:core_node1 x:sial-catalog-material_shard1_replica1]
> > o.a.s.c.S.Request [sial-catalog-material_shard1_replica1]  webapp=/solr
> > path=/cdcr params={qt=/cdcr&action=BOOTSTRAP_STATUS&wt=javabin&
> version=2}
> > status=0 QTime=0
> >
> > I can't tell which class is generating it
> >
> > On Thu, Dec 29, 2016 at 3:15 PM, Erick Erickson <erickerickson@gmail.com
> >
> > wrote:
> >
> >> Seems like a bandaid would be to insure your Solr logs rotate
> >> appropriately quickly.
> >>
> >> That doesn't address the CDCR loging verbosity, but it might get you by.
> >>
> >> You can also change the logging at the class level by appropriately
> >> editing the
> >> log4j properties file. Again perhaps not the best solution but one
> >> that's immediately
> >> available.
> >>
> >> Best,
> >> Erick
> >>
> >> On Thu, Dec 29, 2016 at 10:37 AM, Webster Homer <webster.homer@sial.com
> >
> >> wrote:
> >>> The logs filled up the file system and caused CDCR to fail due to a
> >>> corrupted Tlog file.
> >>>
> >>> On Thu, Dec 22, 2016 at 9:10 AM, Webster Homer <webster.homer@sial.com
> >
> >>> wrote:
> >>>
> >>>> While testing CDCR I found that it is writing tons of log messages per
> >>>> second. Example:
> >>>> 2016-12-21 23:24:41.652 INFO  (qtp110456297-13)
> [c:sial-catalog-material
> >>>> s:shard1 r:core_node1 x:sial-catalog-material_shard1_replica1]
> >>>> o.a.s.c.S.Request [sial-catalog-material_shard1_replica1]
> webapp=/solr
> >>>> path=/cdcr params={qt=/cdcr&action=BOOTSTRAP_STATUS&wt=javabin&
> >> version=2}
> >>>> status=0 QTime=0
> >>>> 2016-12-21 23:24:41.653 INFO  (qtp110456297-18)
> [c:sial-catalog-material
> >>>> s:shard1 r:core_node1 x:sial-catalog-material_shard1_replica1]
> >>>> o.a.s.c.S.Request [sial-catalog-material_shard1_replica1]
> webapp=/solr
> >>>> path=/cdcr params={qt=/cdcr&action=BOOTSTRAP_STATUS&wt=javabin&
> >> version=2}
> >>>> status=0 QTime=0
> >>>> 2016-12-21 23:24:41.655 INFO  (qtp110456297-14)
> [c:sial-catalog-material
> >>>> s:shard1 r:core_node1 x:sial-catalog-material_shard1_replica1]
> >>>> o.a.s.c.S.Request [sial-catalog-material_shard1_replica1]
> webapp=/solr
> >>>> path=/cdcr params={qt=/cdcr&action=BOOTSTRAP_STATUS&wt=javabin&
> >> version=2}
> >>>> status=0 QTime=0
> >>>> 2016-12-21 23:24:41.657 INFO  (qtp110456297-17)
> [c:sial-catalog-material
> >>>> s:shard1 r:core_node1 x:sial-catalog-material_shard1_replica1]
> >>>> o.a.s.c.S.Request [sial-catalog-material_shard1_replica1]
> webapp=/solr
> >>>> path=/cdcr params={qt=/cdcr&action=BOOTSTRAP_STATUS&wt=javabin&
> >> version=2}
> >>>> status=0 QTime=0
> >>>>
> >>>>
> >>>> These should be DEBUG messages and NOT INFO messages. Is there a way
> to
> >>>> selectively turn them off?  The above is from a Target collection, it
> is
> >>>> even worse on the Source side.
> >>>>
> >>>> I'd rather not change my logging level as most INFO messages are
> useful.
> >>>>
> >>>> This is a very poor default logging level for these messages.
> >>>>
> >>>
> >>> --
> >>>
> >>>
> >>> This message and any attachment are confidential and may be privileged
> or
> >>> otherwise protected from disclosure. If you are not the intended
> >> recipient,
> >>> you must not copy this message or attachment or disclose the contents
> to
> >>> any other person. If you have received this transmission in error,
> please
> >>> notify the sender immediately and delete the message and any attachment
> >>> from your system. Merck KGaA, Darmstadt, Germany and any of its
> >>> subsidiaries do not accept liability for any omissions or errors in
> this
> >>> message which may arise as a result of E-Mail-transmission or for
> damages
> >>> resulting from any unauthorized changes of the content of this message
> >> and
> >>> any attachment thereto. Merck KGaA, Darmstadt, Germany and any of its
> >>> subsidiaries do not guarantee that this message is free of viruses and
> >> does
> >>> not accept liability for any damages caused by any virus transmitted
> >>> therewith.
> >>>
> >>> Click http://www.merckgroup.com/disclaimer to access the German,
> French,
> >>> Spanish and Portuguese versions of this disclaimer.
> >>
> >
> > --
> >
> >
> > This message and any attachment are confidential and may be privileged or
> > otherwise protected from disclosure. If you are not the intended
> recipient,
> > you must not copy this message or attachment or disclose the contents to
> > any other person. If you have received this transmission in error, please
> > notify the sender immediately and delete the message and any attachment
> > from your system. Merck KGaA, Darmstadt, Germany and any of its
> > subsidiaries do not accept liability for any omissions or errors in this
> > message which may arise as a result of E-Mail-transmission or for damages
> > resulting from any unauthorized changes of the content of this message
> and
> > any attachment thereto. Merck KGaA, Darmstadt, Germany and any of its
> > subsidiaries do not guarantee that this message is free of viruses and
> does
> > not accept liability for any damages caused by any virus transmitted
> > therewith.
> >
> > Click http://www.emdgroup.com/disclaimer to access the German, French,
> > Spanish and Portuguese versions of this disclaimer.
>
>

-- 


This message and any attachment are confidential and may be privileged or 
otherwise protected from disclosure. If you are not the intended recipient, 
you must not copy this message or attachment or disclose the contents to 
any other person. If you have received this transmission in error, please 
notify the sender immediately and delete the message and any attachment 
from your system. Merck KGaA, Darmstadt, Germany and any of its 
subsidiaries do not accept liability for any omissions or errors in this 
message which may arise as a result of E-Mail-transmission or for damages 
resulting from any unauthorized changes of the content of this message and 
any attachment thereto. Merck KGaA, Darmstadt, Germany and any of its 
subsidiaries do not guarantee that this message is free of viruses and does 
not accept liability for any damages caused by any virus transmitted 
therewith.

Click http://www.emdgroup.com/disclaimer to access the German, French, 
Spanish and Portuguese versions of this disclaimer.

Re: CDCR logging is Needlessly verbose, fills up the file system fast

Posted by Alan Woodward <al...@flax.co.uk>.
It’s org.apache.solr.core.SolrCore.Request - not an actual class.

Alan Woodward
www.flax.co.uk


> On 3 Jan 2017, at 16:08, Webster Homer <we...@sial.com> wrote:
> 
> I am working on changing the log rotation, but looking at the message:
> 
> 2016-12-21 23:24:41.653 INFO  (qtp110456297-18) [c:sial-catalog-material
> s:shard1 r:core_node1 x:sial-catalog-material_shard1_replica1]
> o.a.s.c.S.Request [sial-catalog-material_shard1_replica1]  webapp=/solr
> path=/cdcr params={qt=/cdcr&action=BOOTSTRAP_STATUS&wt=javabin&version=2}
> status=0 QTime=0
> 
> I can't tell which class is generating it
> 
> On Thu, Dec 29, 2016 at 3:15 PM, Erick Erickson <er...@gmail.com>
> wrote:
> 
>> Seems like a bandaid would be to insure your Solr logs rotate
>> appropriately quickly.
>> 
>> That doesn't address the CDCR loging verbosity, but it might get you by.
>> 
>> You can also change the logging at the class level by appropriately
>> editing the
>> log4j properties file. Again perhaps not the best solution but one
>> that's immediately
>> available.
>> 
>> Best,
>> Erick
>> 
>> On Thu, Dec 29, 2016 at 10:37 AM, Webster Homer <we...@sial.com>
>> wrote:
>>> The logs filled up the file system and caused CDCR to fail due to a
>>> corrupted Tlog file.
>>> 
>>> On Thu, Dec 22, 2016 at 9:10 AM, Webster Homer <we...@sial.com>
>>> wrote:
>>> 
>>>> While testing CDCR I found that it is writing tons of log messages per
>>>> second. Example:
>>>> 2016-12-21 23:24:41.652 INFO  (qtp110456297-13) [c:sial-catalog-material
>>>> s:shard1 r:core_node1 x:sial-catalog-material_shard1_replica1]
>>>> o.a.s.c.S.Request [sial-catalog-material_shard1_replica1]  webapp=/solr
>>>> path=/cdcr params={qt=/cdcr&action=BOOTSTRAP_STATUS&wt=javabin&
>> version=2}
>>>> status=0 QTime=0
>>>> 2016-12-21 23:24:41.653 INFO  (qtp110456297-18) [c:sial-catalog-material
>>>> s:shard1 r:core_node1 x:sial-catalog-material_shard1_replica1]
>>>> o.a.s.c.S.Request [sial-catalog-material_shard1_replica1]  webapp=/solr
>>>> path=/cdcr params={qt=/cdcr&action=BOOTSTRAP_STATUS&wt=javabin&
>> version=2}
>>>> status=0 QTime=0
>>>> 2016-12-21 23:24:41.655 INFO  (qtp110456297-14) [c:sial-catalog-material
>>>> s:shard1 r:core_node1 x:sial-catalog-material_shard1_replica1]
>>>> o.a.s.c.S.Request [sial-catalog-material_shard1_replica1]  webapp=/solr
>>>> path=/cdcr params={qt=/cdcr&action=BOOTSTRAP_STATUS&wt=javabin&
>> version=2}
>>>> status=0 QTime=0
>>>> 2016-12-21 23:24:41.657 INFO  (qtp110456297-17) [c:sial-catalog-material
>>>> s:shard1 r:core_node1 x:sial-catalog-material_shard1_replica1]
>>>> o.a.s.c.S.Request [sial-catalog-material_shard1_replica1]  webapp=/solr
>>>> path=/cdcr params={qt=/cdcr&action=BOOTSTRAP_STATUS&wt=javabin&
>> version=2}
>>>> status=0 QTime=0
>>>> 
>>>> 
>>>> These should be DEBUG messages and NOT INFO messages. Is there a way to
>>>> selectively turn them off?  The above is from a Target collection, it is
>>>> even worse on the Source side.
>>>> 
>>>> I'd rather not change my logging level as most INFO messages are useful.
>>>> 
>>>> This is a very poor default logging level for these messages.
>>>> 
>>> 
>>> --
>>> 
>>> 
>>> This message and any attachment are confidential and may be privileged or
>>> otherwise protected from disclosure. If you are not the intended
>> recipient,
>>> you must not copy this message or attachment or disclose the contents to
>>> any other person. If you have received this transmission in error, please
>>> notify the sender immediately and delete the message and any attachment
>>> from your system. Merck KGaA, Darmstadt, Germany and any of its
>>> subsidiaries do not accept liability for any omissions or errors in this
>>> message which may arise as a result of E-Mail-transmission or for damages
>>> resulting from any unauthorized changes of the content of this message
>> and
>>> any attachment thereto. Merck KGaA, Darmstadt, Germany and any of its
>>> subsidiaries do not guarantee that this message is free of viruses and
>> does
>>> not accept liability for any damages caused by any virus transmitted
>>> therewith.
>>> 
>>> Click http://www.merckgroup.com/disclaimer to access the German, French,
>>> Spanish and Portuguese versions of this disclaimer.
>> 
> 
> -- 
> 
> 
> This message and any attachment are confidential and may be privileged or 
> otherwise protected from disclosure. If you are not the intended recipient, 
> you must not copy this message or attachment or disclose the contents to 
> any other person. If you have received this transmission in error, please 
> notify the sender immediately and delete the message and any attachment 
> from your system. Merck KGaA, Darmstadt, Germany and any of its 
> subsidiaries do not accept liability for any omissions or errors in this 
> message which may arise as a result of E-Mail-transmission or for damages 
> resulting from any unauthorized changes of the content of this message and 
> any attachment thereto. Merck KGaA, Darmstadt, Germany and any of its 
> subsidiaries do not guarantee that this message is free of viruses and does 
> not accept liability for any damages caused by any virus transmitted 
> therewith.
> 
> Click http://www.emdgroup.com/disclaimer to access the German, French, 
> Spanish and Portuguese versions of this disclaimer.


Re: CDCR logging is Needlessly verbose, fills up the file system fast

Posted by Webster Homer <we...@sial.com>.
I am working on changing the log rotation, but looking at the message:

2016-12-21 23:24:41.653 INFO  (qtp110456297-18) [c:sial-catalog-material
s:shard1 r:core_node1 x:sial-catalog-material_shard1_replica1]
o.a.s.c.S.Request [sial-catalog-material_shard1_replica1]  webapp=/solr
path=/cdcr params={qt=/cdcr&action=BOOTSTRAP_STATUS&wt=javabin&version=2}
status=0 QTime=0

I can't tell which class is generating it

On Thu, Dec 29, 2016 at 3:15 PM, Erick Erickson <er...@gmail.com>
wrote:

> Seems like a bandaid would be to insure your Solr logs rotate
> appropriately quickly.
>
> That doesn't address the CDCR loging verbosity, but it might get you by.
>
> You can also change the logging at the class level by appropriately
> editing the
> log4j properties file. Again perhaps not the best solution but one
> that's immediately
> available.
>
> Best,
> Erick
>
> On Thu, Dec 29, 2016 at 10:37 AM, Webster Homer <we...@sial.com>
> wrote:
> > The logs filled up the file system and caused CDCR to fail due to a
> > corrupted Tlog file.
> >
> > On Thu, Dec 22, 2016 at 9:10 AM, Webster Homer <we...@sial.com>
> > wrote:
> >
> >> While testing CDCR I found that it is writing tons of log messages per
> >> second. Example:
> >> 2016-12-21 23:24:41.652 INFO  (qtp110456297-13) [c:sial-catalog-material
> >> s:shard1 r:core_node1 x:sial-catalog-material_shard1_replica1]
> >> o.a.s.c.S.Request [sial-catalog-material_shard1_replica1]  webapp=/solr
> >> path=/cdcr params={qt=/cdcr&action=BOOTSTRAP_STATUS&wt=javabin&
> version=2}
> >> status=0 QTime=0
> >> 2016-12-21 23:24:41.653 INFO  (qtp110456297-18) [c:sial-catalog-material
> >> s:shard1 r:core_node1 x:sial-catalog-material_shard1_replica1]
> >> o.a.s.c.S.Request [sial-catalog-material_shard1_replica1]  webapp=/solr
> >> path=/cdcr params={qt=/cdcr&action=BOOTSTRAP_STATUS&wt=javabin&
> version=2}
> >> status=0 QTime=0
> >> 2016-12-21 23:24:41.655 INFO  (qtp110456297-14) [c:sial-catalog-material
> >> s:shard1 r:core_node1 x:sial-catalog-material_shard1_replica1]
> >> o.a.s.c.S.Request [sial-catalog-material_shard1_replica1]  webapp=/solr
> >> path=/cdcr params={qt=/cdcr&action=BOOTSTRAP_STATUS&wt=javabin&
> version=2}
> >> status=0 QTime=0
> >> 2016-12-21 23:24:41.657 INFO  (qtp110456297-17) [c:sial-catalog-material
> >> s:shard1 r:core_node1 x:sial-catalog-material_shard1_replica1]
> >> o.a.s.c.S.Request [sial-catalog-material_shard1_replica1]  webapp=/solr
> >> path=/cdcr params={qt=/cdcr&action=BOOTSTRAP_STATUS&wt=javabin&
> version=2}
> >> status=0 QTime=0
> >>
> >>
> >> These should be DEBUG messages and NOT INFO messages. Is there a way to
> >> selectively turn them off?  The above is from a Target collection, it is
> >> even worse on the Source side.
> >>
> >> I'd rather not change my logging level as most INFO messages are useful.
> >>
> >> This is a very poor default logging level for these messages.
> >>
> >
> > --
> >
> >
> > This message and any attachment are confidential and may be privileged or
> > otherwise protected from disclosure. If you are not the intended
> recipient,
> > you must not copy this message or attachment or disclose the contents to
> > any other person. If you have received this transmission in error, please
> > notify the sender immediately and delete the message and any attachment
> > from your system. Merck KGaA, Darmstadt, Germany and any of its
> > subsidiaries do not accept liability for any omissions or errors in this
> > message which may arise as a result of E-Mail-transmission or for damages
> > resulting from any unauthorized changes of the content of this message
> and
> > any attachment thereto. Merck KGaA, Darmstadt, Germany and any of its
> > subsidiaries do not guarantee that this message is free of viruses and
> does
> > not accept liability for any damages caused by any virus transmitted
> > therewith.
> >
> > Click http://www.merckgroup.com/disclaimer to access the German, French,
> > Spanish and Portuguese versions of this disclaimer.
>

-- 


This message and any attachment are confidential and may be privileged or 
otherwise protected from disclosure. If you are not the intended recipient, 
you must not copy this message or attachment or disclose the contents to 
any other person. If you have received this transmission in error, please 
notify the sender immediately and delete the message and any attachment 
from your system. Merck KGaA, Darmstadt, Germany and any of its 
subsidiaries do not accept liability for any omissions or errors in this 
message which may arise as a result of E-Mail-transmission or for damages 
resulting from any unauthorized changes of the content of this message and 
any attachment thereto. Merck KGaA, Darmstadt, Germany and any of its 
subsidiaries do not guarantee that this message is free of viruses and does 
not accept liability for any damages caused by any virus transmitted 
therewith.

Click http://www.emdgroup.com/disclaimer to access the German, French, 
Spanish and Portuguese versions of this disclaimer.

Re: CDCR logging is Needlessly verbose, fills up the file system fast

Posted by Erick Erickson <er...@gmail.com>.
Seems like a bandaid would be to insure your Solr logs rotate
appropriately quickly.

That doesn't address the CDCR loging verbosity, but it might get you by.

You can also change the logging at the class level by appropriately editing the
log4j properties file. Again perhaps not the best solution but one
that's immediately
available.

Best,
Erick

On Thu, Dec 29, 2016 at 10:37 AM, Webster Homer <we...@sial.com> wrote:
> The logs filled up the file system and caused CDCR to fail due to a
> corrupted Tlog file.
>
> On Thu, Dec 22, 2016 at 9:10 AM, Webster Homer <we...@sial.com>
> wrote:
>
>> While testing CDCR I found that it is writing tons of log messages per
>> second. Example:
>> 2016-12-21 23:24:41.652 INFO  (qtp110456297-13) [c:sial-catalog-material
>> s:shard1 r:core_node1 x:sial-catalog-material_shard1_replica1]
>> o.a.s.c.S.Request [sial-catalog-material_shard1_replica1]  webapp=/solr
>> path=/cdcr params={qt=/cdcr&action=BOOTSTRAP_STATUS&wt=javabin&version=2}
>> status=0 QTime=0
>> 2016-12-21 23:24:41.653 INFO  (qtp110456297-18) [c:sial-catalog-material
>> s:shard1 r:core_node1 x:sial-catalog-material_shard1_replica1]
>> o.a.s.c.S.Request [sial-catalog-material_shard1_replica1]  webapp=/solr
>> path=/cdcr params={qt=/cdcr&action=BOOTSTRAP_STATUS&wt=javabin&version=2}
>> status=0 QTime=0
>> 2016-12-21 23:24:41.655 INFO  (qtp110456297-14) [c:sial-catalog-material
>> s:shard1 r:core_node1 x:sial-catalog-material_shard1_replica1]
>> o.a.s.c.S.Request [sial-catalog-material_shard1_replica1]  webapp=/solr
>> path=/cdcr params={qt=/cdcr&action=BOOTSTRAP_STATUS&wt=javabin&version=2}
>> status=0 QTime=0
>> 2016-12-21 23:24:41.657 INFO  (qtp110456297-17) [c:sial-catalog-material
>> s:shard1 r:core_node1 x:sial-catalog-material_shard1_replica1]
>> o.a.s.c.S.Request [sial-catalog-material_shard1_replica1]  webapp=/solr
>> path=/cdcr params={qt=/cdcr&action=BOOTSTRAP_STATUS&wt=javabin&version=2}
>> status=0 QTime=0
>>
>>
>> These should be DEBUG messages and NOT INFO messages. Is there a way to
>> selectively turn them off?  The above is from a Target collection, it is
>> even worse on the Source side.
>>
>> I'd rather not change my logging level as most INFO messages are useful.
>>
>> This is a very poor default logging level for these messages.
>>
>
> --
>
>
> This message and any attachment are confidential and may be privileged or
> otherwise protected from disclosure. If you are not the intended recipient,
> you must not copy this message or attachment or disclose the contents to
> any other person. If you have received this transmission in error, please
> notify the sender immediately and delete the message and any attachment
> from your system. Merck KGaA, Darmstadt, Germany and any of its
> subsidiaries do not accept liability for any omissions or errors in this
> message which may arise as a result of E-Mail-transmission or for damages
> resulting from any unauthorized changes of the content of this message and
> any attachment thereto. Merck KGaA, Darmstadt, Germany and any of its
> subsidiaries do not guarantee that this message is free of viruses and does
> not accept liability for any damages caused by any virus transmitted
> therewith.
>
> Click http://www.merckgroup.com/disclaimer to access the German, French,
> Spanish and Portuguese versions of this disclaimer.

Re: CDCR logging is Needlessly verbose, fills up the file system fast

Posted by Webster Homer <we...@sial.com>.
The logs filled up the file system and caused CDCR to fail due to a
corrupted Tlog file.

On Thu, Dec 22, 2016 at 9:10 AM, Webster Homer <we...@sial.com>
wrote:

> While testing CDCR I found that it is writing tons of log messages per
> second. Example:
> 2016-12-21 23:24:41.652 INFO  (qtp110456297-13) [c:sial-catalog-material
> s:shard1 r:core_node1 x:sial-catalog-material_shard1_replica1]
> o.a.s.c.S.Request [sial-catalog-material_shard1_replica1]  webapp=/solr
> path=/cdcr params={qt=/cdcr&action=BOOTSTRAP_STATUS&wt=javabin&version=2}
> status=0 QTime=0
> 2016-12-21 23:24:41.653 INFO  (qtp110456297-18) [c:sial-catalog-material
> s:shard1 r:core_node1 x:sial-catalog-material_shard1_replica1]
> o.a.s.c.S.Request [sial-catalog-material_shard1_replica1]  webapp=/solr
> path=/cdcr params={qt=/cdcr&action=BOOTSTRAP_STATUS&wt=javabin&version=2}
> status=0 QTime=0
> 2016-12-21 23:24:41.655 INFO  (qtp110456297-14) [c:sial-catalog-material
> s:shard1 r:core_node1 x:sial-catalog-material_shard1_replica1]
> o.a.s.c.S.Request [sial-catalog-material_shard1_replica1]  webapp=/solr
> path=/cdcr params={qt=/cdcr&action=BOOTSTRAP_STATUS&wt=javabin&version=2}
> status=0 QTime=0
> 2016-12-21 23:24:41.657 INFO  (qtp110456297-17) [c:sial-catalog-material
> s:shard1 r:core_node1 x:sial-catalog-material_shard1_replica1]
> o.a.s.c.S.Request [sial-catalog-material_shard1_replica1]  webapp=/solr
> path=/cdcr params={qt=/cdcr&action=BOOTSTRAP_STATUS&wt=javabin&version=2}
> status=0 QTime=0
>
>
> These should be DEBUG messages and NOT INFO messages. Is there a way to
> selectively turn them off?  The above is from a Target collection, it is
> even worse on the Source side.
>
> I'd rather not change my logging level as most INFO messages are useful.
>
> This is a very poor default logging level for these messages.
>

-- 


This message and any attachment are confidential and may be privileged or 
otherwise protected from disclosure. If you are not the intended recipient, 
you must not copy this message or attachment or disclose the contents to 
any other person. If you have received this transmission in error, please 
notify the sender immediately and delete the message and any attachment 
from your system. Merck KGaA, Darmstadt, Germany and any of its 
subsidiaries do not accept liability for any omissions or errors in this 
message which may arise as a result of E-Mail-transmission or for damages 
resulting from any unauthorized changes of the content of this message and 
any attachment thereto. Merck KGaA, Darmstadt, Germany and any of its 
subsidiaries do not guarantee that this message is free of viruses and does 
not accept liability for any damages caused by any virus transmitted 
therewith.

Click http://www.merckgroup.com/disclaimer to access the German, French, 
Spanish and Portuguese versions of this disclaimer.