You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Nitin Solanki <ni...@gmail.com> on 2015/02/02 13:42:07 UTC
Solr Logging files get high
Hi,
My solr logs directory has been get high. It is seriously problem
or It harms my solr performance in both cases indexing as well as searching.
Fwd: Solr Logging files get high
Posted by Nitin Solanki <ni...@gmail.com>.
Hi Michael Della and Michael Sokolov,
*size of tlog :-*
56K /mnt/nitin/solr/node1/solr/wikingram_shard3_replica1/data/tlog/
56K /mnt/nitin/solr/node1/solr/wikingram_shard7_replica1/data/tlog/
56K /mnt/nitin/solr/node2/solr/wikingram_shard4_replica1/data/tlog/
52K /mnt/nitin/solr/node2/solr/wikingram_shard8_replica1/data/tlog/
52K /mnt/nitin/solr/node3/solr/wikingram_shard1_replica1/data/tlog/
52K /mnt/nitin/solr/node3/solr/wikingram_shard5_replica1/data/tlog/
56K /mnt/nitin/solr/node4/solr/wikingram_shard2_replica1/data/tlog/
48K /mnt/nitin/solr/node4/solr/wikingram_shard6_replica1/data/tlog/
*Size of logs :-*
755M /mnt/nitin/solr/node1/logs/
729M /mnt/nitin/solr/node2/logs/
729M /mnt/nitin/solr/node3/logs/
729M /mnt/nitin/solr/node4/logs/
Which log is reducing performance? I am committing more frequent hard
commits. After 1 second , I am performing soft commit and after 15 seconds,
I am performing hard commit. I indexed 2 GB of data and you can see the
size of tlog that I pasted above. Is this tlog is good for 2GB indexed
data? Or is it high? The main question is that size of log will harm
performance of Solr?
Re: Solr Logging files get high
Posted by Nishanth S <ni...@gmail.com>.
I feel the tlog size is perfectly fine since your hard commit interval is
low.You can try increasing your hard commit and soft commit values.Soft
commit of 1 sec is very low.Soft commit is about visibility of
documents,so you can try and increase this as far your slas.
-Nishanth
On Mon, Feb 2, 2015 at 10:51 PM, Nitin Solanki <ni...@gmail.com> wrote:
> Hi Michael Della and Michael Sokolov,
>
> *size of tlog :-*
> 56K /mnt/nitin/solr/node1/solr/wikingram_shard3_replica1/data/tlog/
> 56K /mnt/nitin/solr/node1/solr/wikingram_shard7_replica1/data/tlog/
> 56K /mnt/nitin/solr/node2/solr/wikingram_shard4_replica1/data/tlog/
> 52K /mnt/nitin/solr/node2/solr/wikingram_shard8_replica1/data/tlog/
> 52K /mnt/nitin/solr/node3/solr/wikingram_shard1_replica1/data/tlog/
> 52K /mnt/nitin/solr/node3/solr/wikingram_shard5_replica1/data/tlog/
> 56K /mnt/nitin/solr/node4/solr/wikingram_shard2_replica1/data/tlog/
> 48K /mnt/nitin/solr/node4/solr/wikingram_shard6_replica1/data/tlog/
>
> *Size of logs :-*
> 755M /mnt/nitin/solr/node1/logs/
> 729M /mnt/nitin/solr/node2/logs/
> 729M /mnt/nitin/solr/node3/logs/
> 729M /mnt/nitin/solr/node4/logs/
>
> Which log is reducing performance? I am committing more frequent hard
> commits. After 1 second , I am performing soft commit and after 15 seconds,
> I am performing hard commit. I indexed 2 GB of data and you can see the
> size of tlog that I pasted above. Is this tlog is good for 2GB indexed
> data? Or is it high? The main question is that size of log will harm
> performance of Solr?
>
>
>
>
> On Mon, Feb 2, 2015 at 10:27 PM, Michael Della Bitta <
> michael.della.bitta@appinions.com> wrote:
>
> > Good call, it could easily be the tlog Nitin is talking about.
> >
> > As for which definition of high, I was making assumptions as well. :)
> >
> > Michael Della Bitta
> >
> > Senior Software Engineer
> >
> > o: +1 646 532 3062
> >
> > appinions inc.
> >
> > “The Science of Influence Marketing”
> >
> > 18 East 41st Street
> >
> > New York, NY 10017
> >
> > t: @appinions <https://twitter.com/Appinions> | g+:
> > plus.google.com/appinions
> > <
> >
> https://plus.google.com/u/0/b/112002776285509593336/112002776285509593336/posts
> > >
> > w: appinions.com <http://www.appinions.com/>
> >
> > On Mon, Feb 2, 2015 at 11:51 AM, Michael Sokolov <
> > msokolov@safaribooksonline.com> wrote:
> >
> > > I was tempted to suggest rehab -- but seriously it wasn't clear if
> Nitin
> > > meant the log files Michael is referring to, or the transaction log
> > > (tlog). If it's the transaction log, the solution is more frequent
> hard
> > > commits.
> > >
> > > -Mike
> > >
> > > On 2/2/2015 11:48 AM, Michael Della Bitta wrote:
> > >
> > >> If you'd like to reduce the amount of lines Solr logs, you need to
> edit
> > >> the
> > >> file example/resources/log4j.properties in Solr's home directory.
> Change
> > >> lines that say INFO to WARN.
> > >>
> > >> Michael Della Bitta
> > >>
> > >> Senior Software Engineer
> > >>
> > >> o: +1 646 532 3062
> > >>
> > >> appinions inc.
> > >>
> > >> “The Science of Influence Marketing”
> > >>
> > >> 18 East 41st Street
> > >>
> > >> New York, NY 10017
> > >>
> > >> t: @appinions <https://twitter.com/Appinions> | g+:
> > >> plus.google.com/appinions
> > >> <https://plus.google.com/u/0/b/112002776285509593336/
> > >> 112002776285509593336/posts>
> > >> w: appinions.com <http://www.appinions.com/>
> > >>
> > >> On Mon, Feb 2, 2015 at 7:42 AM, Nitin Solanki <ni...@gmail.com>
> > >> wrote:
> > >>
> > >> Hi,
> > >>> My solr logs directory has been get high. It is seriously
> > >>> problem
> > >>> or It harms my solr performance in both cases indexing as well as
> > >>> searching.
> > >>>
> > >>>
> > >
> >
>
Re: Solr Logging files get high
Posted by Nitin Solanki <ni...@gmail.com>.
Thanks Michael Della Bitta.
Hi. Mike Sokolov,
There is no DEBUG appears inside logs..
On Tue, Feb 3, 2015 at 10:06 PM, Michael Della Bitta <
michael.della.bitta@appinions.com> wrote:
> If you're trying to do a bulk ingest of data, I recommend committing less
> frequently. Don't soft commit at all until the end of the batch, and hard
> commit every 60 seconds.
>
> Michael Della Bitta
>
> Senior Software Engineer
>
> o: +1 646 532 3062
>
> appinions inc.
>
> “The Science of Influence Marketing”
>
> 18 East 41st Street
>
> New York, NY 10017
>
> t: @appinions <https://twitter.com/Appinions> | g+:
> plus.google.com/appinions
> <
> https://plus.google.com/u/0/b/112002776285509593336/112002776285509593336/posts
> >
> w: appinions.com <http://www.appinions.com/>
>
> On Tue, Feb 3, 2015 at 12:51 AM, Nitin Solanki <ni...@gmail.com>
> wrote:
>
> > Hi Michael Della and Michael Sokolov,
> >
> > *size of tlog :-*
> > 56K /mnt/nitin/solr/node1/solr/wikingram_shard3_replica1/data/tlog/
> > 56K /mnt/nitin/solr/node1/solr/wikingram_shard7_replica1/data/tlog/
> > 56K /mnt/nitin/solr/node2/solr/wikingram_shard4_replica1/data/tlog/
> > 52K /mnt/nitin/solr/node2/solr/wikingram_shard8_replica1/data/tlog/
> > 52K /mnt/nitin/solr/node3/solr/wikingram_shard1_replica1/data/tlog/
> > 52K /mnt/nitin/solr/node3/solr/wikingram_shard5_replica1/data/tlog/
> > 56K /mnt/nitin/solr/node4/solr/wikingram_shard2_replica1/data/tlog/
> > 48K /mnt/nitin/solr/node4/solr/wikingram_shard6_replica1/data/tlog/
> >
> > *Size of logs :-*
> > 755M /mnt/nitin/solr/node1/logs/
> > 729M /mnt/nitin/solr/node2/logs/
> > 729M /mnt/nitin/solr/node3/logs/
> > 729M /mnt/nitin/solr/node4/logs/
> >
> > Which log is reducing performance? I am committing more frequent hard
> > commits. After 1 second , I am performing soft commit and after 15
> seconds,
> > I am performing hard commit. I indexed 2 GB of data and you can see the
> > size of tlog that I pasted above. Is this tlog is good for 2GB indexed
> > data? Or is it high? The main question is that size of log will harm
> > performance of Solr?
> >
> >
> >
> >
> > On Mon, Feb 2, 2015 at 10:27 PM, Michael Della Bitta <
> > michael.della.bitta@appinions.com> wrote:
> >
> > > Good call, it could easily be the tlog Nitin is talking about.
> > >
> > > As for which definition of high, I was making assumptions as well. :)
> > >
> > > Michael Della Bitta
> > >
> > > Senior Software Engineer
> > >
> > > o: +1 646 532 3062
> > >
> > > appinions inc.
> > >
> > > “The Science of Influence Marketing”
> > >
> > > 18 East 41st Street
> > >
> > > New York, NY 10017
> > >
> > > t: @appinions <https://twitter.com/Appinions> | g+:
> > > plus.google.com/appinions
> > > <
> > >
> >
> https://plus.google.com/u/0/b/112002776285509593336/112002776285509593336/posts
> > > >
> > > w: appinions.com <http://www.appinions.com/>
> > >
> > > On Mon, Feb 2, 2015 at 11:51 AM, Michael Sokolov <
> > > msokolov@safaribooksonline.com> wrote:
> > >
> > > > I was tempted to suggest rehab -- but seriously it wasn't clear if
> > Nitin
> > > > meant the log files Michael is referring to, or the transaction log
> > > > (tlog). If it's the transaction log, the solution is more frequent
> > hard
> > > > commits.
> > > >
> > > > -Mike
> > > >
> > > > On 2/2/2015 11:48 AM, Michael Della Bitta wrote:
> > > >
> > > >> If you'd like to reduce the amount of lines Solr logs, you need to
> > edit
> > > >> the
> > > >> file example/resources/log4j.properties in Solr's home directory.
> > Change
> > > >> lines that say INFO to WARN.
> > > >>
> > > >> Michael Della Bitta
> > > >>
> > > >> Senior Software Engineer
> > > >>
> > > >> o: +1 646 532 3062
> > > >>
> > > >> appinions inc.
> > > >>
> > > >> “The Science of Influence Marketing”
> > > >>
> > > >> 18 East 41st Street
> > > >>
> > > >> New York, NY 10017
> > > >>
> > > >> t: @appinions <https://twitter.com/Appinions> | g+:
> > > >> plus.google.com/appinions
> > > >> <https://plus.google.com/u/0/b/112002776285509593336/
> > > >> 112002776285509593336/posts>
> > > >> w: appinions.com <http://www.appinions.com/>
> > > >>
> > > >> On Mon, Feb 2, 2015 at 7:42 AM, Nitin Solanki <nitinmlvya@gmail.com
> >
> > > >> wrote:
> > > >>
> > > >> Hi,
> > > >>> My solr logs directory has been get high. It is seriously
> > > >>> problem
> > > >>> or It harms my solr performance in both cases indexing as well as
> > > >>> searching.
> > > >>>
> > > >>>
> > > >
> > >
> >
>
Re: Solr Logging files get high
Posted by Michael Della Bitta <mi...@appinions.com>.
If you're trying to do a bulk ingest of data, I recommend committing less
frequently. Don't soft commit at all until the end of the batch, and hard
commit every 60 seconds.
Michael Della Bitta
Senior Software Engineer
o: +1 646 532 3062
appinions inc.
“The Science of Influence Marketing”
18 East 41st Street
New York, NY 10017
t: @appinions <https://twitter.com/Appinions> | g+:
plus.google.com/appinions
<https://plus.google.com/u/0/b/112002776285509593336/112002776285509593336/posts>
w: appinions.com <http://www.appinions.com/>
On Tue, Feb 3, 2015 at 12:51 AM, Nitin Solanki <ni...@gmail.com> wrote:
> Hi Michael Della and Michael Sokolov,
>
> *size of tlog :-*
> 56K /mnt/nitin/solr/node1/solr/wikingram_shard3_replica1/data/tlog/
> 56K /mnt/nitin/solr/node1/solr/wikingram_shard7_replica1/data/tlog/
> 56K /mnt/nitin/solr/node2/solr/wikingram_shard4_replica1/data/tlog/
> 52K /mnt/nitin/solr/node2/solr/wikingram_shard8_replica1/data/tlog/
> 52K /mnt/nitin/solr/node3/solr/wikingram_shard1_replica1/data/tlog/
> 52K /mnt/nitin/solr/node3/solr/wikingram_shard5_replica1/data/tlog/
> 56K /mnt/nitin/solr/node4/solr/wikingram_shard2_replica1/data/tlog/
> 48K /mnt/nitin/solr/node4/solr/wikingram_shard6_replica1/data/tlog/
>
> *Size of logs :-*
> 755M /mnt/nitin/solr/node1/logs/
> 729M /mnt/nitin/solr/node2/logs/
> 729M /mnt/nitin/solr/node3/logs/
> 729M /mnt/nitin/solr/node4/logs/
>
> Which log is reducing performance? I am committing more frequent hard
> commits. After 1 second , I am performing soft commit and after 15 seconds,
> I am performing hard commit. I indexed 2 GB of data and you can see the
> size of tlog that I pasted above. Is this tlog is good for 2GB indexed
> data? Or is it high? The main question is that size of log will harm
> performance of Solr?
>
>
>
>
> On Mon, Feb 2, 2015 at 10:27 PM, Michael Della Bitta <
> michael.della.bitta@appinions.com> wrote:
>
> > Good call, it could easily be the tlog Nitin is talking about.
> >
> > As for which definition of high, I was making assumptions as well. :)
> >
> > Michael Della Bitta
> >
> > Senior Software Engineer
> >
> > o: +1 646 532 3062
> >
> > appinions inc.
> >
> > “The Science of Influence Marketing”
> >
> > 18 East 41st Street
> >
> > New York, NY 10017
> >
> > t: @appinions <https://twitter.com/Appinions> | g+:
> > plus.google.com/appinions
> > <
> >
> https://plus.google.com/u/0/b/112002776285509593336/112002776285509593336/posts
> > >
> > w: appinions.com <http://www.appinions.com/>
> >
> > On Mon, Feb 2, 2015 at 11:51 AM, Michael Sokolov <
> > msokolov@safaribooksonline.com> wrote:
> >
> > > I was tempted to suggest rehab -- but seriously it wasn't clear if
> Nitin
> > > meant the log files Michael is referring to, or the transaction log
> > > (tlog). If it's the transaction log, the solution is more frequent
> hard
> > > commits.
> > >
> > > -Mike
> > >
> > > On 2/2/2015 11:48 AM, Michael Della Bitta wrote:
> > >
> > >> If you'd like to reduce the amount of lines Solr logs, you need to
> edit
> > >> the
> > >> file example/resources/log4j.properties in Solr's home directory.
> Change
> > >> lines that say INFO to WARN.
> > >>
> > >> Michael Della Bitta
> > >>
> > >> Senior Software Engineer
> > >>
> > >> o: +1 646 532 3062
> > >>
> > >> appinions inc.
> > >>
> > >> “The Science of Influence Marketing”
> > >>
> > >> 18 East 41st Street
> > >>
> > >> New York, NY 10017
> > >>
> > >> t: @appinions <https://twitter.com/Appinions> | g+:
> > >> plus.google.com/appinions
> > >> <https://plus.google.com/u/0/b/112002776285509593336/
> > >> 112002776285509593336/posts>
> > >> w: appinions.com <http://www.appinions.com/>
> > >>
> > >> On Mon, Feb 2, 2015 at 7:42 AM, Nitin Solanki <ni...@gmail.com>
> > >> wrote:
> > >>
> > >> Hi,
> > >>> My solr logs directory has been get high. It is seriously
> > >>> problem
> > >>> or It harms my solr performance in both cases indexing as well as
> > >>> searching.
> > >>>
> > >>>
> > >
> >
>
Re: Solr Logging files get high
Posted by Nitin Solanki <ni...@gmail.com>.
Hi Michael Della and Michael Sokolov,
*size of tlog :-*
56K /mnt/nitin/solr/node1/solr/wikingram_shard3_replica1/data/tlog/
56K /mnt/nitin/solr/node1/solr/wikingram_shard7_replica1/data/tlog/
56K /mnt/nitin/solr/node2/solr/wikingram_shard4_replica1/data/tlog/
52K /mnt/nitin/solr/node2/solr/wikingram_shard8_replica1/data/tlog/
52K /mnt/nitin/solr/node3/solr/wikingram_shard1_replica1/data/tlog/
52K /mnt/nitin/solr/node3/solr/wikingram_shard5_replica1/data/tlog/
56K /mnt/nitin/solr/node4/solr/wikingram_shard2_replica1/data/tlog/
48K /mnt/nitin/solr/node4/solr/wikingram_shard6_replica1/data/tlog/
*Size of logs :-*
755M /mnt/nitin/solr/node1/logs/
729M /mnt/nitin/solr/node2/logs/
729M /mnt/nitin/solr/node3/logs/
729M /mnt/nitin/solr/node4/logs/
Which log is reducing performance? I am committing more frequent hard
commits. After 1 second , I am performing soft commit and after 15 seconds,
I am performing hard commit. I indexed 2 GB of data and you can see the
size of tlog that I pasted above. Is this tlog is good for 2GB indexed
data? Or is it high? The main question is that size of log will harm
performance of Solr?
On Mon, Feb 2, 2015 at 10:27 PM, Michael Della Bitta <
michael.della.bitta@appinions.com> wrote:
> Good call, it could easily be the tlog Nitin is talking about.
>
> As for which definition of high, I was making assumptions as well. :)
>
> Michael Della Bitta
>
> Senior Software Engineer
>
> o: +1 646 532 3062
>
> appinions inc.
>
> “The Science of Influence Marketing”
>
> 18 East 41st Street
>
> New York, NY 10017
>
> t: @appinions <https://twitter.com/Appinions> | g+:
> plus.google.com/appinions
> <
> https://plus.google.com/u/0/b/112002776285509593336/112002776285509593336/posts
> >
> w: appinions.com <http://www.appinions.com/>
>
> On Mon, Feb 2, 2015 at 11:51 AM, Michael Sokolov <
> msokolov@safaribooksonline.com> wrote:
>
> > I was tempted to suggest rehab -- but seriously it wasn't clear if Nitin
> > meant the log files Michael is referring to, or the transaction log
> > (tlog). If it's the transaction log, the solution is more frequent hard
> > commits.
> >
> > -Mike
> >
> > On 2/2/2015 11:48 AM, Michael Della Bitta wrote:
> >
> >> If you'd like to reduce the amount of lines Solr logs, you need to edit
> >> the
> >> file example/resources/log4j.properties in Solr's home directory. Change
> >> lines that say INFO to WARN.
> >>
> >> Michael Della Bitta
> >>
> >> Senior Software Engineer
> >>
> >> o: +1 646 532 3062
> >>
> >> appinions inc.
> >>
> >> “The Science of Influence Marketing”
> >>
> >> 18 East 41st Street
> >>
> >> New York, NY 10017
> >>
> >> t: @appinions <https://twitter.com/Appinions> | g+:
> >> plus.google.com/appinions
> >> <https://plus.google.com/u/0/b/112002776285509593336/
> >> 112002776285509593336/posts>
> >> w: appinions.com <http://www.appinions.com/>
> >>
> >> On Mon, Feb 2, 2015 at 7:42 AM, Nitin Solanki <ni...@gmail.com>
> >> wrote:
> >>
> >> Hi,
> >>> My solr logs directory has been get high. It is seriously
> >>> problem
> >>> or It harms my solr performance in both cases indexing as well as
> >>> searching.
> >>>
> >>>
> >
>
Re: Solr Logging files get high
Posted by Michael Della Bitta <mi...@appinions.com>.
Good call, it could easily be the tlog Nitin is talking about.
As for which definition of high, I was making assumptions as well. :)
Michael Della Bitta
Senior Software Engineer
o: +1 646 532 3062
appinions inc.
“The Science of Influence Marketing”
18 East 41st Street
New York, NY 10017
t: @appinions <https://twitter.com/Appinions> | g+:
plus.google.com/appinions
<https://plus.google.com/u/0/b/112002776285509593336/112002776285509593336/posts>
w: appinions.com <http://www.appinions.com/>
On Mon, Feb 2, 2015 at 11:51 AM, Michael Sokolov <
msokolov@safaribooksonline.com> wrote:
> I was tempted to suggest rehab -- but seriously it wasn't clear if Nitin
> meant the log files Michael is referring to, or the transaction log
> (tlog). If it's the transaction log, the solution is more frequent hard
> commits.
>
> -Mike
>
> On 2/2/2015 11:48 AM, Michael Della Bitta wrote:
>
>> If you'd like to reduce the amount of lines Solr logs, you need to edit
>> the
>> file example/resources/log4j.properties in Solr's home directory. Change
>> lines that say INFO to WARN.
>>
>> Michael Della Bitta
>>
>> Senior Software Engineer
>>
>> o: +1 646 532 3062
>>
>> appinions inc.
>>
>> “The Science of Influence Marketing”
>>
>> 18 East 41st Street
>>
>> New York, NY 10017
>>
>> t: @appinions <https://twitter.com/Appinions> | g+:
>> plus.google.com/appinions
>> <https://plus.google.com/u/0/b/112002776285509593336/
>> 112002776285509593336/posts>
>> w: appinions.com <http://www.appinions.com/>
>>
>> On Mon, Feb 2, 2015 at 7:42 AM, Nitin Solanki <ni...@gmail.com>
>> wrote:
>>
>> Hi,
>>> My solr logs directory has been get high. It is seriously
>>> problem
>>> or It harms my solr performance in both cases indexing as well as
>>> searching.
>>>
>>>
>
Re: Solr Logging files get high
Posted by Michael Sokolov <ms...@safaribooksonline.com>.
I was tempted to suggest rehab -- but seriously it wasn't clear if Nitin
meant the log files Michael is referring to, or the transaction log
(tlog). If it's the transaction log, the solution is more frequent hard
commits.
-Mike
On 2/2/2015 11:48 AM, Michael Della Bitta wrote:
> If you'd like to reduce the amount of lines Solr logs, you need to edit the
> file example/resources/log4j.properties in Solr's home directory. Change
> lines that say INFO to WARN.
>
> Michael Della Bitta
>
> Senior Software Engineer
>
> o: +1 646 532 3062
>
> appinions inc.
>
> “The Science of Influence Marketing”
>
> 18 East 41st Street
>
> New York, NY 10017
>
> t: @appinions <https://twitter.com/Appinions> | g+:
> plus.google.com/appinions
> <https://plus.google.com/u/0/b/112002776285509593336/112002776285509593336/posts>
> w: appinions.com <http://www.appinions.com/>
>
> On Mon, Feb 2, 2015 at 7:42 AM, Nitin Solanki <ni...@gmail.com> wrote:
>
>> Hi,
>> My solr logs directory has been get high. It is seriously problem
>> or It harms my solr performance in both cases indexing as well as
>> searching.
>>
Re: Solr Logging files get high
Posted by Michael Della Bitta <mi...@appinions.com>.
If you'd like to reduce the amount of lines Solr logs, you need to edit the
file example/resources/log4j.properties in Solr's home directory. Change
lines that say INFO to WARN.
Michael Della Bitta
Senior Software Engineer
o: +1 646 532 3062
appinions inc.
“The Science of Influence Marketing”
18 East 41st Street
New York, NY 10017
t: @appinions <https://twitter.com/Appinions> | g+:
plus.google.com/appinions
<https://plus.google.com/u/0/b/112002776285509593336/112002776285509593336/posts>
w: appinions.com <http://www.appinions.com/>
On Mon, Feb 2, 2015 at 7:42 AM, Nitin Solanki <ni...@gmail.com> wrote:
> Hi,
> My solr logs directory has been get high. It is seriously problem
> or It harms my solr performance in both cases indexing as well as
> searching.
>