transaction log size

[Gurfan <ht...@gmail.com>]

Hi,

As Transaction log(Tlog) play important role while restarting the SolrCloud
cluster, we are trying to decrease the size. Many of the posts on net which
we find describing that -
"decreasing the AutoCommit and increasing autoSoftCommit would generate the
small size of transaction log".

To test the aforesaid statement we executed some Run:

Document Size: ~2KB.

1st Run:

AutoCommit: 30 Sec
autoSoftCommit: 20 Sec
openSearcher:  false
Index size: 4.7 GB
Transaction log:
       Master: 740KB
       Slave: 86 MB

2nd Run:

AutoCommit: 20 Sec
autoSoftCommit: 30 Sec
openSearcher:  false
Index size: 4.7 GB
Transaction log:
       Master: 740KB
       Slave: 202 MB


PFA zip containing cluster(master, slave) transaction logs disk usage at the
interval of 1 Min.

transactionLog.zip
<http://lucene.472066.n3.nabble.com/file/n4128354/transactionLog.zip>  

> Schema.xml
> Solr-config.xml
> transactionLog/master/*
> transactionLog/slave/*



May you please give us some pointer so that we can control on Transaction
log(Tlog) generation.

Thanks,
--Gurfan



--
View this message in context: http://lucene.472066.n3.nabble.com/transaction-log-size-tp4128354.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: transaction log size

[Erick Erickson <er...@gmail.com>]

On the surface, this doesn't make sense, I'd expect that the tlogs
would be roughly the same size on leaders and replicas. Or at least
show the same variance.

If you were to guess how much volume in terms of files being fired at
the index, how much would you expect in 30 seconds? And does it
approximate the size you're seeing in your tlogs (acutally 2x your
data transmission rate over 30 seconds).

Hard commits with openSearcher=false are actually pretty cheap
operations. About all they do is close the currently open segments and
truncate the tlog. What happens if you drop it to 10 seconds?

 Best,
Erick

On Wed, Apr 2, 2014 at 4:04 AM, Gurfan <ht...@gmail.com> wrote:
> Thanks Shawn for the quick reply.
>
> We are using Solr Cloud version 4.6.1
>
> Usually we see higher transaction log on replica. Leader`s tlog size is in
> KB`s. We also tried keeping the hard commit(autoCommit) as 20 Sec and
> autoSoftCommit as 30 Sec.
>
> We written a script to monitor the disk usage of tlog directory in 1 Min
> interval, also noticed that the logs are purge at a particular time. For
> instance: Tlog starts with ~4MB and it increases at some point i.e.
> 20MB,50MB,220MB,600MB again it reduces with ~10MB.
>
> http://searchhub.org/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
>
> How can we reduce Tlog size at its lowest, so that our system restart up
> time will less.
>
> Thanks,
> --Gurfan
>
>
>
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/transaction-log-size-tp4128354p4128547.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: transaction log size

[Gurfan <ht...@gmail.com>]

Thanks Shawn for the quick reply.

We are using Solr Cloud version 4.6.1 

Usually we see higher transaction log on replica. Leader`s tlog size is in
KB`s. We also tried keeping the hard commit(autoCommit) as 20 Sec and
autoSoftCommit as 30 Sec.

We written a script to monitor the disk usage of tlog directory in 1 Min
interval, also noticed that the logs are purge at a particular time. For
instance: Tlog starts with ~4MB and it increases at some point i.e.
20MB,50MB,220MB,600MB again it reduces with ~10MB.

http://searchhub.org/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/

How can we reduce Tlog size at its lowest, so that our system restart up
time will less.

Thanks,
--Gurfan  


    



--
View this message in context: http://lucene.472066.n3.nabble.com/transaction-log-size-tp4128354p4128547.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: transaction log size

[Shawn Heisey <so...@elyograg.org>]

On 4/1/2014 1:23 AM, Gurfan wrote:
> Hi,
> 
> As Transaction log(Tlog) play important role while restarting the SolrCloud
> cluster, we are trying to decrease the size. Many of the posts on net which
> we find describing that -
> "decreasing the AutoCommit and increasing autoSoftCommit would generate the
> small size of transaction log".

Transaction log size is purely controlled by hard commits (autoCommit),
soft commits have no influence at all.

> To test the aforesaid statement we executed some Run:
> 
> Document Size: ~2KB.
> 
> 1st Run:
> 
> AutoCommit: 30 Sec
> autoSoftCommit: 20 Sec
> openSearcher:  false
> Index size: 4.7 GB
> Transaction log:
>        Master: 740KB
>        Slave: 86 MB
> 
> 2nd Run:
> 
> AutoCommit: 20 Sec
> autoSoftCommit: 30 Sec
> openSearcher:  false
> Index size: 4.7 GB
> Transaction log:
>        Master: 740KB
>        Slave: 202 MB

When you say master and slave, are you using old-style replication, or
are you using SolrCloud?

With old-style replication, the slave should not be indexing *anything*
-- the index itself is copied from the master to the slave.  I don't
know whether transaction logs are copied by replication, but I suspect
that they are not.  If they are not, the slave should not have ANY
transaction logs.  If they are, the slave should be identical.  You
should be OK to delete the slave transaction logs.  It's entirely
possible that there is a bug.

With SolrCloud, master and slave have no meaning -- each shard has
replicas, and one of the replicas is elected to be leader.  An election
can happen at any time in response to cluster events, and a different
replica might be elected leader.

Although replication is required for SolrCloud operation, it is not used
except at node startup and if something goes wrong that requires index
recovery.  Each node does its own indexing and will manage its own
transaction logs according to how frequently you do a hard commit.

Thanks,
Shawn