You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by gadde <ga...@gmail.com> on 2013/01/10 11:51:49 UTC
Large transaction logs
we have a SolrCloud with 3 nodes. we add documents to leader node and use
commitwithin(100secs) option in SolrJ to add documents. AutoSoftCommit in
SolrConfig is 1000ms.
Transaction logs on replicas grew bigger than the index and we ran out of
disk space in few days. Leader's tlogs are very small in few hundred MBs.
The following post suggest hard commit is required for "relieving the memory
pressure of the transactionlog"
http://lucene.472066.n3.nabble.com/SolrCloud-is-softcommit-cluster-wide-for-the-collection-td4021584.html#a4021631
what is the best way to do a hard commit on this setup in SolrCloud?
a. Through autoCommit in SolrConfig? which would cause hard commit on all
the nodes at different times
b. Trigger hard commit on leader while updating through SolrJ?
--
View this message in context: http://lucene.472066.n3.nabble.com/Large-transaction-logs-tp4032144.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Large transaction logs
Posted by Sai Gadde <ga...@gmail.com>.
Thanks for the Info . I was thinking that autocommit will be propagated
through the cloud like the explicit commit command. If it is not logged
into tlogs as mentioned we can just set autocommit and forget about it.
Thanks
Shyam
On Thu, Jan 10, 2013 at 8:15 PM, Tomás Fernández Löbbe <
tomasflobbe@gmail.com> wrote:
> Yes, you must issue hard commits. You can use autocommit and use
> openSearcher=false. Autocommit is not distributed, it has to be configured
> in every node (which will automatically be, because you are using the exact
> same solrconfig for all your nodes).
>
> Other option is to issue an explicit hard commit command, those ARE
> distributed across all shards and replicas. You should also use
> "openSearcher=false" option for explicit hard commits (the searcher is now
> being opened by the soft commits).
>
> Both options are fine. Personally I prefer autocommit because then you can
> just "forget" about commits.
>
> Tomás
>
>
> On Thu, Jan 10, 2013 at 7:51 AM, gadde <ga...@gmail.com> wrote:
>
> > we have a SolrCloud with 3 nodes. we add documents to leader node and use
> > commitwithin(100secs) option in SolrJ to add documents. AutoSoftCommit in
> > SolrConfig is 1000ms.
> >
> > Transaction logs on replicas grew bigger than the index and we ran out of
> > disk space in few days. Leader's tlogs are very small in few hundred MBs.
> >
> > The following post suggest hard commit is required for "relieving the
> > memory
> > pressure of the transactionlog"
> >
> >
> http://lucene.472066.n3.nabble.com/SolrCloud-is-softcommit-cluster-wide-for-the-collection-td4021584.html#a4021631
> >
> > what is the best way to do a hard commit on this setup in SolrCloud?
> >
> > a. Through autoCommit in SolrConfig? which would cause hard commit on all
> > the nodes at different times
> > b. Trigger hard commit on leader while updating through SolrJ?
> >
> >
> >
> > --
> > View this message in context:
> > http://lucene.472066.n3.nabble.com/Large-transaction-logs-tp4032144.html
> > Sent from the Solr - User mailing list archive at Nabble.com.
> >
>
Re: Large transaction logs
Posted by Tomás Fernández Löbbe <to...@gmail.com>.
Yes, you must issue hard commits. You can use autocommit and use
openSearcher=false. Autocommit is not distributed, it has to be configured
in every node (which will automatically be, because you are using the exact
same solrconfig for all your nodes).
Other option is to issue an explicit hard commit command, those ARE
distributed across all shards and replicas. You should also use
"openSearcher=false" option for explicit hard commits (the searcher is now
being opened by the soft commits).
Both options are fine. Personally I prefer autocommit because then you can
just "forget" about commits.
Tomás
On Thu, Jan 10, 2013 at 7:51 AM, gadde <ga...@gmail.com> wrote:
> we have a SolrCloud with 3 nodes. we add documents to leader node and use
> commitwithin(100secs) option in SolrJ to add documents. AutoSoftCommit in
> SolrConfig is 1000ms.
>
> Transaction logs on replicas grew bigger than the index and we ran out of
> disk space in few days. Leader's tlogs are very small in few hundred MBs.
>
> The following post suggest hard commit is required for "relieving the
> memory
> pressure of the transactionlog"
>
> http://lucene.472066.n3.nabble.com/SolrCloud-is-softcommit-cluster-wide-for-the-collection-td4021584.html#a4021631
>
> what is the best way to do a hard commit on this setup in SolrCloud?
>
> a. Through autoCommit in SolrConfig? which would cause hard commit on all
> the nodes at different times
> b. Trigger hard commit on leader while updating through SolrJ?
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Large-transaction-logs-tp4032144.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>