You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Sandro Zbinden <zb...@imagic.ch> on 2013/08/27 17:08:09 UTC
Transaction log "on-disk" guarantees
Dear solr users
We are using the solr soft comit feature and we are worried about what happens after we restart the solr server.
Can we activate the transaction log to have on disk guarantees and then use the solr soft commit feature ?
Thanks and Best regards
Sandro Zbinden
________________________________
[imagic]Sandro Zbinden
Software Engineer
Re: Transaction log "on-disk" guarantees
Posted by Jack Krupansky <ja...@basetechnology.com>.
And here I was just about to give Mark credit for updating the wiki!
-- Jack Krupansky
-----Original Message-----
From: Erick Erickson
Sent: Tuesday, August 27, 2013 4:24 PM
To: solr-user@lucene.apache.org
Subject: Re: Transaction log "on-disk" guarantees
Well, when you originally googled it wasn't there <G>, I just put it in
after
reading your post and realizing that it wasn't documented.
Erick
On Tue, Aug 27, 2013 at 2:13 PM, SandroZbinden <zb...@imagic.ch> wrote:
> Hey Jack
>
> Thanks a lot. I just googled for fsync and syncLevel instead of searching
> in
> the solr wiki. Won't happen again.
>
> Here is the link to the solr wiki page that describes to set the syncLevel
>
> http://wiki.apache.org/solr/SolrCloud?highlight=%28fsync%29
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Transaction-log-on-disk-guarantees-tp4086829p4086867.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: Transaction log "on-disk" guarantees
Posted by Erick Erickson <er...@gmail.com>.
Well, when you originally googled it wasn't there <G>, I just put it in
after
reading your post and realizing that it wasn't documented.
Erick
On Tue, Aug 27, 2013 at 2:13 PM, SandroZbinden <zb...@imagic.ch> wrote:
> Hey Jack
>
> Thanks a lot. I just googled for fsync and syncLevel instead of searching
> in
> the solr wiki. Won't happen again.
>
> Here is the link to the solr wiki page that describes to set the syncLevel
>
> http://wiki.apache.org/solr/SolrCloud?highlight=%28fsync%29
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Transaction-log-on-disk-guarantees-tp4086829p4086867.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: Transaction log "on-disk" guarantees
Posted by SandroZbinden <zb...@imagic.ch>.
Hey Jack
Thanks a lot. I just googled for fsync and syncLevel instead of searching in
the solr wiki. Won't happen again.
Here is the link to the solr wiki page that describes to set the syncLevel
http://wiki.apache.org/solr/SolrCloud?highlight=%28fsync%29
--
View this message in context: http://lucene.472066.n3.nabble.com/Transaction-log-on-disk-guarantees-tp4086829p4086867.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Transaction log "on-disk" guarantees
Posted by Jack Krupansky <ja...@basetechnology.com>.
You missed the wiki update that went by a short while ago:
<updateLog>
<str name="dir">${solr.data.dir:}</str>
+ <!-- if you want to take control of the synchronization you may
specify the syncLevel as one of the
+ following where ''flush'' is the default. fsync will reduce
throughput.
+ <str name="syncLevel">flush|fsync|none</str>
+ -->
</updateLog>
-- Jack Krupansky
-----Original Message-----
From: Erick Erickson
Sent: Tuesday, August 27, 2013 11:54 AM
To: solr-user@lucene.apache.org
Subject: Re: Transaction log "on-disk" guarantees
Here's a blog I wrote up a bit ago:
http://searchhub.org/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
Hmmm, unfortunately it doesn't say anything about how to set the fsync
option, but do you really care? Soft commits flush to the op system, so
a JVM crash/termination shouldn't affect it anyway. Turning on the fsync
would just be a little bit of extra protection....
On Tue, Aug 27, 2013 at 11:43 AM, Sandro Zbinden <zb...@imagic.ch> wrote:
> Hey Mark
>
> Thank you very much for the quick answer. We have a single node
> environment.
>
> I try to find the fsync option but was not successful. Ended up in the
> UpdateLog class :-)
>
> How do I enable fsync in the solrconfig.xml ?
>
>
> Besides that:
>
> If solr soft commit feature has a on disk guarantee with a transaction log
> why we don't use solr soft commit as the default commit option ?
>
>
> -----Ursprüngliche Nachricht-----
> Von: Mark Miller [mailto:markrmiller@gmail.com]
> Gesendet: Dienstag, 27. August 2013 17:12
> An: solr-user@lucene.apache.org
> Betreff: Re: Transaction log "on-disk" guarantees
>
>
> On Aug 27, 2013, at 11:08 AM, Sandro Zbinden <zb...@imagic.ch> wrote:
>
> > Can we activate the transaction log to have on disk guarantees and then
> use the solr soft commit feature ?
>
> Yes you can. If you only have a single node (no replication), you probably
> want to turn on fsync via the config.
>
> - Mark
>
>
Re: Transaction log "on-disk" guarantees
Posted by Erick Erickson <er...@gmail.com>.
Here's a blog I wrote up a bit ago:
http://searchhub.org/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
Hmmm, unfortunately it doesn't say anything about how to set the fsync
option, but do you really care? Soft commits flush to the op system, so
a JVM crash/termination shouldn't affect it anyway. Turning on the fsync
would just be a little bit of extra protection....
On Tue, Aug 27, 2013 at 11:43 AM, Sandro Zbinden <zb...@imagic.ch> wrote:
> Hey Mark
>
> Thank you very much for the quick answer. We have a single node
> environment.
>
> I try to find the fsync option but was not successful. Ended up in the
> UpdateLog class :-)
>
> How do I enable fsync in the solrconfig.xml ?
>
>
> Besides that:
>
> If solr soft commit feature has a on disk guarantee with a transaction log
> why we don't use solr soft commit as the default commit option ?
>
>
> -----Ursprüngliche Nachricht-----
> Von: Mark Miller [mailto:markrmiller@gmail.com]
> Gesendet: Dienstag, 27. August 2013 17:12
> An: solr-user@lucene.apache.org
> Betreff: Re: Transaction log "on-disk" guarantees
>
>
> On Aug 27, 2013, at 11:08 AM, Sandro Zbinden <zb...@imagic.ch> wrote:
>
> > Can we activate the transaction log to have on disk guarantees and then
> use the solr soft commit feature ?
>
> Yes you can. If you only have a single node (no replication), you probably
> want to turn on fsync via the config.
>
> - Mark
>
>
Re: Transaction log "on-disk" guarantees
Posted by Erick Erickson <er...@gmail.com>.
I updated the SolrCloud page here: http://wiki.apache.org/solr/SolrCloud to
include how to change this option.
Baaaah, "Soft commits flush to the op system" should have read "Hard
commits flush the transaction log
to the op system". Blame it on just getting back from the dentist, that was
just completely wrong.
Best,
Erick
On Tue, Aug 27, 2013 at 12:08 PM, Mark Miller <ma...@gmail.com> wrote:
> On Aug 27, 2013, at 11:54 AM, Erick Erickson <er...@gmail.com>
> wrote:
>
> > Soft commits flush to the op system, so
> > a JVM crash/termination shouldn't affect it anyway.
>
> A soft commit is not a hard commit, so there are not guarantees like this.
> It searches committed and non committed segments - non committed segments
> will not magically be committed after a JVM crash.
>
> > Turning on the fsync
> > would just be a little bit of extra protection….
>
> If you don't have replication, it turns on strong 'durability' promises.
> Without it, you are on your own if you have a hard machine reset. If
> durability is important to you and you don't have replication, it's
> important to use the fync option here. Unless you have a great, long time,
> battery backup and/or an env such that hard resets don't concern you for
> some reason. It comes down to your requirments.
>
> Responses to Sandro inline below:
>
> On Aug 27, 2013, at 11:43 AM, Sandro Zbinden <zb...@imagic.ch> wrote:
>
> > Hey Mark
> >
> > Thank you very much for the quick answer. We have a single node
> environment.
> >
> > I try to find the fsync option but was not successful. Ended up in the
> UpdateLog class :-)
> >
> > How do I enable fsync in the solrconfig.xml ?
>
> In the updateLog config, its a syncLevel=fsync param.
>
> >
> >
> > Besides that:
> >
> > If solr soft commit feature has a on disk guarantee with a transaction
> log why we don't use solr soft commit as the default commit option ?
>
> Yes, for visibility you should use soft commit. You should also have an
> auto hard commit with openSearcher=false - it's just about flushing the
> transaction log and freeing memory in this configuraiton - which is why it
> makes sense to simply turn on the auto commit for regular hard commits. You
> may or may not want to use auto soft commits.
>
> - Mark
>
> >
> >
> > -----Ursprüngliche Nachricht-----
> > Von: Mark Miller [mailto:markrmiller@gmail.com]
> > Gesendet: Dienstag, 27. August 2013 17:12
> > An: solr-user@lucene.apache.org
> > Betreff: Re: Transaction log "on-disk" guarantees
> >
> >
> > On Aug 27, 2013, at 11:08 AM, Sandro Zbinden <zb...@imagic.ch> wrote:
> >
> >> Can we activate the transaction log to have on disk guarantees and then
> use the solr soft commit feature ?
> >
> > Yes you can. If you only have a single node (no replication), you
> probably want to turn on fsync via the config.
> >
> > - Mark
> >
>
>
Re: Transaction log "on-disk" guarantees
Posted by Erick Erickson <er...@gmail.com>.
Well, that blog post is, at best, an estimate based on
disk head seek times so take it with a large grain of salt,
I probably shouldn't even have put that in the post.....
But for a single node, it's probably not all that noticeable.
Erick
On Tue, Aug 27, 2013 at 12:20 PM, Sandro Zbinden <zb...@imagic.ch> wrote:
> @Mark Do you know how I can set the syncLevel to fsync in the
> solrconfig.xml I can't find in the default solrconfig.xml
>
>
> https://svn.apache.org/repos/asf/lucene/dev/trunk/solr/example/solr/collection1/conf/solrconfig.xml
>
> The blog posts at
> http://searchhub.org/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/says that enabling the fsync is not a big increase in the update time (a
> few milliseconds (say 10-50 ms)). So I think It is useful to turn on fsync.
>
> On Aug 27, 2013, at 11:54 AM, Erick Erickson <er...@gmail.com>
> wrote:
>
> > Soft commits flush to the op system, so a JVM crash/termination
> > shouldn't affect it anyway.
>
> A soft commit is not a hard commit, so there are not guarantees like this.
> It searches committed and non committed segments - non committed segments
> will not magically be committed after a JVM crash.
>
> > Turning on the fsync
> > would just be a little bit of extra protection..
>
> If you don't have replication, it turns on strong 'durability' promises.
> Without it, you are on your own if you have a hard machine reset. If
> durability is important to you and you don't have replication, it's
> important to use the fync option here. Unless you have a great, long time,
> battery backup and/or an env such that hard resets don't concern you for
> some reason. It comes down to your requirments.
>
> Responses to Sandro inline below:
>
> On Aug 27, 2013, at 11:43 AM, Sandro Zbinden <zb...@imagic.ch> wrote:
>
> > Hey Mark
> >
> > Thank you very much for the quick answer. We have a single node
> environment.
> >
> > I try to find the fsync option but was not successful. Ended up in the
> > UpdateLog class :-)
> >
> > How do I enable fsync in the solrconfig.xml ?
>
> In the updateLog config, its a syncLevel=fsync param.
>
> >
> >
> > Besides that:
> >
> > If solr soft commit feature has a on disk guarantee with a transaction
> log why we don't use solr soft commit as the default commit option ?
>
> Yes, for visibility you should use soft commit. You should also have an
> auto hard commit with openSearcher=false - it's just about flushing the
> transaction log and freeing memory in this configuraiton - which is why it
> makes sense to simply turn on the auto commit for regular hard commits. You
> may or may not want to use auto soft commits.
>
> - Mark
>
> >
> >
> > -----Ursprüngliche Nachricht-----
> > Von: Mark Miller [mailto:markrmiller@gmail.com]
> > Gesendet: Dienstag, 27. August 2013 17:12
> > An: solr-user@lucene.apache.org
> > Betreff: Re: Transaction log "on-disk" guarantees
> >
> >
> > On Aug 27, 2013, at 11:08 AM, Sandro Zbinden <zb...@imagic.ch> wrote:
> >
> >> Can we activate the transaction log to have on disk guarantees and then
> use the solr soft commit feature ?
> >
> > Yes you can. If you only have a single node (no replication), you
> probably want to turn on fsync via the config.
> >
> > - Mark
> >
>
>
AW: Transaction log "on-disk" guarantees
Posted by Sandro Zbinden <zb...@imagic.ch>.
@Mark Do you know how I can set the syncLevel to fsync in the solrconfig.xml I can't find in the default solrconfig.xml
https://svn.apache.org/repos/asf/lucene/dev/trunk/solr/example/solr/collection1/conf/solrconfig.xml
The blog posts at http://searchhub.org/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/ says that enabling the fsync is not a big increase in the update time (a few milliseconds (say 10-50 ms)). So I think It is useful to turn on fsync.
On Aug 27, 2013, at 11:54 AM, Erick Erickson <er...@gmail.com> wrote:
> Soft commits flush to the op system, so a JVM crash/termination
> shouldn't affect it anyway.
A soft commit is not a hard commit, so there are not guarantees like this. It searches committed and non committed segments - non committed segments will not magically be committed after a JVM crash.
> Turning on the fsync
> would just be a little bit of extra protection..
If you don't have replication, it turns on strong 'durability' promises. Without it, you are on your own if you have a hard machine reset. If durability is important to you and you don't have replication, it's important to use the fync option here. Unless you have a great, long time, battery backup and/or an env such that hard resets don't concern you for some reason. It comes down to your requirments.
Responses to Sandro inline below:
On Aug 27, 2013, at 11:43 AM, Sandro Zbinden <zb...@imagic.ch> wrote:
> Hey Mark
>
> Thank you very much for the quick answer. We have a single node environment.
>
> I try to find the fsync option but was not successful. Ended up in the
> UpdateLog class :-)
>
> How do I enable fsync in the solrconfig.xml ?
In the updateLog config, its a syncLevel=fsync param.
>
>
> Besides that:
>
> If solr soft commit feature has a on disk guarantee with a transaction log why we don't use solr soft commit as the default commit option ?
Yes, for visibility you should use soft commit. You should also have an auto hard commit with openSearcher=false - it's just about flushing the transaction log and freeing memory in this configuraiton - which is why it makes sense to simply turn on the auto commit for regular hard commits. You may or may not want to use auto soft commits.
- Mark
>
>
> -----Ursprüngliche Nachricht-----
> Von: Mark Miller [mailto:markrmiller@gmail.com]
> Gesendet: Dienstag, 27. August 2013 17:12
> An: solr-user@lucene.apache.org
> Betreff: Re: Transaction log "on-disk" guarantees
>
>
> On Aug 27, 2013, at 11:08 AM, Sandro Zbinden <zb...@imagic.ch> wrote:
>
>> Can we activate the transaction log to have on disk guarantees and then use the solr soft commit feature ?
>
> Yes you can. If you only have a single node (no replication), you probably want to turn on fsync via the config.
>
> - Mark
>
Re: Transaction log "on-disk" guarantees
Posted by Mark Miller <ma...@gmail.com>.
On Aug 27, 2013, at 11:54 AM, Erick Erickson <er...@gmail.com> wrote:
> Soft commits flush to the op system, so
> a JVM crash/termination shouldn't affect it anyway.
A soft commit is not a hard commit, so there are not guarantees like this. It searches committed and non committed segments - non committed segments will not magically be committed after a JVM crash.
> Turning on the fsync
> would just be a little bit of extra protection….
If you don't have replication, it turns on strong 'durability' promises. Without it, you are on your own if you have a hard machine reset. If durability is important to you and you don't have replication, it's important to use the fync option here. Unless you have a great, long time, battery backup and/or an env such that hard resets don't concern you for some reason. It comes down to your requirments.
Responses to Sandro inline below:
On Aug 27, 2013, at 11:43 AM, Sandro Zbinden <zb...@imagic.ch> wrote:
> Hey Mark
>
> Thank you very much for the quick answer. We have a single node environment.
>
> I try to find the fsync option but was not successful. Ended up in the UpdateLog class :-)
>
> How do I enable fsync in the solrconfig.xml ?
In the updateLog config, its a syncLevel=fsync param.
>
>
> Besides that:
>
> If solr soft commit feature has a on disk guarantee with a transaction log why we don't use solr soft commit as the default commit option ?
Yes, for visibility you should use soft commit. You should also have an auto hard commit with openSearcher=false - it's just about flushing the transaction log and freeing memory in this configuraiton - which is why it makes sense to simply turn on the auto commit for regular hard commits. You may or may not want to use auto soft commits.
- Mark
>
>
> -----Ursprüngliche Nachricht-----
> Von: Mark Miller [mailto:markrmiller@gmail.com]
> Gesendet: Dienstag, 27. August 2013 17:12
> An: solr-user@lucene.apache.org
> Betreff: Re: Transaction log "on-disk" guarantees
>
>
> On Aug 27, 2013, at 11:08 AM, Sandro Zbinden <zb...@imagic.ch> wrote:
>
>> Can we activate the transaction log to have on disk guarantees and then use the solr soft commit feature ?
>
> Yes you can. If you only have a single node (no replication), you probably want to turn on fsync via the config.
>
> - Mark
>
AW: Transaction log "on-disk" guarantees
Posted by Sandro Zbinden <zb...@imagic.ch>.
Hey Mark
Thank you very much for the quick answer. We have a single node environment.
I try to find the fsync option but was not successful. Ended up in the UpdateLog class :-)
How do I enable fsync in the solrconfig.xml ?
Besides that:
If solr soft commit feature has a on disk guarantee with a transaction log why we don't use solr soft commit as the default commit option ?
-----Ursprüngliche Nachricht-----
Von: Mark Miller [mailto:markrmiller@gmail.com]
Gesendet: Dienstag, 27. August 2013 17:12
An: solr-user@lucene.apache.org
Betreff: Re: Transaction log "on-disk" guarantees
On Aug 27, 2013, at 11:08 AM, Sandro Zbinden <zb...@imagic.ch> wrote:
> Can we activate the transaction log to have on disk guarantees and then use the solr soft commit feature ?
Yes you can. If you only have a single node (no replication), you probably want to turn on fsync via the config.
- Mark
Re: Transaction log "on-disk" guarantees
Posted by Mark Miller <ma...@gmail.com>.
On Aug 27, 2013, at 11:08 AM, Sandro Zbinden <zb...@imagic.ch> wrote:
> Can we activate the transaction log to have on disk guarantees and then use the solr soft commit feature ?
Yes you can. If you only have a single node (no replication), you probably want to turn on fsync via the config.
- Mark