You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "yuanyun.cn (JIRA)" <ji...@apache.org> on 2014/12/01 18:21:13 UTC
[jira] [Comment Edited] (SOLR-6776) Data lost when use SoftCommit
and TLog
[ https://issues.apache.org/jira/browse/SOLR-6776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14230053#comment-14230053 ]
yuanyun.cn edited comment on SOLR-6776 at 12/1/14 5:20 PM:
-----------------------------------------------------------
The finish of UpdateProcessoris is always called in org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest, SolrQueryResponse).
When we add a doc without softcommit or commit, the org.apache.solr.update.processor.RunUpdateProcessor.finish() will call getUpdateLog().finish() to fsync the tlog.
But if add a doc with softcommit=true, RunUpdateProcessor.finish() will not call getUpdateLog().finish(), and will not fsync the tlog.
This is kind of not right.
User enables transaction log for data durability, to make sure there is no data lost.
So I think it should always fsync the tlog after add this doc to solr and before the hard commit.
was (Author: yuanyun.cn):
The finish of UpdateProcessoris always in org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQueryRequest, SolrQueryResponse).
> Data lost when use SoftCommit and TLog
> --------------------------------------
>
> Key: SOLR-6776
> URL: https://issues.apache.org/jira/browse/SOLR-6776
> Project: Solr
> Issue Type: Bug
> Affects Versions: 4.10
> Reporter: yuanyun.cn
> Labels: softCommit, updateLog
> Fix For: 4.10.3
>
>
> We enabled update log and change autoCommit to some bigger value 10 mins.
> After restart, we push one doc with softCommit=true
> http://localhost:8983/solr/update?stream.body=<add><doc><field name="id">id1</field></doc></add>&softCommit=true
> Then we kill the java process after a min.
> After restart, Tlog failed to replay with following exception, and there is no data in solr.
> 6245 [coreLoadExecutor-5-thread-1] ERROR org.apache.solr.update.UpdateLog รป Failure to open existing log file (non fatal) E:\jeffery\src\apache\solr\4.10.2\solr-4.10.2\example\solr\collection1\data\t
> log\tlog.0000000000000000000:org.apache.solr.common.SolrException: java.io.EOFException
> at org.apache.solr.update.TransactionLog.<init>(TransactionLog.java:181)
> at org.apache.solr.update.UpdateLog.init(UpdateLog.java:261)
> at org.apache.solr.update.UpdateHandler.<init>(UpdateHandler.java:134)
> at org.apache.solr.update.UpdateHandler.<init>(UpdateHandler.java:94)
> at org.apache.solr.update.DirectUpdateHandler2.<init>(DirectUpdateHandler2.java:100)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
> at java.lang.reflect.Constructor.newInstance(Unknown Source)
> at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:550)
> at org.apache.solr.core.SolrCore.createUpdateHandler(SolrCore.java:620)
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:835)
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:646)
> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:491)
> at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:255)
> at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
> at java.util.concurrent.FutureTask.run(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.EOFException
> at org.apache.solr.common.util.FastInputStream.readUnsignedByte(FastInputStream.java:73)
> at org.apache.solr.common.util.FastInputStream.readInt(FastInputStream.java:216)
> at org.apache.solr.update.TransactionLog.readHeader(TransactionLog.java:268)
> at org.apache.solr.update.TransactionLog.<init>(TransactionLog.java:159)
> ... 19 more
> Check the code: seems this is related with: org.apache.solr.update.processor.RunUpdateProcessor, in processCommit, it sets changesSinceCommit=false(even we are using softCommit)
> So in finish, updateLog.finish will not be called.
> public void finish() throws IOException {
> if (changesSinceCommit && updateHandler.getUpdateLog() != null) {
> updateHandler.getUpdateLog().finish(null);
> }
> super.finish();
> }
> To fix this issue: I have to change RunUpdateProcessor.processCommit like below:
> if (!cmd.softCommit) {
> changesSinceCommit = false;
> }
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org