You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Arkadi Colson <ar...@smartbit.be> on 2016/09/01 12:35:24 UTC
Replication Index fetch failed
Hi
Replication seems to be in an endless loop. Anybody any idea?
See below for logs.
If you need more info, just let me know...
INFO - 2016-09-01 14:30:42.563; [c:lvs s:shard1 r:core_node10
x:lvs_shard1_replica1] org.apache.solr.core.SolrDeletionPolicy;
SolrDeletionPolicy.onCommit: commits: num=2
commit{dir=NRTCachingDirectory(MMapDirectory@/var/solr/data/lvs_shard1_replica1/data/index.20160901140036922
lockFactory=org.apache.lucene.store.NativeFSLockFactory@59509f2;
maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_vpo,generation=41100}
commit{dir=NRTCachingDirectory(MMapDirectory@/var/solr/data/lvs_shard1_replica1/data/index.20160901140036922
lockFactory=org.apache.lucene.store.NativeFSLockFactory@59509f2;
maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_vpp,generation=41101}
INFO - 2016-09-01 14:30:42.563; [c:lvs s:shard1 r:core_node10
x:lvs_shard1_replica1] org.apache.solr.core.SolrDeletionPolicy; newest
commit generation = 41101
INFO - 2016-09-01 14:30:42.565; [c:lvs s:shard1 r:core_node10
x:lvs_shard1_replica1] org.apache.solr.update.DirectUpdateHandler2;
end_commit_flush
INFO - 2016-09-01 14:30:42.565; [c:lvs s:shard1 r:core_node10
x:lvs_shard1_replica1] org.apache.solr.update.DirectUpdateHandler2;
start
commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
INFO - 2016-09-01 14:30:42.603; [c:lvs s:shard1 r:core_node10
x:lvs_shard1_replica1] org.apache.solr.core.SolrDeletionPolicy;
SolrDeletionPolicy.onCommit: commits: num=2
commit{dir=NRTCachingDirectory(MMapDirectory@/var/solr/data/lvs_shard1_replica1/data/index.20160901140036922
lockFactory=org.apache.lucene.store.NativeFSLockFactory@59509f2;
maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_vpp,generation=41101}
commit{dir=NRTCachingDirectory(MMapDirectory@/var/solr/data/lvs_shard1_replica1/data/index.20160901140036922
lockFactory=org.apache.lucene.store.NativeFSLockFactory@59509f2;
maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_vpq,generation=41102}
INFO - 2016-09-01 14:30:42.603; [c:lvs s:shard1 r:core_node10
x:lvs_shard1_replica1] org.apache.solr.core.SolrDeletionPolicy; newest
commit generation = 41102
INFO - 2016-09-01 14:30:42.664; [c:lvs s:shard1 r:core_node10
x:lvs_shard1_replica1] org.apache.solr.search.SolrIndexSearcher; Opening
[Searcher@3436f207[lvs_shard1_replica1] realtime]
INFO - 2016-09-01 14:30:42.674; [c:lvs s:shard1 r:core_node10
x:lvs_shard1_replica1] org.apache.solr.update.DirectUpdateHandler2;
end_commit_flush
ERROR - 2016-09-01 14:30:43.653; [c:intradesk s:shard1 r:core_node5
x:intradesk_shard1_replica1] org.apache.solr.common.SolrException; Index
fetch failed :org.apache.solr.common.SolrException: Unable to download
_6f46_cj.liv completely. Downloaded 0!=5596
at
org.apache.solr.handler.IndexFetcher$FileFetcher.cleanup(IndexFetcher.java:1554)
at
org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1437)
at
org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:852)
at
org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:428)
at
org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:251)
at
org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:388)
at
org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:156)
at
org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:408)
at
org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:221)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$22(ExecutorUtil.java:229)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
ERROR - 2016-09-01 14:30:43.654; [c:intradesk s:shard1 r:core_node5
x:intradesk_shard1_replica1] org.apache.solr.common.SolrException; Error
while trying to recover:org.apache.solr.common.SolrException:
Replication for recovery failed.
at
org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:159)
at
org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:408)
at
org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:221)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$22(ExecutorUtil.java:229)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
INFO - 2016-09-01 14:30:43.654; [c:intradesk s:shard1 r:core_node5
x:intradesk_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy;
Replay not started, or was not successful... still buffering updates.
ERROR - 2016-09-01 14:30:43.654; [c:intradesk s:shard1 r:core_node5
x:intradesk_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy;
Recovery failed - trying again... (0)
INFO - 2016-09-01 14:30:43.654; [c:intradesk s:shard1 r:core_node5
x:intradesk_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy;
Wait [2.0] seconds before trying to recover again (attempt=1)
INFO - 2016-09-01 14:30:53.655; [c:intradesk s:shard1 r:core_node5
x:intradesk_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy;
Begin buffering updates. core=[intradesk_shard1_replica1]
INFO - 2016-09-01 14:30:53.747; [c:intradesk s:shard1 r:core_node5
x:intradesk_shard1_replica1] org.apache.solr.update.UpdateLog;
Restarting buffering. previous=RecoveryInfo{adds=0 deletes=0
deleteByQuery=0 errors=0 positionOfStart=0}
INFO - 2016-09-01 14:30:53.748; [c:intradesk s:shard1 r:core_node5
x:intradesk_shard1_replica1] org.apache.solr.update.UpdateLog; Starting
to buffer updates. FSUpdateLog{state=BUFFERING,
tlog=tlog{file=/var/solr/data/intradesk_shard1_replica1/data/tlog/tlog.0000000000000122833
refcount=1}}
INFO - 2016-09-01 14:30:53.748; [c:intradesk s:shard1 r:core_node5
x:intradesk_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy;
Publishing state of core [intradesk_shard1_replica1] as recovering,
leader is [http://solr20-gs:8983/solr/intradesk_shard1_replica2/] and I
am [http://solr20-dcg:8983/solr/intradesk_shard1_replica1/]
INFO - 2016-09-01 14:30:53.748; [c:intradesk s:shard1 r:core_node5
x:intradesk_shard1_replica1] org.apache.solr.cloud.ZkController;
publishing state=recovering
INFO - 2016-09-01 14:30:53.768; [c:intradesk s:shard1 r:core_node5
x:intradesk_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy;
Sending prep recovery command to [http://solr20-gs:8983/solr];
[WaitForState:
action=PREPRECOVERY&core=intradesk_shard1_replica2&nodeName=solr20-dcg:8983_solr&coreNodeName=core_node5&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
INFO - 2016-09-01 14:30:53.780; [ ]
org.apache.solr.common.cloud.ZkStateReader$StateWatcher; A cluster state
change: [WatchedEvent state:SyncConnected type:NodeDataChanged
path:/collections/intradesk/state.json] for collection [intradesk] has
occurred - updating... (live nodes size: [21])
INFO - 2016-09-01 14:30:53.781; [ ]
org.apache.solr.common.cloud.ZkStateReader; Updating data for
[intradesk] from [12593] to [12594]
INFO - 2016-09-01 14:31:00.776; [c:intradesk s:shard1 r:core_node5
x:intradesk_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy;
Starting Replication Recovery.
INFO - 2016-09-01 14:31:00.776; [c:intradesk s:shard1 r:core_node5
x:intradesk_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy;
Attempting to replicate from
[http://solr20-gs:8983/solr/intradesk_shard1_replica2/].
INFO - 2016-09-01 14:31:00.783; [c:intradesk s:shard1 r:core_node5
x:intradesk_shard1_replica1] org.apache.solr.handler.IndexFetcher;
Master's generation: 123527
INFO - 2016-09-01 14:31:00.783; [c:intradesk s:shard1 r:core_node5
x:intradesk_shard1_replica1] org.apache.solr.handler.IndexFetcher;
Slave's generation: 2
INFO - 2016-09-01 14:31:00.783; [c:intradesk s:shard1 r:core_node5
x:intradesk_shard1_replica1] org.apache.solr.handler.IndexFetcher;
Starting replication process
INFO - 2016-09-01 14:31:01.461; [c:intradesk s:shard1 r:core_node5
x:intradesk_shard1_replica1] org.apache.solr.handler.IndexFetcher;
Number of files in latest index in master: 210
INFO - 2016-09-01 14:31:01.462; [c:intradesk s:shard1 r:core_node5
x:intradesk_shard1_replica1]
org.apache.solr.core.CachingDirectoryFactory; return new directory for
/var/solr/data/intradesk_shard1_replica1/data/index.20160901143101462
INFO - 2016-09-01 14:31:01.463; [c:intradesk s:shard1 r:core_node5
x:intradesk_shard1_replica1] org.apache.solr.handler.IndexFetcher;
Starting download (fullCopy=true) to
NRTCachingDirectory(MMapDirectory@/var/solr/data/intradesk_shard1_replica1/data/index.20160901143101462
lockFactory=org.apache.lucene.store.NativeFSLockFactory@59509f2;
maxCacheMB=48.0 maxMergeSizeMB=4.0)
Re: Replication Index fetch failed
Posted by Arkadi Colson <ar...@smartbit.be>.
Hi
I could not find "Could not download file" in the logs. Should I
increase the log level somewhere? Just let me know... so I can provide
you more detailed logs...
Thx!
Arkadi
On 02-09-16 11:21, Arkadi Colson wrote:
> Hi
>
> I cannot find a string in the logs matching "Could not download file...".
>
> This info is logged on the slave:
>
> WARN - 2016-09-02 09:28:36.923; [c:intradesk s:shard10 r:core_node23
> x:intradesk_shard10_replica1]
> org.apache.solr.handler.IndexFetcher$FileFetcher; Error in fetching
> file: _5qd6_ya.liv (downloaded 0 of 13692 bytes)
> java.io.EOFException
> at
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
> at
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1460)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1426)
> at
> org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:852)
> at
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:428)
> at
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:251)
> at
> org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:388)
> at
> org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:156)
> at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:408)
> at
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:221)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$22(ExecutorUtil.java:229)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> And this on the master:
>
> WARN - 2016-09-02 09:28:36.936; [c:intradesk s:shard10 r:core_node13
> x:intradesk_shard10_replica2]
> org.apache.solr.handler.ReplicationHandler$DirectoryFileStream;
> Exception while writing response for params:
> generation=124148&qt=/replication&file=_5qd6_ya.liv&checksum=true&wt=filestream&command=filecontent&maxWriteM
> BPerSec=18.75
> java.nio.file.NoSuchFileException:
> /var/solr/data/intradesk_shard10_replica2/data/index.20160816102332501/_5qd6_ya.liv
> at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
> at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> at
> sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
> at java.nio.channels.FileChannel.open(FileChannel.java:287)
> at java.nio.channels.FileChannel.open(FileChannel.java:335)
> at
> org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:238)
> at
> org.apache.lucene.store.NRTCachingDirectory.openInput(NRTCachingDirectory.java:192)
> at
> org.apache.solr.handler.ReplicationHandler$DirectoryFileStream.write(ReplicationHandler.java:1435)
> at org.apache.solr.core.SolrCore$3.write(SolrCore.java:2154)
> at
> org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:49)
> at
> org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:731)
> at
> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:473)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.eclipse.jetty.server.Server.handle(Server.java:518)
> at
> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
> at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
> at
> org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
> at
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
>
> Or should I enable more logging somewhere? FYI, the file
> /var/solr/data/intradesk_shard10_replica2/data/index.20160816102332501/_5qd6_ya.liv
> does NOT exist on the master.
>
> root@solr29-gs:~# ls
> /var/solr/data/intradesk_shard10_replica2/data/index.20160816102332501/
> | grep 5qd6
> _5qd6.fdt
> _5qd6.fdx
> _5qd6.fnm
> _5qd6_Lucene50_0.doc
> _5qd6_Lucene50_0.pos
> _5qd6_Lucene50_0.tim
> _5qd6_Lucene50_0.tip
> _5qd6.si
> _5qd6_yi.liv
>
> Thanks!
>
> Arkadi
>
>
> On 02-09-16 06:43, Shalin Shekhar Mangar wrote:
>>
>> On Thu, Sep 1, 2016 at 6:05 PM, Arkadi Colson <arkadi@smartbit.be
>> <ma...@smartbit.be>> wrote:
>>
>> ERROR - 2016-09-01 14:30:43.653; [c:intradesk s:shard1
>> r:core_node5 x:intradesk_shard1_replica1]
>> org.apache.solr.common.SolrException; Index fetch failed
>> :org.apache.solr.common.SolrException: Unable to download
>> _6f46_cj.liv completely. Downloaded 0!=5596
>> at
>> org.apache.solr.handler.IndexFetcher$FileFetcher.cleanup(IndexFetcher.java:1554)
>> at
>> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1437)
>> at
>> org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:852)
>> at
>> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:428)
>> at
>> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:251)
>>
>>
>>
>> There should be another exception in the logs that looks like the
>> following:
>> "Could not download file"...
>>
>> That one will have a more useful stack trace. Can you please find it
>> and paste it on email?
>>
>> --
>> Regards,
>> Shalin Shekhar Mangar.
>
>
Re: Replication Index fetch failed
Posted by Arkadi Colson <ar...@smartbit.be>.
Hi
I cannot find a string in the logs matching "Could not download file...".
This info is logged on the slave:
WARN - 2016-09-02 09:28:36.923; [c:intradesk s:shard10 r:core_node23
x:intradesk_shard10_replica1]
org.apache.solr.handler.IndexFetcher$FileFetcher; Error in fetching
file: _5qd6_ya.liv (downloaded 0 of 13692 bytes)
java.io.EOFException
at
org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
at
org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
at
org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1460)
at
org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1426)
at
org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:852)
at
org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:428)
at
org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:251)
at
org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:388)
at
org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:156)
at
org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:408)
at
org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:221)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$22(ExecutorUtil.java:229)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
And this on the master:
WARN - 2016-09-02 09:28:36.936; [c:intradesk s:shard10 r:core_node13
x:intradesk_shard10_replica2]
org.apache.solr.handler.ReplicationHandler$DirectoryFileStream;
Exception while writing response for params:
generation=124148&qt=/replication&file=_5qd6_ya.liv&checksum=true&wt=filestream&command=filecontent&maxWriteM
BPerSec=18.75
java.nio.file.NoSuchFileException:
/var/solr/data/intradesk_shard10_replica2/data/index.20160816102332501/_5qd6_ya.liv
at
sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at
sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
at java.nio.channels.FileChannel.open(FileChannel.java:287)
at java.nio.channels.FileChannel.open(FileChannel.java:335)
at
org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:238)
at
org.apache.lucene.store.NRTCachingDirectory.openInput(NRTCachingDirectory.java:192)
at
org.apache.solr.handler.ReplicationHandler$DirectoryFileStream.write(ReplicationHandler.java:1435)
at org.apache.solr.core.SolrCore$3.write(SolrCore.java:2154)
at
org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:49)
at
org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:731)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:473)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:518)
at
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
Or should I enable more logging somewhere? FYI, the file
/var/solr/data/intradesk_shard10_replica2/data/index.20160816102332501/_5qd6_ya.liv
does NOT exist on the master.
root@solr29-gs:~# ls
/var/solr/data/intradesk_shard10_replica2/data/index.20160816102332501/
| grep 5qd6
_5qd6.fdt
_5qd6.fdx
_5qd6.fnm
_5qd6_Lucene50_0.doc
_5qd6_Lucene50_0.pos
_5qd6_Lucene50_0.tim
_5qd6_Lucene50_0.tip
_5qd6.si
_5qd6_yi.liv
Thanks!
Arkadi
On 02-09-16 06:43, Shalin Shekhar Mangar wrote:
>
> On Thu, Sep 1, 2016 at 6:05 PM, Arkadi Colson <arkadi@smartbit.be
> <ma...@smartbit.be>> wrote:
>
> ERROR - 2016-09-01 14:30:43.653; [c:intradesk s:shard1
> r:core_node5 x:intradesk_shard1_replica1]
> org.apache.solr.common.SolrException; Index fetch failed
> :org.apache.solr.common.SolrException: Unable to download
> _6f46_cj.liv completely. Downloaded 0!=5596
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.cleanup(IndexFetcher.java:1554)
> at
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1437)
> at
> org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:852)
> at
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:428)
> at
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:251)
>
>
>
> There should be another exception in the logs that looks like the
> following:
> "Could not download file"...
>
> That one will have a more useful stack trace. Can you please find it
> and paste it on email?
>
> --
> Regards,
> Shalin Shekhar Mangar.
Re: Replication Index fetch failed
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
On Thu, Sep 1, 2016 at 6:05 PM, Arkadi Colson <ar...@smartbit.be> wrote:
> ERROR - 2016-09-01 14:30:43.653; [c:intradesk s:shard1 r:core_node5
> x:intradesk_shard1_replica1] org.apache.solr.common.SolrException; Index
> fetch failed :org.apache.solr.common.SolrException: Unable to download
> _6f46_cj.liv completely. Downloaded 0!=5596
> at org.apache.solr.handler.IndexFetcher$FileFetcher.cleanup(
> IndexFetcher.java:1554)
> at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(
> IndexFetcher.java:1437)
> at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(Inde
> xFetcher.java:852)
> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexF
> etcher.java:428)
> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexF
> etcher.java:251)
>
There should be another exception in the logs that looks like the following:
"Could not download file"...
That one will have a more useful stack trace. Can you please find it and
paste it on email?
--
Regards,
Shalin Shekhar Mangar.