You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Matthieu Labour <ma...@yahoo.com> on 2010/02/26 19:43:01 UTC
replication issue
Hi
I am still having issues with the replication and wonder if things are working properly
So I have 1 master and 1 slave
On the slave, I deleted the data/index directory and data/replication.properties file and restarted solr.
When slave is pulling data from master, I can see that the size of data directory is growing
root@slr8:/raid/data# du -sh
3.7M .
root@slr8:/raid/data# du -sh
4.7M .
and I can see that data/replication.properties file got created and also a directory data/index.20100226063400
soon after index.20100226063400 disapears and the size of data/index is back to 12K
root@slr8:/raid/data/index# du -sh
12K .
And when I look for the number of documents via the admin interface, I still see 0 documents so I feel something is wrong
One more thing, I have a symlink for /solr/data ---> /raid/data
Thank you for your help !
matt
Re: replication issue
Posted by Matthieu Labour <ma...@yahoo.com>.
Shalin
Thank you so much for your answer
This is the case here
How can I find out which temp directory Solr replication is using?
Do you have a way to set up the source (temp directory? used by solr) and target directory via solr config file so that they live on the same partition ?
Thank you
matt
--- On Fri, 2/26/10, Shalin Shekhar Mangar <sh...@gmail.com> wrote:
From: Shalin Shekhar Mangar <sh...@gmail.com>
Subject: Re: replication issue
To: solr-user@lucene.apache.org
Date: Friday, February 26, 2010, 2:06 PM
On Sat, Feb 27, 2010 at 12:13 AM, Matthieu Labour <matthieu_labour@yahoo.com
> wrote:
> Hi
>
> I am still having issues with the replication and wonder if things are
> working properly
>
> So I have 1 master and 1 slave
>
> On the slave, I deleted the data/index directory and
> data/replication.properties file and restarted solr.
>
> When slave is pulling data from master, I can see that the size of data
> directory is growing
>
> root@slr8:/raid/data# du -sh
> 3.7M .
> root@slr8:/raid/data# du -sh
> 4.7M .
>
> and I can see that data/replication.properties file got created and also a
> directory data/index.20100226063400
>
> soon after index.20100226063400 disapears and the size of data/index is
> back to 12K
>
> root@slr8:/raid/data/index# du -sh
> 12K .
>
> And when I look for the number of documents via the admin interface, I
> still see 0 documents so I feel something is wrong
>
> One more thing, I have a symlink for /solr/data ---> /raid/data
>
The ReplicationHandler moves files out of the temp directory into the index
directory. Java's File#renameTo can fail if the source and target
directories are on different partitions/disks. Is that the case here? I
believe SOLR-1736 fixes this issue in trunk but that was implemented after
the 1.4 release.
--
Regards,
Shalin Shekhar Mangar.
Re: replication issue
Posted by Matthieu Labour <ma...@yahoo.com>.
This replication does not work well. temp directory and /data/index are on different device/disks
I see the following message
[2010-03-02 01:22:07] [pool-3-thread-1] ERROR(ReplicationHandler.java:266) - SnapPull failed
And Yet I applied the patch SOLR-1736
I ll uni test patch SOLR-1736 and see what tmpIdxDir gets picked up...
What would be cool is the ability to set up solr temp file via config file so that it can live in the same partition than the data directory
Thank you
--- On Fri, 2/26/10, Shalin Shekhar Mangar <sh...@gmail.com> wrote:
From: Shalin Shekhar Mangar <sh...@gmail.com>
Subject: Re: replication issue
To: solr-user@lucene.apache.org
Date: Friday, February 26, 2010, 2:06 PM
On Sat, Feb 27, 2010 at 12:13 AM, Matthieu Labour <matthieu_labour@yahoo.com
> wrote:
> Hi
>
> I am still having issues with the replication and wonder if things are
> working properly
>
> So I have 1 master and 1 slave
>
> On the slave, I deleted the data/index directory and
> data/replication.properties file and restarted solr.
>
> When slave is pulling data from master, I can see that the size of data
> directory is growing
>
> root@slr8:/raid/data# du -sh
> 3.7M .
> root@slr8:/raid/data# du -sh
> 4.7M .
>
> and I can see that data/replication.properties file got created and also a
> directory data/index.20100226063400
>
> soon after index.20100226063400 disapears and the size of data/index is
> back to 12K
>
> root@slr8:/raid/data/index# du -sh
> 12K .
>
> And when I look for the number of documents via the admin interface, I
> still see 0 documents so I feel something is wrong
>
> One more thing, I have a symlink for /solr/data ---> /raid/data
>
The ReplicationHandler moves files out of the temp directory into the index
directory. Java's File#renameTo can fail if the source and target
directories are on different partitions/disks. Is that the case here? I
believe SOLR-1736 fixes this issue in trunk but that was implemented after
the 1.4 release.
--
Regards,
Shalin Shekhar Mangar.
Re: replication issue
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
On Sat, Feb 27, 2010 at 12:13 AM, Matthieu Labour <matthieu_labour@yahoo.com
> wrote:
> Hi
>
> I am still having issues with the replication and wonder if things are
> working properly
>
> So I have 1 master and 1 slave
>
> On the slave, I deleted the data/index directory and
> data/replication.properties file and restarted solr.
>
> When slave is pulling data from master, I can see that the size of data
> directory is growing
>
> root@slr8:/raid/data# du -sh
> 3.7M .
> root@slr8:/raid/data# du -sh
> 4.7M .
>
> and I can see that data/replication.properties file got created and also a
> directory data/index.20100226063400
>
> soon after index.20100226063400 disapears and the size of data/index is
> back to 12K
>
> root@slr8:/raid/data/index# du -sh
> 12K .
>
> And when I look for the number of documents via the admin interface, I
> still see 0 documents so I feel something is wrong
>
> One more thing, I have a symlink for /solr/data ---> /raid/data
>
The ReplicationHandler moves files out of the temp directory into the index
directory. Java's File#renameTo can fail if the source and target
directories are on different partitions/disks. Is that the case here? I
believe SOLR-1736 fixes this issue in trunk but that was implemented after
the 1.4 release.
--
Regards,
Shalin Shekhar Mangar.
Re: replication issue
Posted by Matthieu Labour <ma...@yahoo.com>.
Hi
I just want to post a follow up on the replication issue I encountered
I have a master on which many document updates (delete and add) are happening
There is one slave replicating from the master. There is only search request hitting the slave.
I can see the size of the downloaded data increasing on the slave in the index.XXXXX.
And then I see the following error in the log file
[2010-03-02 21:24:40] [pool-3-thread-1] ERROR(ReplicationHandler.java:266) - SnapPull failed org.apache.solr.common.SolrException: Unable to download _7h0y.fdx completely.
the entire index.XXXXX gets deleted and no data is being merged with index even though some data got downloaded already...like if all the files downloaded were part of the same transaction
_7h0y.fdx is no longer on the master
Increasing commitReserveDuration to 1 hour allowed enough time for the data to be downloaded on the slave without any deletion happening on the master. Therefore I didn't see the SolrException in the slave log files and the replication worked
Thank you
--- On Tue, 3/2/10, Matthieu Labour <ma...@yahoo.com> wrote:
From: Matthieu Labour <ma...@yahoo.com>
Subject: Re: replication issue
To: solr-user@lucene.apache.org
Date: Tuesday, March 2, 2010, 4:59 PM
Otis
Thank your for your response. I apologize for not being specific enough
--> yes it happened over & over.
--> apache-solr-1.4.0
--> I restarted the indexing+replication from scratch. Before I did that, I backed up the master index directory. I don't see _7h0y.fdx in it ....
What could have possibly happen?
--- On Tue, 3/2/10, Otis Gospodnetic <ot...@yahoo.com> wrote:
From: Otis Gospodnetic <ot...@yahoo.com>
Subject: Re: replication issue
To: solr-user@lucene.apache.org
Date: Tuesday, March 2, 2010, 4:40 PM
Hi Matthieu,
Does this happen over and over?
Is this with Solr 1.4 or some other version?
Is there anything unusual about _7h0y.fdx?
Does _7h0y.fdx still exist on the master when the replication fails?
...
Otis
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
Hadoop ecosystem search :: http://search-hadoop.com/
----- Original Message ----
> From: Matthieu Labour <ma...@yahoo.com>
> To: solr-user@lucene.apache.org
> Sent: Tue, March 2, 2010 4:35:46 PM
> Subject: Re: replication issue
>
> The replication does not work for me
>
>
> I have a big master solr and I want to start replicating it. I can see that the
> slave is downloading data from the master... I see a directory
> index.20100302093000 gets created in data/ next to index... I can see its size
> growing but then the directory gets deleted
>
> Here is the complete trace (I added a couple of LOG messages and compile solr)
>
> [2010-03-02 21:24:00] [pool-3-thread-1]
> DEBUG(MultiThreadedHttpConnectionManager.java:961) - Notifying no-one, there are
> no waiting threads
> [2010-03-02 21:24:00] [pool-3-thread-1] INFO (SnapPuller.java:278) - Number of
> files in latest index in master: 163
> [2010-03-02 21:24:00] [pool-3-thread-1] DEBUG(SnapPuller.java:536) -
> downloadIndexFiles(downloadCompleteIndex=false,tmpIdxDir=../solr/data/index.20100302092400,latestVersion=1266003907838)
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(SnapPuller.java:541) -
> -->localIndexFile=/opt/solr_env/solr/data/index/_7h0y.fdx
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(SnapPuller.java:900) - fetchFile()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:265) - enter
> PostMethod.addParameter(String, String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:150) -
> enter EntityEnclosingMethod.clearRequestBody()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:265) - enter
> PostMethod.addParameter(String, String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:150) -
> enter EntityEnclosingMethod.clearRequestBody()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:265) - enter
> PostMethod.addParameter(String, String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:150) -
> enter EntityEnclosingMethod.clearRequestBody()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:265) - enter
> PostMethod.addParameter(String, String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:150) -
> enter EntityEnclosingMethod.clearRequestBody()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:265) - enter
> PostMethod.addParameter(String, String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:150) -
> enter EntityEnclosingMethod.clearRequestBody()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpClient.java:321) - enter
> HttpClient.executeMethod(HttpMethod)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpClient.java:374) - enter
> HttpClient.executeMethod(HostConfiguration,HttpMethod,HttpState)
> [2010-03-02 21:24:40] [pool-3-thread-1]
> TRACE(MultiThreadedHttpConnectionManager.java:405) - enter
> HttpConnectionManager.getConnectionWithTimeout(HostConfiguration, long)
> [2010-03-02 21:24:40] [pool-3-thread-1]
> DEBUG(MultiThreadedHttpConnectionManager.java:412) -
> HttpConnectionManager.getConnection: config =
> HostConfiguration[host=http://myserver.com:8983], timeout = 0
> [2010-03-02 21:24:40] [pool-3-thread-1]
> TRACE(MultiThreadedHttpConnectionManager.java:805) - enter
> HttpConnectionManager.ConnectionPool.getHostPool(HostConfiguration)
> [2010-03-02 21:24:40] [pool-3-thread-1]
> TRACE(MultiThreadedHttpConnectionManager.java:805) - enter
> HttpConnectionManager.ConnectionPool.getHostPool(HostConfiguration)
> [2010-03-02 21:24:40] [pool-3-thread-1]
> DEBUG(MultiThreadedHttpConnectionManager.java:839) - Getting free connection,
> hostConfig=HostConfiguration[host=http://myserver.com:8983]
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodDirector.java:379) -
> Attempt number 1 to process request
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1079) - enter
> HttpMethodBase.execute(HttpState, HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:2057) - enter
> HttpMethodBase.writeRequest(HttpState, HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:2212) - enter
> HttpMethodBase.writeRequestLine(HttpState, HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1496) - enter
> HttpMethodBase.generateRequestLine(HttpConnection, String, String, String,
> String)
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - >> "POST
> /solr/replication HTTP/1.1[\r][\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:1032) - enter
> HttpConnection.print(String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:942) - enter
> HttpConnection.write(byte[])
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:963) - enter
> HttpConnection.write(byte[], int, int)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:2175) - enter
> HttpMethodBase.writeRequestHeaders(HttpState,HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:370) -
> enter EntityEnclosingMethod.addRequestHeaders(HttpState, HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(ExpectContinueMethod.java:183) -
> enter ExpectContinueMethod.addRequestHeaders(HttpState, HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1417) - enter
> HttpMethodBase.addRequestHeaders(HttpState, HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1442) - enter
> HttpMethodBase.addUserAgentRequestHeaders(HttpState, HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1328) - enter
> HttpMethodBase.addHostRequestHeader(HttpState, HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(HttpMethodBase.java:1352) - Adding
> Host request header
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1266) - enter
> HttpMethodBase.addCookieRequestHeader(HttpState, HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpState.java:173) - enter
> HttpState.getCookies()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(CookieSpecBase.java:549) - enter
> CookieSpecBase.match(String, int, String, boolean, Cookie[])
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1379) - enter
> HttpMethodBase.addProxyConnectionHeader(HttpState, HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:401) -
> enter EntityEnclosingMethod.addContentLengthRequestHeader(HttpState,
> HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:325) -
> enter EntityEnclosingMethod.getRequestContentLength()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:135) - enter
> PostMethod.hasRequestContent()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:240) - enter
> PostMethod.getParameters()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:2324) - enter
> getContentCharSet( Header contentheader )
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(HttpMethodBase.java:2342) -
> Default charset used: ISO-8859-1
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:386) -
> HttpMethodBase.addRequestHeader(Header)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:240) - enter
> PostMethod.getParameters()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:2324) - enter
> getContentCharSet( Header contentheader )
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HeaderElement.java:238) - enter
> HeaderElement.parseElements(String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HeaderElement.java:194) - enter
> HeaderElement.parseElements(char[])
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HeaderElement.java:279) - enter
> HeaderElement.getParameterByName(String)
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(HttpMethodBase.java:2342) -
> Default charset used: ISO-8859-1
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - >> "User-Agent:
> Jakarta Commons-HttpClient/3.1[\r][\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:1032) - enter
> HttpConnection.print(String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:942) - enter
> HttpConnection.write(byte[])
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:963) - enter
> HttpConnection.write(byte[], int, int)
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - >> "Host:
> myserver.com:8983[\r][\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:1032) - enter
> HttpConnection.print(String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:942) - enter
> HttpConnection.write(byte[])
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:963) - enter
> HttpConnection.write(byte[], int, int)
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - >>
> "Content-Length: 89[\r][\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:1032) - enter
> HttpConnection.print(String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:942) - enter
> HttpConnection.write(byte[])
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:963) - enter
> HttpConnection.write(byte[], int, int)
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - >> "Content-Type:
> application/x-www-form-urlencoded[\r][\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:1032) - enter
> HttpConnection.print(String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:942) - enter
> HttpConnection.write(byte[])
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:963) - enter
> HttpConnection.write(byte[], int, int)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:1001) - enter
> HttpConnection.writeLine()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:942) - enter
> HttpConnection.write(byte[])
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:963) - enter
> HttpConnection.write(byte[], int, int)
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - >> "[\r][\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:469) -
> enter EntityEnclosingMethod.writeRequestBody(HttpState, HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:135) - enter
> PostMethod.hasRequestContent()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:325) -
> enter EntityEnclosingMethod.getRequestContentLength()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:135) - enter
> PostMethod.hasRequestContent()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:840) - enter
> HttpConnection.getRequestOutputStream()
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:84) - >>
> "command=filecontent&indexversion=1266003907838&file=_7h0y.fdx&checksum=true&wt=filestream"
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(EntityEnclosingMethod.java:508) -
> Request body sent
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:826) - enter
> HttpConnection.flushRequestOutputStream()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1730) - enter
> HttpMethodBase.readResponse(HttpState, HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1964) - enter
> HttpMethodBase.readStatusLine(HttpState, HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:1113) - enter
> HttpConnection.readLine()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:105) - enter
> HttpParser.readLine(InputStream, String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:74) - enter
> HttpParser.readRawLine()
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "HTTP/1.1 200
> OK[\r][\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "HTTP/1.1 200
> OK[\r][\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1930) - enter
> HttpMethodBase.readResponseHeaders(HttpState,HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:857) - enter
> HttpConnection.getResponseInputStream()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:170) - enter
> HeaderParser.parseHeaders(InputStream, String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:105) - enter
> HttpParser.readLine(InputStream, String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:74) - enter
> HttpParser.readRawLine()
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "Date: Tue, 02
> Mar 2010 21:24:40 GMT[\r][\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:105) - enter
> HttpParser.readLine(InputStream, String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:74) - enter
> HttpParser.readRawLine()
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "Content-Type:
> application/octet-stream[\r][\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:105) - enter
> HttpParser.readLine(InputStream, String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:74) - enter
> HttpParser.readRawLine()
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - <<
> "Transfer-Encoding: chunked[\r][\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:105) - enter
> HttpParser.readLine(InputStream, String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:74) - enter
> HttpParser.readRawLine()
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "Server:
> Jetty(6.1.19)[\r][\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:105) - enter
> HttpParser.readLine(InputStream, String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:74) - enter
> HttpParser.readRawLine()
>
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "[\r][\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1579) - enter
> HttpMethodBase.processResponseHeaders(HttpState, HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1615) - enter
> HttpMethodBase.processCookieHeaders(Header[], HttpState, HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1781) - enter
> HttpMethodBase.readResponseBody(HttpState, HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1816) - enter
> HttpMethodBase.readResponseBody(HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:857) - enter
> HttpConnection.getResponseInputStream()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:2281) - enter
> HttpMethodBase.canResponseHaveBody(int)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HeaderElement.java:238) - enter
> HeaderElement.parseElements(String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HeaderElement.java:194) - enter
> HeaderElement.parseElements(char[])
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:894) - enter
> HttpConnection.isResponseAvailable(int)
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:84) - << "4"
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:84) - << "[\r]"
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "[\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:84) - <<
> "[0x0][0x0][0x0][0x0]"
> [2010-03-02 21:24:40] [pool-3-thread-1] WARN (SnapPuller.java:950) - No content
> recieved for file: {size=5324, name=_7h0y.fdx, lastmodified=1267565039000}
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:84) - << "[\r]"
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "[\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:84) - << "0"
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:84) - << "[\r]"
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "[\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:170) - enter
> HeaderParser.parseHeaders(InputStream, String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:105) - enter
> HttpParser.readLine(InputStream, String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:74) - enter
> HttpParser.readRawLine()
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:84) - << "[\r]"
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "[\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "[\r][\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(HttpMethodBase.java:1024) -
> Resorting to protocol version default close connection policy
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(HttpMethodBase.java:1028) - Should
> NOT close connection, using HTTP/1.1
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:874) - enter
> HttpConnection.isResponseAvailable()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:1174) - enter
> HttpConnection.releaseConnection()
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(HttpConnection.java:1178) -
> Releasing connection back to connection manager.
> [2010-03-02 21:24:40] [pool-3-thread-1]
> TRACE(MultiThreadedHttpConnectionManager.java:630) - enter
> HttpConnectionManager.releaseConnection(HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1]
> DEBUG(MultiThreadedHttpConnectionManager.java:979) - Freeing connection,
> hostConfig=HostConfiguration[host=http://myserver.com:8983]
> [2010-03-02 21:24:40] [pool-3-thread-1]
> TRACE(MultiThreadedHttpConnectionManager.java:805) - enter
> HttpConnectionManager.ConnectionPool.getHostPool(HostConfiguration)
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(IdleConnectionHandler.java:76) -
> Adding connection at: 1267565080137
> [2010-03-02 21:24:40] [pool-3-thread-1]
> DEBUG(MultiThreadedHttpConnectionManager.java:961) - Notifying no-one, there are
> no waiting threads
> [2010-03-02 21:24:40] [pool-3-thread-1] ERROR(ReplicationHandler.java:266) -
> SnapPull failed
> org.apache.solr.common.SolrException: Unable to download _7h0y.fdx completely.
> Downloaded 0!=5324
> at
> org.apache.solr.handler.SnapPuller$FileFetcher.cleanup(SnapPuller.java:1041)
> at
> org.apache.solr.handler.SnapPuller$FileFetcher.fetchFile(SnapPuller.java:921)
> at
> org.apache.solr.handler.SnapPuller.downloadIndexFiles(SnapPuller.java:545)
> at
> org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:294)
> at
> org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:264)
> at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:159)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:280)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:135)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:65)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:146)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:170)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
> at java.lang.Thread.run(Thread.java:595)
Re: replication issue
Posted by Matthieu Labour <ma...@yahoo.com>.
Otis
Thank your for your response. I apologize for not being specific enough
--> yes it happened over & over.
--> apache-solr-1.4.0
--> I restarted the indexing+replication from scratch. Before I did that, I backed up the master index directory. I don't see _7h0y.fdx in it ....
What could have possibly happen?
--- On Tue, 3/2/10, Otis Gospodnetic <ot...@yahoo.com> wrote:
From: Otis Gospodnetic <ot...@yahoo.com>
Subject: Re: replication issue
To: solr-user@lucene.apache.org
Date: Tuesday, March 2, 2010, 4:40 PM
Hi Matthieu,
Does this happen over and over?
Is this with Solr 1.4 or some other version?
Is there anything unusual about _7h0y.fdx?
Does _7h0y.fdx still exist on the master when the replication fails?
...
Otis
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
Hadoop ecosystem search :: http://search-hadoop.com/
----- Original Message ----
> From: Matthieu Labour <ma...@yahoo.com>
> To: solr-user@lucene.apache.org
> Sent: Tue, March 2, 2010 4:35:46 PM
> Subject: Re: replication issue
>
> The replication does not work for me
>
>
> I have a big master solr and I want to start replicating it. I can see that the
> slave is downloading data from the master... I see a directory
> index.20100302093000 gets created in data/ next to index... I can see its size
> growing but then the directory gets deleted
>
> Here is the complete trace (I added a couple of LOG messages and compile solr)
>
> [2010-03-02 21:24:00] [pool-3-thread-1]
> DEBUG(MultiThreadedHttpConnectionManager.java:961) - Notifying no-one, there are
> no waiting threads
> [2010-03-02 21:24:00] [pool-3-thread-1] INFO (SnapPuller.java:278) - Number of
> files in latest index in master: 163
> [2010-03-02 21:24:00] [pool-3-thread-1] DEBUG(SnapPuller.java:536) -
> downloadIndexFiles(downloadCompleteIndex=false,tmpIdxDir=../solr/data/index.20100302092400,latestVersion=1266003907838)
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(SnapPuller.java:541) -
> -->localIndexFile=/opt/solr_env/solr/data/index/_7h0y.fdx
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(SnapPuller.java:900) - fetchFile()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:265) - enter
> PostMethod.addParameter(String, String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:150) -
> enter EntityEnclosingMethod.clearRequestBody()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:265) - enter
> PostMethod.addParameter(String, String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:150) -
> enter EntityEnclosingMethod.clearRequestBody()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:265) - enter
> PostMethod.addParameter(String, String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:150) -
> enter EntityEnclosingMethod.clearRequestBody()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:265) - enter
> PostMethod.addParameter(String, String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:150) -
> enter EntityEnclosingMethod.clearRequestBody()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:265) - enter
> PostMethod.addParameter(String, String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:150) -
> enter EntityEnclosingMethod.clearRequestBody()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpClient.java:321) - enter
> HttpClient.executeMethod(HttpMethod)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpClient.java:374) - enter
> HttpClient.executeMethod(HostConfiguration,HttpMethod,HttpState)
> [2010-03-02 21:24:40] [pool-3-thread-1]
> TRACE(MultiThreadedHttpConnectionManager.java:405) - enter
> HttpConnectionManager.getConnectionWithTimeout(HostConfiguration, long)
> [2010-03-02 21:24:40] [pool-3-thread-1]
> DEBUG(MultiThreadedHttpConnectionManager.java:412) -
> HttpConnectionManager.getConnection: config =
> HostConfiguration[host=http://myserver.com:8983], timeout = 0
> [2010-03-02 21:24:40] [pool-3-thread-1]
> TRACE(MultiThreadedHttpConnectionManager.java:805) - enter
> HttpConnectionManager.ConnectionPool.getHostPool(HostConfiguration)
> [2010-03-02 21:24:40] [pool-3-thread-1]
> TRACE(MultiThreadedHttpConnectionManager.java:805) - enter
> HttpConnectionManager.ConnectionPool.getHostPool(HostConfiguration)
> [2010-03-02 21:24:40] [pool-3-thread-1]
> DEBUG(MultiThreadedHttpConnectionManager.java:839) - Getting free connection,
> hostConfig=HostConfiguration[host=http://myserver.com:8983]
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodDirector.java:379) -
> Attempt number 1 to process request
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1079) - enter
> HttpMethodBase.execute(HttpState, HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:2057) - enter
> HttpMethodBase.writeRequest(HttpState, HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:2212) - enter
> HttpMethodBase.writeRequestLine(HttpState, HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1496) - enter
> HttpMethodBase.generateRequestLine(HttpConnection, String, String, String,
> String)
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - >> "POST
> /solr/replication HTTP/1.1[\r][\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:1032) - enter
> HttpConnection.print(String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:942) - enter
> HttpConnection.write(byte[])
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:963) - enter
> HttpConnection.write(byte[], int, int)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:2175) - enter
> HttpMethodBase.writeRequestHeaders(HttpState,HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:370) -
> enter EntityEnclosingMethod.addRequestHeaders(HttpState, HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(ExpectContinueMethod.java:183) -
> enter ExpectContinueMethod.addRequestHeaders(HttpState, HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1417) - enter
> HttpMethodBase.addRequestHeaders(HttpState, HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1442) - enter
> HttpMethodBase.addUserAgentRequestHeaders(HttpState, HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1328) - enter
> HttpMethodBase.addHostRequestHeader(HttpState, HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(HttpMethodBase.java:1352) - Adding
> Host request header
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1266) - enter
> HttpMethodBase.addCookieRequestHeader(HttpState, HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpState.java:173) - enter
> HttpState.getCookies()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(CookieSpecBase.java:549) - enter
> CookieSpecBase.match(String, int, String, boolean, Cookie[])
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1379) - enter
> HttpMethodBase.addProxyConnectionHeader(HttpState, HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:401) -
> enter EntityEnclosingMethod.addContentLengthRequestHeader(HttpState,
> HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:325) -
> enter EntityEnclosingMethod.getRequestContentLength()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:135) - enter
> PostMethod.hasRequestContent()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:240) - enter
> PostMethod.getParameters()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:2324) - enter
> getContentCharSet( Header contentheader )
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(HttpMethodBase.java:2342) -
> Default charset used: ISO-8859-1
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:386) -
> HttpMethodBase.addRequestHeader(Header)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:240) - enter
> PostMethod.getParameters()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:2324) - enter
> getContentCharSet( Header contentheader )
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HeaderElement.java:238) - enter
> HeaderElement.parseElements(String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HeaderElement.java:194) - enter
> HeaderElement.parseElements(char[])
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HeaderElement.java:279) - enter
> HeaderElement.getParameterByName(String)
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(HttpMethodBase.java:2342) -
> Default charset used: ISO-8859-1
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - >> "User-Agent:
> Jakarta Commons-HttpClient/3.1[\r][\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:1032) - enter
> HttpConnection.print(String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:942) - enter
> HttpConnection.write(byte[])
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:963) - enter
> HttpConnection.write(byte[], int, int)
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - >> "Host:
> myserver.com:8983[\r][\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:1032) - enter
> HttpConnection.print(String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:942) - enter
> HttpConnection.write(byte[])
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:963) - enter
> HttpConnection.write(byte[], int, int)
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - >>
> "Content-Length: 89[\r][\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:1032) - enter
> HttpConnection.print(String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:942) - enter
> HttpConnection.write(byte[])
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:963) - enter
> HttpConnection.write(byte[], int, int)
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - >> "Content-Type:
> application/x-www-form-urlencoded[\r][\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:1032) - enter
> HttpConnection.print(String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:942) - enter
> HttpConnection.write(byte[])
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:963) - enter
> HttpConnection.write(byte[], int, int)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:1001) - enter
> HttpConnection.writeLine()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:942) - enter
> HttpConnection.write(byte[])
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:963) - enter
> HttpConnection.write(byte[], int, int)
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - >> "[\r][\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:469) -
> enter EntityEnclosingMethod.writeRequestBody(HttpState, HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:135) - enter
> PostMethod.hasRequestContent()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:325) -
> enter EntityEnclosingMethod.getRequestContentLength()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:135) - enter
> PostMethod.hasRequestContent()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:840) - enter
> HttpConnection.getRequestOutputStream()
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:84) - >>
> "command=filecontent&indexversion=1266003907838&file=_7h0y.fdx&checksum=true&wt=filestream"
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(EntityEnclosingMethod.java:508) -
> Request body sent
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:826) - enter
> HttpConnection.flushRequestOutputStream()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1730) - enter
> HttpMethodBase.readResponse(HttpState, HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1964) - enter
> HttpMethodBase.readStatusLine(HttpState, HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:1113) - enter
> HttpConnection.readLine()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:105) - enter
> HttpParser.readLine(InputStream, String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:74) - enter
> HttpParser.readRawLine()
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "HTTP/1.1 200
> OK[\r][\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "HTTP/1.1 200
> OK[\r][\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1930) - enter
> HttpMethodBase.readResponseHeaders(HttpState,HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:857) - enter
> HttpConnection.getResponseInputStream()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:170) - enter
> HeaderParser.parseHeaders(InputStream, String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:105) - enter
> HttpParser.readLine(InputStream, String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:74) - enter
> HttpParser.readRawLine()
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "Date: Tue, 02
> Mar 2010 21:24:40 GMT[\r][\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:105) - enter
> HttpParser.readLine(InputStream, String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:74) - enter
> HttpParser.readRawLine()
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "Content-Type:
> application/octet-stream[\r][\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:105) - enter
> HttpParser.readLine(InputStream, String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:74) - enter
> HttpParser.readRawLine()
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - <<
> "Transfer-Encoding: chunked[\r][\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:105) - enter
> HttpParser.readLine(InputStream, String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:74) - enter
> HttpParser.readRawLine()
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "Server:
> Jetty(6.1.19)[\r][\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:105) - enter
> HttpParser.readLine(InputStream, String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:74) - enter
> HttpParser.readRawLine()
>
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "[\r][\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1579) - enter
> HttpMethodBase.processResponseHeaders(HttpState, HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1615) - enter
> HttpMethodBase.processCookieHeaders(Header[], HttpState, HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1781) - enter
> HttpMethodBase.readResponseBody(HttpState, HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1816) - enter
> HttpMethodBase.readResponseBody(HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:857) - enter
> HttpConnection.getResponseInputStream()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:2281) - enter
> HttpMethodBase.canResponseHaveBody(int)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HeaderElement.java:238) - enter
> HeaderElement.parseElements(String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HeaderElement.java:194) - enter
> HeaderElement.parseElements(char[])
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:894) - enter
> HttpConnection.isResponseAvailable(int)
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:84) - << "4"
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:84) - << "[\r]"
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "[\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:84) - <<
> "[0x0][0x0][0x0][0x0]"
> [2010-03-02 21:24:40] [pool-3-thread-1] WARN (SnapPuller.java:950) - No content
> recieved for file: {size=5324, name=_7h0y.fdx, lastmodified=1267565039000}
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:84) - << "[\r]"
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "[\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:84) - << "0"
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:84) - << "[\r]"
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "[\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:170) - enter
> HeaderParser.parseHeaders(InputStream, String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:105) - enter
> HttpParser.readLine(InputStream, String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:74) - enter
> HttpParser.readRawLine()
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:84) - << "[\r]"
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "[\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "[\r][\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(HttpMethodBase.java:1024) -
> Resorting to protocol version default close connection policy
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(HttpMethodBase.java:1028) - Should
> NOT close connection, using HTTP/1.1
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:874) - enter
> HttpConnection.isResponseAvailable()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:1174) - enter
> HttpConnection.releaseConnection()
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(HttpConnection.java:1178) -
> Releasing connection back to connection manager.
> [2010-03-02 21:24:40] [pool-3-thread-1]
> TRACE(MultiThreadedHttpConnectionManager.java:630) - enter
> HttpConnectionManager.releaseConnection(HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1]
> DEBUG(MultiThreadedHttpConnectionManager.java:979) - Freeing connection,
> hostConfig=HostConfiguration[host=http://myserver.com:8983]
> [2010-03-02 21:24:40] [pool-3-thread-1]
> TRACE(MultiThreadedHttpConnectionManager.java:805) - enter
> HttpConnectionManager.ConnectionPool.getHostPool(HostConfiguration)
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(IdleConnectionHandler.java:76) -
> Adding connection at: 1267565080137
> [2010-03-02 21:24:40] [pool-3-thread-1]
> DEBUG(MultiThreadedHttpConnectionManager.java:961) - Notifying no-one, there are
> no waiting threads
> [2010-03-02 21:24:40] [pool-3-thread-1] ERROR(ReplicationHandler.java:266) -
> SnapPull failed
> org.apache.solr.common.SolrException: Unable to download _7h0y.fdx completely.
> Downloaded 0!=5324
> at
> org.apache.solr.handler.SnapPuller$FileFetcher.cleanup(SnapPuller.java:1041)
> at
> org.apache.solr.handler.SnapPuller$FileFetcher.fetchFile(SnapPuller.java:921)
> at
> org.apache.solr.handler.SnapPuller.downloadIndexFiles(SnapPuller.java:545)
> at
> org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:294)
> at
> org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:264)
> at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:159)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:280)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:135)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:65)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:146)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:170)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
> at java.lang.Thread.run(Thread.java:595)
Re: replication issue
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Hi Matthieu,
Does this happen over and over?
Is this with Solr 1.4 or some other version?
Is there anything unusual about _7h0y.fdx?
Does _7h0y.fdx still exist on the master when the replication fails?
...
Otis
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
Hadoop ecosystem search :: http://search-hadoop.com/
----- Original Message ----
> From: Matthieu Labour <ma...@yahoo.com>
> To: solr-user@lucene.apache.org
> Sent: Tue, March 2, 2010 4:35:46 PM
> Subject: Re: replication issue
>
> The replication does not work for me
>
>
> I have a big master solr and I want to start replicating it. I can see that the
> slave is downloading data from the master... I see a directory
> index.20100302093000 gets created in data/ next to index... I can see its size
> growing but then the directory gets deleted
>
> Here is the complete trace (I added a couple of LOG messages and compile solr)
>
> [2010-03-02 21:24:00] [pool-3-thread-1]
> DEBUG(MultiThreadedHttpConnectionManager.java:961) - Notifying no-one, there are
> no waiting threads
> [2010-03-02 21:24:00] [pool-3-thread-1] INFO (SnapPuller.java:278) - Number of
> files in latest index in master: 163
> [2010-03-02 21:24:00] [pool-3-thread-1] DEBUG(SnapPuller.java:536) -
> downloadIndexFiles(downloadCompleteIndex=false,tmpIdxDir=../solr/data/index.20100302092400,latestVersion=1266003907838)
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(SnapPuller.java:541) -
> -->localIndexFile=/opt/solr_env/solr/data/index/_7h0y.fdx
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(SnapPuller.java:900) - fetchFile()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:265) - enter
> PostMethod.addParameter(String, String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:150) -
> enter EntityEnclosingMethod.clearRequestBody()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:265) - enter
> PostMethod.addParameter(String, String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:150) -
> enter EntityEnclosingMethod.clearRequestBody()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:265) - enter
> PostMethod.addParameter(String, String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:150) -
> enter EntityEnclosingMethod.clearRequestBody()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:265) - enter
> PostMethod.addParameter(String, String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:150) -
> enter EntityEnclosingMethod.clearRequestBody()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:265) - enter
> PostMethod.addParameter(String, String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:150) -
> enter EntityEnclosingMethod.clearRequestBody()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpClient.java:321) - enter
> HttpClient.executeMethod(HttpMethod)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpClient.java:374) - enter
> HttpClient.executeMethod(HostConfiguration,HttpMethod,HttpState)
> [2010-03-02 21:24:40] [pool-3-thread-1]
> TRACE(MultiThreadedHttpConnectionManager.java:405) - enter
> HttpConnectionManager.getConnectionWithTimeout(HostConfiguration, long)
> [2010-03-02 21:24:40] [pool-3-thread-1]
> DEBUG(MultiThreadedHttpConnectionManager.java:412) -
> HttpConnectionManager.getConnection: config =
> HostConfiguration[host=http://myserver.com:8983], timeout = 0
> [2010-03-02 21:24:40] [pool-3-thread-1]
> TRACE(MultiThreadedHttpConnectionManager.java:805) - enter
> HttpConnectionManager.ConnectionPool.getHostPool(HostConfiguration)
> [2010-03-02 21:24:40] [pool-3-thread-1]
> TRACE(MultiThreadedHttpConnectionManager.java:805) - enter
> HttpConnectionManager.ConnectionPool.getHostPool(HostConfiguration)
> [2010-03-02 21:24:40] [pool-3-thread-1]
> DEBUG(MultiThreadedHttpConnectionManager.java:839) - Getting free connection,
> hostConfig=HostConfiguration[host=http://myserver.com:8983]
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodDirector.java:379) -
> Attempt number 1 to process request
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1079) - enter
> HttpMethodBase.execute(HttpState, HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:2057) - enter
> HttpMethodBase.writeRequest(HttpState, HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:2212) - enter
> HttpMethodBase.writeRequestLine(HttpState, HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1496) - enter
> HttpMethodBase.generateRequestLine(HttpConnection, String, String, String,
> String)
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - >> "POST
> /solr/replication HTTP/1.1[\r][\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:1032) - enter
> HttpConnection.print(String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:942) - enter
> HttpConnection.write(byte[])
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:963) - enter
> HttpConnection.write(byte[], int, int)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:2175) - enter
> HttpMethodBase.writeRequestHeaders(HttpState,HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:370) -
> enter EntityEnclosingMethod.addRequestHeaders(HttpState, HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(ExpectContinueMethod.java:183) -
> enter ExpectContinueMethod.addRequestHeaders(HttpState, HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1417) - enter
> HttpMethodBase.addRequestHeaders(HttpState, HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1442) - enter
> HttpMethodBase.addUserAgentRequestHeaders(HttpState, HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1328) - enter
> HttpMethodBase.addHostRequestHeader(HttpState, HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(HttpMethodBase.java:1352) - Adding
> Host request header
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1266) - enter
> HttpMethodBase.addCookieRequestHeader(HttpState, HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpState.java:173) - enter
> HttpState.getCookies()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(CookieSpecBase.java:549) - enter
> CookieSpecBase.match(String, int, String, boolean, Cookie[])
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1379) - enter
> HttpMethodBase.addProxyConnectionHeader(HttpState, HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:401) -
> enter EntityEnclosingMethod.addContentLengthRequestHeader(HttpState,
> HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:325) -
> enter EntityEnclosingMethod.getRequestContentLength()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:135) - enter
> PostMethod.hasRequestContent()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:240) - enter
> PostMethod.getParameters()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:2324) - enter
> getContentCharSet( Header contentheader )
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(HttpMethodBase.java:2342) -
> Default charset used: ISO-8859-1
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:386) -
> HttpMethodBase.addRequestHeader(Header)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:240) - enter
> PostMethod.getParameters()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:2324) - enter
> getContentCharSet( Header contentheader )
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HeaderElement.java:238) - enter
> HeaderElement.parseElements(String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HeaderElement.java:194) - enter
> HeaderElement.parseElements(char[])
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HeaderElement.java:279) - enter
> HeaderElement.getParameterByName(String)
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(HttpMethodBase.java:2342) -
> Default charset used: ISO-8859-1
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - >> "User-Agent:
> Jakarta Commons-HttpClient/3.1[\r][\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:1032) - enter
> HttpConnection.print(String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:942) - enter
> HttpConnection.write(byte[])
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:963) - enter
> HttpConnection.write(byte[], int, int)
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - >> "Host:
> myserver.com:8983[\r][\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:1032) - enter
> HttpConnection.print(String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:942) - enter
> HttpConnection.write(byte[])
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:963) - enter
> HttpConnection.write(byte[], int, int)
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - >>
> "Content-Length: 89[\r][\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:1032) - enter
> HttpConnection.print(String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:942) - enter
> HttpConnection.write(byte[])
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:963) - enter
> HttpConnection.write(byte[], int, int)
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - >> "Content-Type:
> application/x-www-form-urlencoded[\r][\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:1032) - enter
> HttpConnection.print(String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:942) - enter
> HttpConnection.write(byte[])
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:963) - enter
> HttpConnection.write(byte[], int, int)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:1001) - enter
> HttpConnection.writeLine()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:942) - enter
> HttpConnection.write(byte[])
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:963) - enter
> HttpConnection.write(byte[], int, int)
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - >> "[\r][\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:469) -
> enter EntityEnclosingMethod.writeRequestBody(HttpState, HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:135) - enter
> PostMethod.hasRequestContent()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:325) -
> enter EntityEnclosingMethod.getRequestContentLength()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:135) - enter
> PostMethod.hasRequestContent()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:840) - enter
> HttpConnection.getRequestOutputStream()
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:84) - >>
> "command=filecontent&indexversion=1266003907838&file=_7h0y.fdx&checksum=true&wt=filestream"
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(EntityEnclosingMethod.java:508) -
> Request body sent
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:826) - enter
> HttpConnection.flushRequestOutputStream()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1730) - enter
> HttpMethodBase.readResponse(HttpState, HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1964) - enter
> HttpMethodBase.readStatusLine(HttpState, HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:1113) - enter
> HttpConnection.readLine()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:105) - enter
> HttpParser.readLine(InputStream, String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:74) - enter
> HttpParser.readRawLine()
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "HTTP/1.1 200
> OK[\r][\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "HTTP/1.1 200
> OK[\r][\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1930) - enter
> HttpMethodBase.readResponseHeaders(HttpState,HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:857) - enter
> HttpConnection.getResponseInputStream()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:170) - enter
> HeaderParser.parseHeaders(InputStream, String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:105) - enter
> HttpParser.readLine(InputStream, String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:74) - enter
> HttpParser.readRawLine()
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "Date: Tue, 02
> Mar 2010 21:24:40 GMT[\r][\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:105) - enter
> HttpParser.readLine(InputStream, String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:74) - enter
> HttpParser.readRawLine()
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "Content-Type:
> application/octet-stream[\r][\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:105) - enter
> HttpParser.readLine(InputStream, String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:74) - enter
> HttpParser.readRawLine()
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - <<
> "Transfer-Encoding: chunked[\r][\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:105) - enter
> HttpParser.readLine(InputStream, String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:74) - enter
> HttpParser.readRawLine()
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "Server:
> Jetty(6.1.19)[\r][\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:105) - enter
> HttpParser.readLine(InputStream, String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:74) - enter
> HttpParser.readRawLine()
>
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "[\r][\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1579) - enter
> HttpMethodBase.processResponseHeaders(HttpState, HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1615) - enter
> HttpMethodBase.processCookieHeaders(Header[], HttpState, HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1781) - enter
> HttpMethodBase.readResponseBody(HttpState, HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1816) - enter
> HttpMethodBase.readResponseBody(HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:857) - enter
> HttpConnection.getResponseInputStream()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:2281) - enter
> HttpMethodBase.canResponseHaveBody(int)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HeaderElement.java:238) - enter
> HeaderElement.parseElements(String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HeaderElement.java:194) - enter
> HeaderElement.parseElements(char[])
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:894) - enter
> HttpConnection.isResponseAvailable(int)
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:84) - << "4"
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:84) - << "[\r]"
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "[\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:84) - <<
> "[0x0][0x0][0x0][0x0]"
> [2010-03-02 21:24:40] [pool-3-thread-1] WARN (SnapPuller.java:950) - No content
> recieved for file: {size=5324, name=_7h0y.fdx, lastmodified=1267565039000}
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:84) - << "[\r]"
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "[\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:84) - << "0"
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:84) - << "[\r]"
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "[\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:170) - enter
> HeaderParser.parseHeaders(InputStream, String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:105) - enter
> HttpParser.readLine(InputStream, String)
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:74) - enter
> HttpParser.readRawLine()
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:84) - << "[\r]"
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "[\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "[\r][\n]"
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(HttpMethodBase.java:1024) -
> Resorting to protocol version default close connection policy
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(HttpMethodBase.java:1028) - Should
> NOT close connection, using HTTP/1.1
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:874) - enter
> HttpConnection.isResponseAvailable()
> [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:1174) - enter
> HttpConnection.releaseConnection()
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(HttpConnection.java:1178) -
> Releasing connection back to connection manager.
> [2010-03-02 21:24:40] [pool-3-thread-1]
> TRACE(MultiThreadedHttpConnectionManager.java:630) - enter
> HttpConnectionManager.releaseConnection(HttpConnection)
> [2010-03-02 21:24:40] [pool-3-thread-1]
> DEBUG(MultiThreadedHttpConnectionManager.java:979) - Freeing connection,
> hostConfig=HostConfiguration[host=http://myserver.com:8983]
> [2010-03-02 21:24:40] [pool-3-thread-1]
> TRACE(MultiThreadedHttpConnectionManager.java:805) - enter
> HttpConnectionManager.ConnectionPool.getHostPool(HostConfiguration)
> [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(IdleConnectionHandler.java:76) -
> Adding connection at: 1267565080137
> [2010-03-02 21:24:40] [pool-3-thread-1]
> DEBUG(MultiThreadedHttpConnectionManager.java:961) - Notifying no-one, there are
> no waiting threads
> [2010-03-02 21:24:40] [pool-3-thread-1] ERROR(ReplicationHandler.java:266) -
> SnapPull failed
> org.apache.solr.common.SolrException: Unable to download _7h0y.fdx completely.
> Downloaded 0!=5324
> at
> org.apache.solr.handler.SnapPuller$FileFetcher.cleanup(SnapPuller.java:1041)
> at
> org.apache.solr.handler.SnapPuller$FileFetcher.fetchFile(SnapPuller.java:921)
> at
> org.apache.solr.handler.SnapPuller.downloadIndexFiles(SnapPuller.java:545)
> at
> org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:294)
> at
> org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:264)
> at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:159)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:280)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:135)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:65)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:146)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:170)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
> at java.lang.Thread.run(Thread.java:595)
Re: replication issue
Posted by Matthieu Labour <ma...@yahoo.com>.
One More information
I deleted the index on the master and I restarted the master and restarted the slave and now the replication works
Would it be possible that the replication doesn work well when started against an already existing big index ?
Thank you
--- On Tue, 3/2/10, Matthieu Labour <ma...@yahoo.com> wrote:
From: Matthieu Labour <ma...@yahoo.com>
Subject: Re: replication issue
To: solr-user@lucene.apache.org
Date: Tuesday, March 2, 2010, 3:35 PM
The replication does not work for me
I have a big master solr and I want to start replicating it. I can see that the slave is downloading data from the master... I see a directory index.20100302093000 gets created in data/ next to index... I can see its size growing but then the directory gets deleted
Here is the complete trace (I added a couple of LOG messages and compile solr)
[2010-03-02 21:24:00] [pool-3-thread-1] DEBUG(MultiThreadedHttpConnectionManager.java:961) - Notifying no-one, there are no waiting threads
[2010-03-02 21:24:00] [pool-3-thread-1] INFO (SnapPuller.java:278) - Number of files in latest index in master: 163
[2010-03-02 21:24:00] [pool-3-thread-1] DEBUG(SnapPuller.java:536) - downloadIndexFiles(downloadCompleteIndex=false,tmpIdxDir=../solr/data/index.20100302092400,latestVersion=1266003907838)
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(SnapPuller.java:541) - -->localIndexFile=/opt/solr_env/solr/data/index/_7h0y.fdx
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(SnapPuller.java:900) - fetchFile()
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:265) - enter PostMethod.addParameter(String, String)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:150) - enter EntityEnclosingMethod.clearRequestBody()
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:265) - enter PostMethod.addParameter(String, String)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:150) - enter EntityEnclosingMethod.clearRequestBody()
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:265) - enter PostMethod.addParameter(String, String)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:150) - enter EntityEnclosingMethod.clearRequestBody()
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:265) - enter PostMethod.addParameter(String, String)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:150) - enter EntityEnclosingMethod.clearRequestBody()
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:265) - enter PostMethod.addParameter(String, String)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:150) - enter EntityEnclosingMethod.clearRequestBody()
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpClient.java:321) - enter HttpClient.executeMethod(HttpMethod)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpClient.java:374) - enter HttpClient.executeMethod(HostConfiguration,HttpMethod,HttpState)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(MultiThreadedHttpConnectionManager.java:405) - enter HttpConnectionManager.getConnectionWithTimeout(HostConfiguration, long)
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(MultiThreadedHttpConnectionManager.java:412) - HttpConnectionManager.getConnection: config = HostConfiguration[host=http://myserver.com:8983], timeout = 0
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(MultiThreadedHttpConnectionManager.java:805) - enter HttpConnectionManager.ConnectionPool.getHostPool(HostConfiguration)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(MultiThreadedHttpConnectionManager.java:805) - enter HttpConnectionManager.ConnectionPool.getHostPool(HostConfiguration)
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(MultiThreadedHttpConnectionManager.java:839) - Getting free connection, hostConfig=HostConfiguration[host=http://myserver.com:8983]
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodDirector.java:379) - Attempt number 1 to process request
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1079) - enter HttpMethodBase.execute(HttpState, HttpConnection)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:2057) - enter HttpMethodBase.writeRequest(HttpState, HttpConnection)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:2212) - enter HttpMethodBase.writeRequestLine(HttpState, HttpConnection)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1496) - enter HttpMethodBase.generateRequestLine(HttpConnection, String, String, String, String)
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - >> "POST /solr/replication HTTP/1.1[\r][\n]"
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:1032) - enter HttpConnection.print(String)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:942) - enter HttpConnection.write(byte[])
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:963) - enter HttpConnection.write(byte[], int, int)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:2175) - enter HttpMethodBase.writeRequestHeaders(HttpState,HttpConnection)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:370) - enter EntityEnclosingMethod.addRequestHeaders(HttpState, HttpConnection)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(ExpectContinueMethod.java:183) - enter ExpectContinueMethod.addRequestHeaders(HttpState, HttpConnection)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1417) - enter HttpMethodBase.addRequestHeaders(HttpState, HttpConnection)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1442) - enter HttpMethodBase.addUserAgentRequestHeaders(HttpState, HttpConnection)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1328) - enter HttpMethodBase.addHostRequestHeader(HttpState, HttpConnection)
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(HttpMethodBase.java:1352) - Adding Host request header
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1266) - enter HttpMethodBase.addCookieRequestHeader(HttpState, HttpConnection)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpState.java:173) - enter HttpState.getCookies()
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(CookieSpecBase.java:549) - enter CookieSpecBase.match(String, int, String, boolean, Cookie[])
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1379) - enter HttpMethodBase.addProxyConnectionHeader(HttpState, HttpConnection)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:401) - enter EntityEnclosingMethod.addContentLengthRequestHeader(HttpState, HttpConnection)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:325) - enter EntityEnclosingMethod.getRequestContentLength()
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:135) - enter PostMethod.hasRequestContent()
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:240) - enter PostMethod.getParameters()
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:2324) - enter getContentCharSet( Header contentheader )
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(HttpMethodBase.java:2342) - Default charset used: ISO-8859-1
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:386) - HttpMethodBase.addRequestHeader(Header)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:240) - enter PostMethod.getParameters()
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:2324) - enter getContentCharSet( Header contentheader )
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HeaderElement.java:238) - enter HeaderElement.parseElements(String)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HeaderElement.java:194) - enter HeaderElement.parseElements(char[])
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HeaderElement.java:279) - enter HeaderElement.getParameterByName(String)
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(HttpMethodBase.java:2342) - Default charset used: ISO-8859-1
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - >> "User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n]"
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:1032) - enter HttpConnection.print(String)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:942) - enter HttpConnection.write(byte[])
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:963) - enter HttpConnection.write(byte[], int, int)
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - >> "Host: myserver.com:8983[\r][\n]"
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:1032) - enter HttpConnection.print(String)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:942) - enter HttpConnection.write(byte[])
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:963) - enter HttpConnection.write(byte[], int, int)
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - >> "Content-Length: 89[\r][\n]"
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:1032) - enter HttpConnection.print(String)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:942) - enter HttpConnection.write(byte[])
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:963) - enter HttpConnection.write(byte[], int, int)
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - >> "Content-Type: application/x-www-form-urlencoded[\r][\n]"
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:1032) - enter HttpConnection.print(String)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:942) - enter HttpConnection.write(byte[])
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:963) - enter HttpConnection.write(byte[], int, int)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:1001) - enter HttpConnection.writeLine()
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:942) - enter HttpConnection.write(byte[])
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:963) - enter HttpConnection.write(byte[], int, int)
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - >> "[\r][\n]"
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:469) - enter EntityEnclosingMethod.writeRequestBody(HttpState, HttpConnection)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:135) - enter PostMethod.hasRequestContent()
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:325) - enter EntityEnclosingMethod.getRequestContentLength()
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:135) - enter PostMethod.hasRequestContent()
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:840) - enter HttpConnection.getRequestOutputStream()
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:84) - >> "command=filecontent&indexversion=1266003907838&file=_7h0y.fdx&checksum=true&wt=filestream"
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(EntityEnclosingMethod.java:508) - Request body sent
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:826) - enter HttpConnection.flushRequestOutputStream()
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1730) - enter HttpMethodBase.readResponse(HttpState, HttpConnection)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1964) - enter HttpMethodBase.readStatusLine(HttpState, HttpConnection)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:1113) - enter HttpConnection.readLine()
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:105) - enter HttpParser.readLine(InputStream, String)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:74) - enter HttpParser.readRawLine()
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "HTTP/1.1 200 OK[\r][\n]"
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "HTTP/1.1 200 OK[\r][\n]"
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1930) - enter HttpMethodBase.readResponseHeaders(HttpState,HttpConnection)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:857) - enter HttpConnection.getResponseInputStream()
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:170) - enter HeaderParser.parseHeaders(InputStream, String)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:105) - enter HttpParser.readLine(InputStream, String)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:74) - enter HttpParser.readRawLine()
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "Date: Tue, 02 Mar 2010 21:24:40 GMT[\r][\n]"
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:105) - enter HttpParser.readLine(InputStream, String)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:74) - enter HttpParser.readRawLine()
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "Content-Type: application/octet-stream[\r][\n]"
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:105) - enter HttpParser.readLine(InputStream, String)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:74) - enter HttpParser.readRawLine()
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "Transfer-Encoding: chunked[\r][\n]"
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:105) - enter HttpParser.readLine(InputStream, String)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:74) - enter HttpParser.readRawLine()
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "Server: Jetty(6.1.19)[\r][\n]"
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:105) - enter HttpParser.readLine(InputStream, String)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:74) - enter HttpParser.readRawLine()
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "[\r][\n]"
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1579) - enter HttpMethodBase.processResponseHeaders(HttpState, HttpConnection)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1615) - enter HttpMethodBase.processCookieHeaders(Header[], HttpState, HttpConnection)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1781) - enter HttpMethodBase.readResponseBody(HttpState, HttpConnection)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1816) - enter HttpMethodBase.readResponseBody(HttpConnection)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:857) - enter HttpConnection.getResponseInputStream()
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:2281) - enter HttpMethodBase.canResponseHaveBody(int)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HeaderElement.java:238) - enter HeaderElement.parseElements(String)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HeaderElement.java:194) - enter HeaderElement.parseElements(char[])
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:894) - enter HttpConnection.isResponseAvailable(int)
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:84) - << "4"
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:84) - << "[\r]"
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "[\n]"
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:84) - << "[0x0][0x0][0x0][0x0]"
[2010-03-02 21:24:40] [pool-3-thread-1] WARN (SnapPuller.java:950) - No content recieved for file: {size=5324, name=_7h0y.fdx, lastmodified=1267565039000}
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:84) - << "[\r]"
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "[\n]"
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:84) - << "0"
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:84) - << "[\r]"
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "[\n]"
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:170) - enter HeaderParser.parseHeaders(InputStream, String)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:105) - enter HttpParser.readLine(InputStream, String)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:74) - enter HttpParser.readRawLine()
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:84) - << "[\r]"
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "[\n]"
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "[\r][\n]"
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(HttpMethodBase.java:1024) - Resorting to protocol version default close connection policy
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(HttpMethodBase.java:1028) - Should NOT close connection, using HTTP/1.1
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:874) - enter HttpConnection.isResponseAvailable()
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:1174) - enter HttpConnection.releaseConnection()
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(HttpConnection.java:1178) - Releasing connection back to connection manager.
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(MultiThreadedHttpConnectionManager.java:630) - enter HttpConnectionManager.releaseConnection(HttpConnection)
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(MultiThreadedHttpConnectionManager.java:979) - Freeing connection, hostConfig=HostConfiguration[host=http://myserver.com:8983]
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(MultiThreadedHttpConnectionManager.java:805) - enter HttpConnectionManager.ConnectionPool.getHostPool(HostConfiguration)
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(IdleConnectionHandler.java:76) - Adding connection at: 1267565080137
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(MultiThreadedHttpConnectionManager.java:961) - Notifying no-one, there are no waiting threads
[2010-03-02 21:24:40] [pool-3-thread-1] ERROR(ReplicationHandler.java:266) - SnapPull failed
org.apache.solr.common.SolrException: Unable to download _7h0y.fdx completely. Downloaded 0!=5324
at org.apache.solr.handler.SnapPuller$FileFetcher.cleanup(SnapPuller.java:1041)
at org.apache.solr.handler.SnapPuller$FileFetcher.fetchFile(SnapPuller.java:921)
at org.apache.solr.handler.SnapPuller.downloadIndexFiles(SnapPuller.java:545)
at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:294)
at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:264)
at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:159)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:280)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:135)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:65)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:146)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:170)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)
Re: replication issue
Posted by Matthieu Labour <ma...@yahoo.com>.
The replication does not work for me
I have a big master solr and I want to start replicating it. I can see that the slave is downloading data from the master... I see a directory index.20100302093000 gets created in data/ next to index... I can see its size growing but then the directory gets deleted
Here is the complete trace (I added a couple of LOG messages and compile solr)
[2010-03-02 21:24:00] [pool-3-thread-1] DEBUG(MultiThreadedHttpConnectionManager.java:961) - Notifying no-one, there are no waiting threads
[2010-03-02 21:24:00] [pool-3-thread-1] INFO (SnapPuller.java:278) - Number of files in latest index in master: 163
[2010-03-02 21:24:00] [pool-3-thread-1] DEBUG(SnapPuller.java:536) - downloadIndexFiles(downloadCompleteIndex=false,tmpIdxDir=../solr/data/index.20100302092400,latestVersion=1266003907838)
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(SnapPuller.java:541) - -->localIndexFile=/opt/solr_env/solr/data/index/_7h0y.fdx
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(SnapPuller.java:900) - fetchFile()
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:265) - enter PostMethod.addParameter(String, String)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:150) - enter EntityEnclosingMethod.clearRequestBody()
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:265) - enter PostMethod.addParameter(String, String)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:150) - enter EntityEnclosingMethod.clearRequestBody()
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:265) - enter PostMethod.addParameter(String, String)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:150) - enter EntityEnclosingMethod.clearRequestBody()
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:265) - enter PostMethod.addParameter(String, String)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:150) - enter EntityEnclosingMethod.clearRequestBody()
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:265) - enter PostMethod.addParameter(String, String)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:150) - enter EntityEnclosingMethod.clearRequestBody()
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpClient.java:321) - enter HttpClient.executeMethod(HttpMethod)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpClient.java:374) - enter HttpClient.executeMethod(HostConfiguration,HttpMethod,HttpState)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(MultiThreadedHttpConnectionManager.java:405) - enter HttpConnectionManager.getConnectionWithTimeout(HostConfiguration, long)
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(MultiThreadedHttpConnectionManager.java:412) - HttpConnectionManager.getConnection: config = HostConfiguration[host=http://myserver.com:8983], timeout = 0
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(MultiThreadedHttpConnectionManager.java:805) - enter HttpConnectionManager.ConnectionPool.getHostPool(HostConfiguration)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(MultiThreadedHttpConnectionManager.java:805) - enter HttpConnectionManager.ConnectionPool.getHostPool(HostConfiguration)
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(MultiThreadedHttpConnectionManager.java:839) - Getting free connection, hostConfig=HostConfiguration[host=http://myserver.com:8983]
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodDirector.java:379) - Attempt number 1 to process request
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1079) - enter HttpMethodBase.execute(HttpState, HttpConnection)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:2057) - enter HttpMethodBase.writeRequest(HttpState, HttpConnection)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:2212) - enter HttpMethodBase.writeRequestLine(HttpState, HttpConnection)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1496) - enter HttpMethodBase.generateRequestLine(HttpConnection, String, String, String, String)
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - >> "POST /solr/replication HTTP/1.1[\r][\n]"
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:1032) - enter HttpConnection.print(String)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:942) - enter HttpConnection.write(byte[])
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:963) - enter HttpConnection.write(byte[], int, int)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:2175) - enter HttpMethodBase.writeRequestHeaders(HttpState,HttpConnection)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:370) - enter EntityEnclosingMethod.addRequestHeaders(HttpState, HttpConnection)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(ExpectContinueMethod.java:183) - enter ExpectContinueMethod.addRequestHeaders(HttpState, HttpConnection)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1417) - enter HttpMethodBase.addRequestHeaders(HttpState, HttpConnection)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1442) - enter HttpMethodBase.addUserAgentRequestHeaders(HttpState, HttpConnection)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1328) - enter HttpMethodBase.addHostRequestHeader(HttpState, HttpConnection)
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(HttpMethodBase.java:1352) - Adding Host request header
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1266) - enter HttpMethodBase.addCookieRequestHeader(HttpState, HttpConnection)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpState.java:173) - enter HttpState.getCookies()
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(CookieSpecBase.java:549) - enter CookieSpecBase.match(String, int, String, boolean, Cookie[])
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1379) - enter HttpMethodBase.addProxyConnectionHeader(HttpState, HttpConnection)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:401) - enter EntityEnclosingMethod.addContentLengthRequestHeader(HttpState, HttpConnection)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:325) - enter EntityEnclosingMethod.getRequestContentLength()
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:135) - enter PostMethod.hasRequestContent()
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:240) - enter PostMethod.getParameters()
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:2324) - enter getContentCharSet( Header contentheader )
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(HttpMethodBase.java:2342) - Default charset used: ISO-8859-1
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:386) - HttpMethodBase.addRequestHeader(Header)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:240) - enter PostMethod.getParameters()
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:2324) - enter getContentCharSet( Header contentheader )
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HeaderElement.java:238) - enter HeaderElement.parseElements(String)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HeaderElement.java:194) - enter HeaderElement.parseElements(char[])
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HeaderElement.java:279) - enter HeaderElement.getParameterByName(String)
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(HttpMethodBase.java:2342) - Default charset used: ISO-8859-1
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - >> "User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n]"
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:1032) - enter HttpConnection.print(String)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:942) - enter HttpConnection.write(byte[])
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:963) - enter HttpConnection.write(byte[], int, int)
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - >> "Host: myserver.com:8983[\r][\n]"
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:1032) - enter HttpConnection.print(String)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:942) - enter HttpConnection.write(byte[])
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:963) - enter HttpConnection.write(byte[], int, int)
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - >> "Content-Length: 89[\r][\n]"
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:1032) - enter HttpConnection.print(String)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:942) - enter HttpConnection.write(byte[])
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:963) - enter HttpConnection.write(byte[], int, int)
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - >> "Content-Type: application/x-www-form-urlencoded[\r][\n]"
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:1032) - enter HttpConnection.print(String)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:942) - enter HttpConnection.write(byte[])
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:963) - enter HttpConnection.write(byte[], int, int)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:1001) - enter HttpConnection.writeLine()
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:942) - enter HttpConnection.write(byte[])
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:963) - enter HttpConnection.write(byte[], int, int)
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - >> "[\r][\n]"
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:469) - enter EntityEnclosingMethod.writeRequestBody(HttpState, HttpConnection)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:135) - enter PostMethod.hasRequestContent()
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:325) - enter EntityEnclosingMethod.getRequestContentLength()
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:135) - enter PostMethod.hasRequestContent()
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:840) - enter HttpConnection.getRequestOutputStream()
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:84) - >> "command=filecontent&indexversion=1266003907838&file=_7h0y.fdx&checksum=true&wt=filestream"
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(EntityEnclosingMethod.java:508) - Request body sent
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:826) - enter HttpConnection.flushRequestOutputStream()
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1730) - enter HttpMethodBase.readResponse(HttpState, HttpConnection)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1964) - enter HttpMethodBase.readStatusLine(HttpState, HttpConnection)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:1113) - enter HttpConnection.readLine()
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:105) - enter HttpParser.readLine(InputStream, String)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:74) - enter HttpParser.readRawLine()
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "HTTP/1.1 200 OK[\r][\n]"
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "HTTP/1.1 200 OK[\r][\n]"
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1930) - enter HttpMethodBase.readResponseHeaders(HttpState,HttpConnection)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:857) - enter HttpConnection.getResponseInputStream()
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:170) - enter HeaderParser.parseHeaders(InputStream, String)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:105) - enter HttpParser.readLine(InputStream, String)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:74) - enter HttpParser.readRawLine()
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "Date: Tue, 02 Mar 2010 21:24:40 GMT[\r][\n]"
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:105) - enter HttpParser.readLine(InputStream, String)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:74) - enter HttpParser.readRawLine()
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "Content-Type: application/octet-stream[\r][\n]"
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:105) - enter HttpParser.readLine(InputStream, String)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:74) - enter HttpParser.readRawLine()
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "Transfer-Encoding: chunked[\r][\n]"
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:105) - enter HttpParser.readLine(InputStream, String)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:74) - enter HttpParser.readRawLine()
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "Server: Jetty(6.1.19)[\r][\n]"
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:105) - enter HttpParser.readLine(InputStream, String)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:74) - enter HttpParser.readRawLine()
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "[\r][\n]"
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1579) - enter HttpMethodBase.processResponseHeaders(HttpState, HttpConnection)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1615) - enter HttpMethodBase.processCookieHeaders(Header[], HttpState, HttpConnection)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1781) - enter HttpMethodBase.readResponseBody(HttpState, HttpConnection)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1816) - enter HttpMethodBase.readResponseBody(HttpConnection)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:857) - enter HttpConnection.getResponseInputStream()
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:2281) - enter HttpMethodBase.canResponseHaveBody(int)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HeaderElement.java:238) - enter HeaderElement.parseElements(String)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HeaderElement.java:194) - enter HeaderElement.parseElements(char[])
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:894) - enter HttpConnection.isResponseAvailable(int)
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:84) - << "4"
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:84) - << "[\r]"
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "[\n]"
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:84) - << "[0x0][0x0][0x0][0x0]"
[2010-03-02 21:24:40] [pool-3-thread-1] WARN (SnapPuller.java:950) - No content recieved for file: {size=5324, name=_7h0y.fdx, lastmodified=1267565039000}
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:84) - << "[\r]"
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "[\n]"
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:84) - << "0"
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:84) - << "[\r]"
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "[\n]"
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:170) - enter HeaderParser.parseHeaders(InputStream, String)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:105) - enter HttpParser.readLine(InputStream, String)
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:74) - enter HttpParser.readRawLine()
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:84) - << "[\r]"
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "[\n]"
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "[\r][\n]"
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(HttpMethodBase.java:1024) - Resorting to protocol version default close connection policy
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(HttpMethodBase.java:1028) - Should NOT close connection, using HTTP/1.1
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:874) - enter HttpConnection.isResponseAvailable()
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:1174) - enter HttpConnection.releaseConnection()
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(HttpConnection.java:1178) - Releasing connection back to connection manager.
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(MultiThreadedHttpConnectionManager.java:630) - enter HttpConnectionManager.releaseConnection(HttpConnection)
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(MultiThreadedHttpConnectionManager.java:979) - Freeing connection, hostConfig=HostConfiguration[host=http://myserver.com:8983]
[2010-03-02 21:24:40] [pool-3-thread-1] TRACE(MultiThreadedHttpConnectionManager.java:805) - enter HttpConnectionManager.ConnectionPool.getHostPool(HostConfiguration)
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(IdleConnectionHandler.java:76) - Adding connection at: 1267565080137
[2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(MultiThreadedHttpConnectionManager.java:961) - Notifying no-one, there are no waiting threads
[2010-03-02 21:24:40] [pool-3-thread-1] ERROR(ReplicationHandler.java:266) - SnapPull failed
org.apache.solr.common.SolrException: Unable to download _7h0y.fdx completely. Downloaded 0!=5324
at org.apache.solr.handler.SnapPuller$FileFetcher.cleanup(SnapPuller.java:1041)
at org.apache.solr.handler.SnapPuller$FileFetcher.fetchFile(SnapPuller.java:921)
at org.apache.solr.handler.SnapPuller.downloadIndexFiles(SnapPuller.java:545)
at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:294)
at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:264)
at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:159)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:280)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:135)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:65)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:146)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:170)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)
Re: replication issue
Posted by Matthieu Labour <ma...@yahoo.com>.
I think this issue is tot related to patch SOLR-1736
Here is the error I get ... Thank you for any help
[2010-03-02 19:07:26] [pool-3-thread-1] ERROR(ReplicationHandler.java:266) - SnapPull failed
org.apache.solr.common.SolrException: Unable to download _7bre.fdt completely. Downloaded 0!=15591
at org.apache.solr.handler.SnapPuller$FileFetcher.cleanup(SnapPuller.java:1036)
at org.apache.solr.handler.SnapPuller$FileFetcher.fetchFile(SnapPuller.java:916)
at org.apache.solr.handler.SnapPuller.downloadIndexFiles(SnapPuller.java:541)
at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:294)
at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:264)
at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:159)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:280)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:135)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:65)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:146)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:170)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)
--- On Tue, 3/2/10, Matthieu Labour <ma...@yahoo.com> wrote:
From: Matthieu Labour <ma...@yahoo.com>
Subject: Re: replication issue
To: solr-user@lucene.apache.org
Date: Tuesday, March 2, 2010, 11:23 AM
Hi Paul
Thank you for your amswer
I did put all the directory structure on /raid ... /raid/solr_env/solr ... , /raid/solr_env/jetty ...
And it still didn't work even after I applied patch SOLR-1736
I am investigating if this is because tempDir and data dir are not on the same partition
matt
--- On Mon, 3/1/10, Noble Paul നോബിള് नोब्ळ् <no...@gmail.com> wrote:
From: Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>
Subject: Re: replication issue
To: solr-user@lucene.apache.org
Date: Monday, March 1, 2010, 10:30 PM
The data/index.20100226063400 dir is a temporary dir and isc reated in
the same dir where the index dir is located.
I'm wondering if the symlink is causing the problem. Why don't you set
the data dir as /raid/data instead of /solr/data
On Sat, Feb 27, 2010 at 12:13 AM, Matthieu Labour
<ma...@yahoo.com> wrote:
> Hi
>
> I am still having issues with the replication and wonder if things are working properly
>
> So I have 1 master and 1 slave
>
> On the slave, I deleted the data/index directory and data/replication.properties file and restarted solr.
>
> When slave is pulling data from master, I can see that the size of data directory is growing
>
> root@slr8:/raid/data# du -sh
> 3.7M .
> root@slr8:/raid/data# du -sh
> 4.7M .
>
> and I can see that data/replication.properties file got created and also a directory data/index.20100226063400
>
> soon after index.20100226063400 disapears and the size of data/index is back to 12K
>
> root@slr8:/raid/data/index# du -sh
> 12K .
>
> And when I look for the number of documents via the admin interface, I still see 0 documents so I feel something is wrong
>
> One more thing, I have a symlink for /solr/data ---> /raid/data
>
> Thank you for your help !
>
> matt
>
>
>
>
>
>
>
--
-----------------------------------------------------
Noble Paul | Systems Architect| AOL | http://aol.com
Re: replication issue
Posted by Matthieu Labour <ma...@yahoo.com>.
Hi Paul
Thank you for your amswer
I did put all the directory structure on /raid ... /raid/solr_env/solr ... , /raid/solr_env/jetty ...
And it still didn't work even after I applied patch SOLR-1736
I am investigating if this is because tempDir and data dir are not on the same partition
matt
--- On Mon, 3/1/10, Noble Paul നോബിള് नोब्ळ् <no...@gmail.com> wrote:
From: Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>
Subject: Re: replication issue
To: solr-user@lucene.apache.org
Date: Monday, March 1, 2010, 10:30 PM
The data/index.20100226063400 dir is a temporary dir and isc reated in
the same dir where the index dir is located.
I'm wondering if the symlink is causing the problem. Why don't you set
the data dir as /raid/data instead of /solr/data
On Sat, Feb 27, 2010 at 12:13 AM, Matthieu Labour
<ma...@yahoo.com> wrote:
> Hi
>
> I am still having issues with the replication and wonder if things are working properly
>
> So I have 1 master and 1 slave
>
> On the slave, I deleted the data/index directory and data/replication.properties file and restarted solr.
>
> When slave is pulling data from master, I can see that the size of data directory is growing
>
> root@slr8:/raid/data# du -sh
> 3.7M .
> root@slr8:/raid/data# du -sh
> 4.7M .
>
> and I can see that data/replication.properties file got created and also a directory data/index.20100226063400
>
> soon after index.20100226063400 disapears and the size of data/index is back to 12K
>
> root@slr8:/raid/data/index# du -sh
> 12K .
>
> And when I look for the number of documents via the admin interface, I still see 0 documents so I feel something is wrong
>
> One more thing, I have a symlink for /solr/data ---> /raid/data
>
> Thank you for your help !
>
> matt
>
>
>
>
>
>
>
--
-----------------------------------------------------
Noble Paul | Systems Architect| AOL | http://aol.com
Re: replication issue
Posted by Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>.
The data/index.20100226063400 dir is a temporary dir and isc reated in
the same dir where the index dir is located.
I'm wondering if the symlink is causing the problem. Why don't you set
the data dir as /raid/data instead of /solr/data
On Sat, Feb 27, 2010 at 12:13 AM, Matthieu Labour
<ma...@yahoo.com> wrote:
> Hi
>
> I am still having issues with the replication and wonder if things are working properly
>
> So I have 1 master and 1 slave
>
> On the slave, I deleted the data/index directory and data/replication.properties file and restarted solr.
>
> When slave is pulling data from master, I can see that the size of data directory is growing
>
> root@slr8:/raid/data# du -sh
> 3.7M .
> root@slr8:/raid/data# du -sh
> 4.7M .
>
> and I can see that data/replication.properties file got created and also a directory data/index.20100226063400
>
> soon after index.20100226063400 disapears and the size of data/index is back to 12K
>
> root@slr8:/raid/data/index# du -sh
> 12K .
>
> And when I look for the number of documents via the admin interface, I still see 0 documents so I feel something is wrong
>
> One more thing, I have a symlink for /solr/data ---> /raid/data
>
> Thank you for your help !
>
> matt
>
>
>
>
>
>
>
--
-----------------------------------------------------
Noble Paul | Systems Architect| AOL | http://aol.com