You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Lance Norskog <go...@gmail.com> on 2010/11/01 03:12:38 UTC
Re: problem of solr replcation's speed
If you are copying from an indexer while you are indexing new content,
this would cause contention for the disk head. Does indexing slow down
during this period?
Lance
2010/10/31 Peter Karich <pe...@yahoo.de>:
> we have an identical-sized index and it takes ~5minutes
>
>
>> It takes about one hour to replacate 6G index for solr in my env. But my
>> network can transfer file about 10-20M/s using scp. So solr's http
>> replcation is too slow, it's normal or I do something wrong?
>>
>
>
--
Lance Norskog
goksron@gmail.com
Re: Re:Re: Re:Re: problem of solr replcation's speed
Posted by astubbs <an...@gmail.com>.
For future reference, I had this problem, and it was the debug statements in
commons HTTP that were printing out all the binary data to the log, but my
console appender was set to INFO so I wasn't seeing them. Setting http
commons to INFO level fixed my speed issue (two orders of magnitude faster).
--
View this message in context: http://lucene.472066.n3.nabble.com/problem-of-solr-replcation-s-speed-tp1815764p3664835.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re:Re: Re:Re: problem of solr replcation's speed
Posted by kafka0102 <ka...@163.com>.
sometorment later....
I found the reason ofsolr replication'slow speed. It's not solr's problem.It's jetty's. I used to embed jetty7 in my app. But when I found solr's demo use jetty6 , I tried to use jetty6 in my app and I was so happy to get the fast speed.
actually, I tried to change solr's demo in jetty7 by default's conf, the replication's speed was slow too.
I don't know why the default jetty7 server is so slow. I wanna to find the reason.Maybe I can ask the jetty maillist or continue to read the codes.
At 2010-11-02 07:28:54，"Lance Norskog" <go...@gmail.com> wrote:
>This is the time to replicate and open the new index, right? Opening a
>new index can take a lot of time. How many autowarmers and queries are
>there in the caches? Opening a new index re-runs all of the queries in
>all of the caches.
>
>2010/11/1 kafka0102 <ka...@163.com>:
>> I suspected my app has some sleeping op every 1s, so
>> I changed ReplicationHandler.PACKET_SZ to 1024 * 1024*10; // 10MB
>>
>> and log result is like thus :
>> [2010-11-01 17:49:29][INFO][pool-6-thread-1][SnapPuller.java(1038)]readFully10485760 cost 3184
>> [2010-11-01 17:49:32][INFO][pool-6-thread-1][SnapPuller.java(1038)]readFully10485760 cost 3426
>> [2010-11-01 17:49:36][INFO][pool-6-thread-1][SnapPuller.java(1038)]readFully10485760 cost 3359
>> [2010-11-01 17:49:39][INFO][pool-6-thread-1][SnapPuller.java(1038)]readFully10485760 cost 3166
>> [2010-11-01 17:49:42][INFO][pool-6-thread-1][SnapPuller.java(1038)]readFully10485760 cost 3513
>> [2010-11-01 17:49:46][INFO][pool-6-thread-1][SnapPuller.java(1038)]readFully10485760 cost 3140
>> [2010-11-01 17:49:50][INFO][pool-6-thread-1][SnapPuller.java(1038)]readFully10485760 cost 3471
>>
>> That means It's still slow like before. what's wrong with my env????
>>
>> At 2010-11-01 17:30:32，kafka0102 <ka...@163.com> wrote:
>> I hacked SnapPuller to log the cost, and the log is like thus:
>> [2010-11-01 17:21:19][INFO][pool-6-thread-1][SnapPuller.java(1037)]readFully1048576 cost 979
>> [2010-11-01 17:21:19][INFO][pool-6-thread-1][SnapPuller.java(1037)]readFully1048576 cost 4
>> [2010-11-01 17:21:19][INFO][pool-6-thread-1][SnapPuller.java(1037)]readFully1048576 cost 4
>> [2010-11-01 17:21:20][INFO][pool-6-thread-1][SnapPuller.java(1037)]readFully1048576 cost 980
>> [2010-11-01 17:21:20][INFO][pool-6-thread-1][SnapPuller.java(1037)]readFully1048576 cost 4
>> [2010-11-01 17:21:20][INFO][pool-6-thread-1][SnapPuller.java(1037)]readFully1048576 cost 5
>> [2010-11-01 17:21:21][INFO][pool-6-thread-1][SnapPuller.java(1037)]readFully1048576 cost 979
>>
>>
>> It's saying it cost about 1000ms for transfering 1M data every 2 times. I used jetty as server and embeded solr in my app.I'm so confused.What I have done wrong?
>>
>>
>> At 2010-11-01 10:12:38，"Lance Norskog" <go...@gmail.com> wrote:
>>
>>>If you are copying from an indexer while you are indexing new content,
>>>this would cause contention for the disk head. Does indexing slow down
>>>during this period?
>>>
>>>Lance
>>>
>>>2010/10/31 Peter Karich <pe...@yahoo.de>:
>>>> we have an identical-sized index and it takes ~5minutes
>>>>
>>>>
>>>>> It takes about one hour to replacate 6G index for solr in my env. But my
>>>>> network can transfer file about 10-20M/s using scp. So solr's http
>>>>> replcation is too slow, it's normal or I do something wrong?
>>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>--
>>>Lance Norskog
>>>goksron@gmail.com
>>
>>
>>
>
>
>
>--
>Lance Norskog
>goksron@gmail.com
Re: Re:Re: problem of solr replcation's speed
Posted by Lance Norskog <go...@gmail.com>.
This is the time to replicate and open the new index, right? Opening a
new index can take a lot of time. How many autowarmers and queries are
there in the caches? Opening a new index re-runs all of the queries in
all of the caches.
2010/11/1 kafka0102 <ka...@163.com>:
> I suspected my app has some sleeping op every 1s, so
> I changed ReplicationHandler.PACKET_SZ to 1024 * 1024*10; // 10MB
>
> and log result is like thus :
> [2010-11-01 17:49:29][INFO][pool-6-thread-1][SnapPuller.java(1038)]readFully10485760 cost 3184
> [2010-11-01 17:49:32][INFO][pool-6-thread-1][SnapPuller.java(1038)]readFully10485760 cost 3426
> [2010-11-01 17:49:36][INFO][pool-6-thread-1][SnapPuller.java(1038)]readFully10485760 cost 3359
> [2010-11-01 17:49:39][INFO][pool-6-thread-1][SnapPuller.java(1038)]readFully10485760 cost 3166
> [2010-11-01 17:49:42][INFO][pool-6-thread-1][SnapPuller.java(1038)]readFully10485760 cost 3513
> [2010-11-01 17:49:46][INFO][pool-6-thread-1][SnapPuller.java(1038)]readFully10485760 cost 3140
> [2010-11-01 17:49:50][INFO][pool-6-thread-1][SnapPuller.java(1038)]readFully10485760 cost 3471
>
> That means It's still slow like before. what's wrong with my env????
>
> At 2010-11-01 17:30:32，kafka0102 <ka...@163.com> wrote:
> I hacked SnapPuller to log the cost, and the log is like thus:
> [2010-11-01 17:21:19][INFO][pool-6-thread-1][SnapPuller.java(1037)]readFully1048576 cost 979
> [2010-11-01 17:21:19][INFO][pool-6-thread-1][SnapPuller.java(1037)]readFully1048576 cost 4
> [2010-11-01 17:21:19][INFO][pool-6-thread-1][SnapPuller.java(1037)]readFully1048576 cost 4
> [2010-11-01 17:21:20][INFO][pool-6-thread-1][SnapPuller.java(1037)]readFully1048576 cost 980
> [2010-11-01 17:21:20][INFO][pool-6-thread-1][SnapPuller.java(1037)]readFully1048576 cost 4
> [2010-11-01 17:21:20][INFO][pool-6-thread-1][SnapPuller.java(1037)]readFully1048576 cost 5
> [2010-11-01 17:21:21][INFO][pool-6-thread-1][SnapPuller.java(1037)]readFully1048576 cost 979
>
>
> It's saying it cost about 1000ms for transfering 1M data every 2 times. I used jetty as server and embeded solr in my app.I'm so confused.What I have done wrong?
>
>
> At 2010-11-01 10:12:38，"Lance Norskog" <go...@gmail.com> wrote:
>
>>If you are copying from an indexer while you are indexing new content,
>>this would cause contention for the disk head. Does indexing slow down
>>during this period?
>>
>>Lance
>>
>>2010/10/31 Peter Karich <pe...@yahoo.de>:
>>> we have an identical-sized index and it takes ~5minutes
>>>
>>>
>>>> It takes about one hour to replacate 6G index for solr in my env. But my
>>>> network can transfer file about 10-20M/s using scp. So solr's http
>>>> replcation is too slow, it's normal or I do something wrong?
>>>>
>>>
>>>
>>
>>
>>
>>--
>>Lance Norskog
>>goksron@gmail.com
>
>
>
--
Lance Norskog
goksron@gmail.com
Re:Re:Re: problem of solr replcation's speed
Posted by kafka0102 <ka...@163.com>.
I suspected my app has some sleeping op every 1s, so
I changed ReplicationHandler.PACKET_SZ to 1024 * 1024*10; // 10MB
and log result is like thus :
[2010-11-01 17:49:29][INFO][pool-6-thread-1][SnapPuller.java(1038)]readFully10485760 cost 3184
[2010-11-01 17:49:32][INFO][pool-6-thread-1][SnapPuller.java(1038)]readFully10485760 cost 3426
[2010-11-01 17:49:36][INFO][pool-6-thread-1][SnapPuller.java(1038)]readFully10485760 cost 3359
[2010-11-01 17:49:39][INFO][pool-6-thread-1][SnapPuller.java(1038)]readFully10485760 cost 3166
[2010-11-01 17:49:42][INFO][pool-6-thread-1][SnapPuller.java(1038)]readFully10485760 cost 3513
[2010-11-01 17:49:46][INFO][pool-6-thread-1][SnapPuller.java(1038)]readFully10485760 cost 3140
[2010-11-01 17:49:50][INFO][pool-6-thread-1][SnapPuller.java(1038)]readFully10485760 cost 3471
That means It's still slow like before. what's wrong with my env????
At 2010-11-01 17:30:32，kafka0102 <ka...@163.com> wrote:
I hacked SnapPuller to log the cost, and the log is like thus:
[2010-11-01 17:21:19][INFO][pool-6-thread-1][SnapPuller.java(1037)]readFully1048576 cost 979
[2010-11-01 17:21:19][INFO][pool-6-thread-1][SnapPuller.java(1037)]readFully1048576 cost 4
[2010-11-01 17:21:19][INFO][pool-6-thread-1][SnapPuller.java(1037)]readFully1048576 cost 4
[2010-11-01 17:21:20][INFO][pool-6-thread-1][SnapPuller.java(1037)]readFully1048576 cost 980
[2010-11-01 17:21:20][INFO][pool-6-thread-1][SnapPuller.java(1037)]readFully1048576 cost 4
[2010-11-01 17:21:20][INFO][pool-6-thread-1][SnapPuller.java(1037)]readFully1048576 cost 5
[2010-11-01 17:21:21][INFO][pool-6-thread-1][SnapPuller.java(1037)]readFully1048576 cost 979
It's saying it cost about 1000ms for transfering 1M data every 2 times. I used jetty as server and embeded solr in my app.I'm so confused.What I have done wrong?
At 2010-11-01 10:12:38，"Lance Norskog" <go...@gmail.com> wrote:
>If you are copying from an indexer while you are indexing new content,
>this would cause contention for the disk head. Does indexing slow down
>during this period?
>
>Lance
>
>2010/10/31 Peter Karich <pe...@yahoo.de>:
>> we have an identical-sized index and it takes ~5minutes
>>
>>
>>> It takes about one hour to replacate 6G index for solr in my env. But my
>>> network can transfer file about 10-20M/s using scp. So solr's http
>>> replcation is too slow, it's normal or I do something wrong?
>>>
>>
>>
>
>
>
>--
>Lance Norskog
>goksron@gmail.com
Re:Re: problem of solr replcation's speed
Posted by kafka0102 <ka...@163.com>.
I hacked SnapPuller to log the cost, and the log is like thus:
[2010-11-01 17:21:19][INFO][pool-6-thread-1][SnapPuller.java(1037)]readFully1048576 cost 979
[2010-11-01 17:21:19][INFO][pool-6-thread-1][SnapPuller.java(1037)]readFully1048576 cost 4
[2010-11-01 17:21:19][INFO][pool-6-thread-1][SnapPuller.java(1037)]readFully1048576 cost 4
[2010-11-01 17:21:20][INFO][pool-6-thread-1][SnapPuller.java(1037)]readFully1048576 cost 980
[2010-11-01 17:21:20][INFO][pool-6-thread-1][SnapPuller.java(1037)]readFully1048576 cost 4
[2010-11-01 17:21:20][INFO][pool-6-thread-1][SnapPuller.java(1037)]readFully1048576 cost 5
[2010-11-01 17:21:21][INFO][pool-6-thread-1][SnapPuller.java(1037)]readFully1048576 cost 979
It's saying it cost about 1000ms for transfering 1M data every 2 times. I used jetty as server and embeded solr in my app.I'm so confused.What I have done wrong?
At 2010-11-01 10:12:38，"Lance Norskog" <go...@gmail.com> wrote:
>If you are copying from an indexer while you are indexing new content,
>this would cause contention for the disk head. Does indexing slow down
>during this period?
>
>Lance
>
>2010/10/31 Peter Karich <pe...@yahoo.de>:
>> we have an identical-sized index and it takes ~5minutes
>>
>>
>>> It takes about one hour to replacate 6G index for solr in my env. But my
>>> network can transfer file about 10-20M/s using scp. So solr's http
>>> replcation is too slow, it's normal or I do something wrong?
>>>
>>
>>
>
>
>
>--
>Lance Norskog
>goksron@gmail.com