You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@lucene.apache.org by Bruno Roustant <br...@gmail.com> on 2020/04/17 15:49:40 UTC
Re: High concurrency scenarios IndexReader‘s registerParentReader method takes a lot of CPU time
Hi,
Next time please reply/send these technical questions to
java-user@lucene.apache.org or dev@lucene.apache.org to get more audience.
538 segments seems huge. What is your configuration to get this number of
segments?
Le jeu. 16 avr. 2020 à 16:28, 张海雷 <zh...@bytedance.com> a écrit :
>
>
>
> Hi, all
> In our Elastic Search benchmark, found out that IndexReader‘s
> registerParentReader method takes a lot of CPU time
> Environment:
> ES Version: 7.5.1
> Lucene Version: 8.3.0
>
> Index:
> Shard Num 1
> Replicator’s Num:1
> Segment Count: 538
> Size:49.5GB
>
> Benchmark result: qps:5200,pct99 600ms
> CPU Util:93%
>
> Use flame graph find the hot method IndexReader‘s registerParentReader.
> dive into code find out that Have a race condition that on the
> synchronizedSet Because of only one
> Shard and one shard have too many segments, this intensify competition
>
>
>
> To verify this assumption,execute force merge on this index，after force
> merge:
> QPS: more then 5600
> PCT99: 60ms
> CPU Util: 11%
> The performance improved by a factor of 10x
>
> So we can use ConcurrentWeakKeyHashmap instead of synchronizedSet to
> reduce lock competition?
>
>
>
>
>
>
>
Re: [Phishing Risk] [External] Re: High concurrency scenarios IndexReader‘s registerParentReader method takes a lot of CPU time
Posted by 张海雷 <zh...@bytedance.com>.
Using elastic search index stats apihttps://www.elastic.co/guide/en/elasticsearch/reference/current/indices-stats.html#indices-stats <https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-stats.html#indices-stats>
> 2020年4月17日 下午11:49，Bruno Roustant <br...@gmail.com> 写道：
>
> Hi,
>
> Next time please reply/send these technical questions to java-user@lucene.apache.org <ma...@lucene.apache.org> or dev@lucene.apache.org <ma...@lucene.apache.org> to get more audience.
>
> 538 segments seems huge. What is your configuration to get this number of segments?
>
> Le jeu. 16 avr. 2020 à 16:28, 张海雷 <zhanghailei@bytedance.com <ma...@bytedance.com>> a écrit :
>
>
>>
>> Hi, all
>> In our Elastic Search benchmark, found out that IndexReader‘s registerParentReader method takes a lot of CPU time
>> Environment:
>> ES Version: 7.5.1
>> Lucene Version: 8.3.0
>>
>> Index:
>> Shard Num 1
>> Replicator’s Num:1
>> Segment Count: 538
>> Size:49.5GB
>>
>> Benchmark result: qps:5200,pct99 600ms
>> CPU Util:93%
>>
>> Use flame graph find the hot method IndexReader‘s registerParentReader. dive into code find out that Have a race condition that on the synchronizedSet Because of only one
>> Shard and one shard have too many segments, this intensify competition
>>
>>
>>
>>
>> To verify this assumption,execute force merge on this index，after force merge:
>> QPS: more then 5600
>> PCT99: 60ms
>> CPU Util: 11%
>> The performance improved by a factor of 10x
>>
>> So we can use ConcurrentWeakKeyHashmap instead of synchronizedSet to reduce lock competition?
>>
>>
>>
>>
>>
>
Re: [Phishing Risk] [External] Re: High concurrency scenarios IndexReader‘s registerParentReader method takes a lot of CPU time
Posted by 张海雷 <zh...@bytedance.com>.
Using elastic search index stats apihttps://www.elastic.co/guide/en/elasticsearch/reference/current/indices-stats.html#indices-stats <https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-stats.html#indices-stats>
> 2020年4月17日 下午11:49，Bruno Roustant <br...@gmail.com> 写道：
>
> Hi,
>
> Next time please reply/send these technical questions to java-user@lucene.apache.org <ma...@lucene.apache.org> or dev@lucene.apache.org <ma...@lucene.apache.org> to get more audience.
>
> 538 segments seems huge. What is your configuration to get this number of segments?
>
> Le jeu. 16 avr. 2020 à 16:28, 张海雷 <zhanghailei@bytedance.com <ma...@bytedance.com>> a écrit :
>
>
>>
>> Hi, all
>> In our Elastic Search benchmark, found out that IndexReader‘s registerParentReader method takes a lot of CPU time
>> Environment:
>> ES Version: 7.5.1
>> Lucene Version: 8.3.0
>>
>> Index:
>> Shard Num 1
>> Replicator’s Num:1
>> Segment Count: 538
>> Size:49.5GB
>>
>> Benchmark result: qps:5200,pct99 600ms
>> CPU Util:93%
>>
>> Use flame graph find the hot method IndexReader‘s registerParentReader. dive into code find out that Have a race condition that on the synchronizedSet Because of only one
>> Shard and one shard have too many segments, this intensify competition
>>
>>
>>
>>
>> To verify this assumption,execute force merge on this index，after force merge:
>> QPS: more then 5600
>> PCT99: 60ms
>> CPU Util: 11%
>> The performance improved by a factor of 10x
>>
>> So we can use ConcurrentWeakKeyHashmap instead of synchronizedSet to reduce lock competition?
>>
>>
>>
>>
>>
>