You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "zhangchao.es (Jira)" <ji...@apache.org> on 2021/07/14 08:55:00 UTC
[jira] [Updated] (LUCENE-10025)
SoftDeletesRetentionMergePolicy#numDeletesToMerge caused indexing
backlogged
[ https://issues.apache.org/jira/browse/LUCENE-10025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
zhangchao.es updated LUCENE-10025:
----------------------------------
Attachment: image-2021-07-14-16-52-34-740.png
flamegraph.html
Description:
In lucene-8246, numDeletesToMerge is added in SoftDeletesRetentionMergePolicy.
if soft deleted doc is very more, and they are also in retention lease,the numDeletesToMerge funcation have performance issue
for instance，a update indexing is writing to elasticsearch, then we move one a shard to an other node,If the moving continues for a long time, the size of old shard will become very big,because soft-deleted operations need to held by retention lease. The more soft-deleted documents, the slower the indexing. if the shard size is about 20GB, we can get the below flamegraph
!image-2021-07-14-16-52-34-740.png!
was:In lucene-8246, numDeletesToMerge is added in SoftDeletesRetentionMergePolicy.
> SoftDeletesRetentionMergePolicy#numDeletesToMerge caused indexing backlogged
> ----------------------------------------------------------------------------
>
> Key: LUCENE-10025
> URL: https://issues.apache.org/jira/browse/LUCENE-10025
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/index
> Affects Versions: 8.4
> Reporter: zhangchao.es
> Priority: Major
> Labels: indexing, soft-delete
> Attachments: flamegraph.html, image-2021-07-14-16-52-34-740.png
>
>
> In lucene-8246, numDeletesToMerge is added in SoftDeletesRetentionMergePolicy.
> if soft deleted doc is very more, and they are also in retention lease,the numDeletesToMerge funcation have performance issue
> for instance，a update indexing is writing to elasticsearch, then we move one a shard to an other node,If the moving continues for a long time, the size of old shard will become very big,because soft-deleted operations need to held by retention lease. The more soft-deleted documents, the slower the indexing. if the shard size is about 20GB, we can get the below flamegraph
>
> !image-2021-07-14-16-52-34-740.png!
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org