You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "Adrien Grand (Jira)" <ji...@apache.org> on 2020/09/29 15:22:00 UTC
[jira] [Resolved] (LUCENE-9535) Investigate recent indexing
slowdown for wikimedium documents
[ https://issues.apache.org/jira/browse/LUCENE-9535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Adrien Grand resolved LUCENE-9535.
----------------------------------
Fix Version/s: 8.7
Resolution: Fixed
This looks fixed now, so I'm closing this issue. Thanks Mike, Robert and Simon for helping dig this one.
> Investigate recent indexing slowdown for wikimedium documents
> -------------------------------------------------------------
>
> Key: LUCENE-9535
> URL: https://issues.apache.org/jira/browse/LUCENE-9535
> Project: Lucene - Core
> Issue Type: Task
> Reporter: Adrien Grand
> Priority: Minor
> Fix For: 8.7
>
> Attachments: cpu_profile.svg
>
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> Nightly benchmarks report a ~10% slowdown for 1kB documents as of September 9th: [http://people.apache.org/~mikemccand/lucenebench/indexing.html].
> On that day, we added stored fields in DWPT accounting (LUCENE-9511), so I first thought this could be due to smaller flushed segments and more merging, but I still wonder whether there's something else. The benchmark runs with 8GB of heap, 2GB of RAM buffer and 36 indexing threads. So it's about 2GB/36 = 57MB of RAM buffer per thread in the worst-case scenario that all DWPTs get full at the same time. Stored fields account for about 0.7MB of memory, or 1% of the indexing buffer size. How can a 1% reduction of buffering capacity explain a 10% indexing slowdown? I looked into this further by running indexing benchmarks locally with 8 indexing threads and 128MB of indexing buffer memory, which would make this issue even more apparent if the smaller RAM buffer was the cause, but I'm not seeing a regression and actually I'm seeing similar number of flushes when I disabled memory accounting for stored fields.
> I ran indexing under a profiler to see whether something else could cause this slowdown, e.g. slow implementations of ramBytesUsed on stored fields writers, but nothing surprising showed up and the profile looked just like I would have expected.
> Another question I have is why the 4kB benchmark is not affected at all.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org