You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by "Rahul Goswami (Jira)" <ji...@apache.org> on 2023/07/12 17:37:00 UTC
[jira] [Commented] (SOLR-16838) Atomic updates too slow in Solr 8 vs Solr 7
[ https://issues.apache.org/jira/browse/SOLR-16838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17742498#comment-17742498 ]
Rahul Goswami commented on SOLR-16838:
--------------------------------------
Missed the last couple of comments, sorry! [~janhoy] I backported the Lucene fix to 8.11.1 which is the version I have been testing on and found a dramatic improvement in performance. For a 20 million dataset, indexing in 15 parallel threads in batches of 1000, here are the before and after fix times:
Before fix: 370 mins
After fix: 65 mins
Note that this performance on an average is still tad slower than 7.7.2 across multiple runs, but I guess that can be attributed to the fact that the terms index is no longer loaded on-heap as of Lucene 8.6 (https://github.com/apache/lucene/issues/10297).
> Atomic updates too slow in Solr 8 vs Solr 7
> -------------------------------------------
>
> Key: SOLR-16838
> URL: https://issues.apache.org/jira/browse/SOLR-16838
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SearchComponents - other
> Affects Versions: 8.11.1
> Reporter: Rahul Goswami
> Priority: Major
> Labels: RTG, RealTimeGet, atomicupdate
>
> Started experiencing slowness with updates in production after upgrading from Solr 7.7.2 to 8.11.1. Upon comparing the performance it turns out that indexing 20 million docs via atomic updates through the same client program (running 15 parallel threads indexing in batches of 1000) takes below time:
>
> Solr 7 : 78 mins
> Solr 8: 370 mins
>
> Environment details:
> - Java 11 on Windows server
> - Xms1536m Xmx3072m
> - Indexing client code running 15 parallel threads indexing in batches of 1000
> - using SimpleFSDirectoryFactory (since Mmap doesn't quite work well on Windows for our index sizes which commonly run north of 1 TB)
>
> Looking at the thread dump, the bottleneck seems to be RealTimeGet and I can see that Solr 7 takes a different code path than Solr 8. Note that the performance of regular updates (non-atomic) is still pretty good on Solr 8 completing in < 1 hour for the same 20 million data set.
>
> Sharing the indexing code, solrconfig, schema and thread dumps in the link below:
> [https://drive.google.com/drive/folders/1q2DPNTYQEU6fi3NeXIKJhaoq3KPnms0h?usp=sharing]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org