You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Rupa Shankar (JIRA)" <ji...@apache.org> on 2018/01/30 22:54:00 UTC
[jira] [Commented] (SOLR-11277) Add auto hard commit setting based
on tlog size
[ https://issues.apache.org/jira/browse/SOLR-11277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16345951#comment-16345951 ]
Rupa Shankar commented on SOLR-11277:
-------------------------------------
Thanks [~anshumg] for the feedback! Just uploaded a new patch incorporating your suggestions.
As for the benchmarking, I wrote a quick script that has 5 threads concurrently adding X documents to a collection. Here were the results:
*MASTER*
*5 threads, each adding 1000 docs*
Iteration #0: 5232 ms
Iteration #1: 4238 ms
Iteration #2: 3798 ms
Iteration #3: 3144 ms
Iteration #4: 2774 ms
Iteration #5: 2589 ms
Iteration #6: 2513 ms
Iteration #7: 2452 ms
Iteration #8: 2452 ms
Iteration #9: 2482 ms
AVG: 3167.4 ms
*5 threads, each adding 5000 docs*
Iteration #0: 12720 ms
Iteration #1: 12191 ms
Iteration #2: 12340 ms
Iteration #3: 12124 ms
Iteration #4: 12170 ms
Iteration #5: 12058 ms
Iteration #6: 11491 ms
Iteration #7: 12274 ms
Iteration #8: 12208 ms
Iteration #9: 12100 ms
AVG: 12167.6 ms
*HARD COMMIT FEATURE BRANCH*
*5 threads, each adding 1000 docs*
Iteration #0: 2758 ms
Iteration #1: 2576 ms
Iteration #2: 2440 ms
Iteration #3: 2416 ms
Iteration #4: 2443 ms
Iteration #5: 2406 ms
Iteration #6: 2419 ms
Iteration #7: 2430 ms
Iteration #8: 2421 ms
Iteration #9: 2448 ms
AVG: 2475.7 ms
*5 threads, each adding 5000 docs*
Iteration #0: 12683 ms
Iteration #1: 12520 ms
Iteration #2: 12264 ms
Iteration #3: 12223 ms
Iteration #4: 12148 ms
Iteration #5: 12161 ms
Iteration #6: 11810 ms
Iteration #7: 12398 ms
Iteration #8: 12168 ms
Iteration #9: 12522 ms
AVG: 12289.7 ms
Seems like the impact is minimal.
> Add auto hard commit setting based on tlog size
> -----------------------------------------------
>
> Key: SOLR-11277
> URL: https://issues.apache.org/jira/browse/SOLR-11277
> Project: Solr
> Issue Type: New Feature
> Security Level: Public(Default Security Level. Issues are Public)
> Reporter: Rupa Shankar
> Assignee: Anshum Gupta
> Priority: Major
> Attachments: max_size_auto_commit.patch, max_size_auto_commit.patch
>
>
> When indexing documents of variable sizes and at variable schedules, it can be hard to estimate the optimal auto hard commit maxDocs or maxTime settings. We’ve had some occurrences of really huge tlogs, resulting in serious issues, so in an attempt to avoid this, it would be great to have a “maxSize” setting based on the tlog size on disk.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org