[jira] [Commented] (SOLR-15210) ParallelStream should execute hashing & filtering directly in ExportWriter

["Jan Høydahl (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SOLR-15210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17306291#comment-17306291 ] 

Jan Høydahl commented on SOLR-15210:
------------------------------------

Please consider deleting the remote branch https://github.com/apache/solr/tree/jira/solr-15210 and instead use a private fork for new PRs against new solr.git. See SOLR-15253 for the central branch cleanup effort - I'm planning to clean up remaining non-active branches in a week or so...

> ParallelStream should execute hashing & filtering directly in ExportWriter
> --------------------------------------------------------------------------
>
>                 Key: SOLR-15210
>                 URL: https://issues.apache.org/jira/browse/SOLR-15210
>             Project: Solr
>          Issue Type: Improvement
>      Security Level: Public(Default Security Level. Issues are Public) 
>            Reporter: Andrzej Bialecki
>            Assignee: Andrzej Bialecki
>            Priority: Major
>         Attachments: SOLR-15210.patch
>
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> Currently ParallelStream uses {{HashQParserPlugin}} to partition the work based on a hashed value of {{partitionKeys}}. Unfortunately, this filter has a high initial runtime cost because it has to materialize all values of {{partitionKeys}} on each worker in order to calculate their hash and decide whether a particular doc belongs to the worker's partition.
> The alternative approach would be for the worker to collect and sort all documents and only then filter out the ones that belong to the current partition just before they are written out by {{ExportWriter}} - at this point we have to materialize the fields anyway but also we can benefit from a (minimal) BytesRef caching that the FieldWriters use. On the other hand we pay the price of sorting all documents, and we also lose the query filter caching that the {{HashQParserPlugin}} uses.
> This tradeoff is not obvious but should be investigated to see if it offers better performance.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)