[jira] [Created] (SOLR-14470) Add streaming expressions to /export handler

["Andrzej Bialecki (Jira)" <ji...@apache.org>]

Andrzej Bialecki created SOLR-14470:
---------------------------------------

             Summary: Add streaming expressions to /export handler
                 Key: SOLR-14470
                 URL: https://issues.apache.org/jira/browse/SOLR-14470
             Project: Solr
          Issue Type: Improvement
      Security Level: Public (Default Security Level. Issues are Public)
          Components: Export Writer, streaming expressions
            Reporter: Andrzej Bialecki
            Assignee: Andrzej Bialecki


Many streaming scenarios would greatly benefit from the ability to perform partial rollups (or other transformations) as early as possible, in order to minimize the amount of data that has to be sent from shards to the aggregating node.

This can be implemented as a subset of streaming expressions that process the data directly inside each local {{ExportHandler}} and outputs only the records from the resulting stream. 

Conceptually it would be similar to the way Hadoop {{Combiner}} works. As is the case with {{Combiner}}, because the input data is processed in batches there would be no guarantee that only 1 record per unique sort values would be emitted - in fact, in most cases multiple partial aggregations would be emitted. Still, in many scenarios this would allow reducing the amount of data to be sent by several orders of magnitude.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org