[jira] [Created] (SOLR-9096) Add PartitionStream to Streaming Expressions

["Dennis Gove (JIRA)" <ji...@apache.org>]

Dennis Gove created SOLR-9096:
---------------------------------

             Summary: Add PartitionStream to Streaming Expressions
                 Key: SOLR-9096
                 URL: https://issues.apache.org/jira/browse/SOLR-9096
             Project: Solr
          Issue Type: New Feature
            Reporter: Dennis Gove


The basic idea of a PartitionStream is to take a one or more input streams of tuples, partition them out to a set of workers such that each worker can work with a subset of the tuples, and then bring them all back into a single stream. This differs from a ParallelStream because in ParallelStream the data is partitioned at the source whereas with a PartitionStream one can take an existing stream and spread it out across workers.

{code}
                            /--- sort ----\
                           /               \                      /--- Collection A
                          / ---- sort ----- \                    /
Client <--- rollup  <----<                    <----- innerJoin <
                          \ ---- sort ----- /                    \
                           \               /                      \--- Collection B
                            \--- sort ----/
{code}

{code}
                            /--- sort -- rollup ----\
                           /                         \                      /--- Collection A
                          / ---- sort -- rollup ----- \                    /
Client <-- innerJoin <---<                              <----- innerJoin <
                    \     \ ---- sort -- rollup ----- /                    \
                     \     \                         /                      \--- Collection B
                      \     \--- sort -- rollup ----/
                       \
                        \
                         \ <--- jdbc source
{code}

{code}
                            /--- sort -- innerJoin ----\
                           /                            \  
                          / ---- sort -- innerJoin ----- \  <--- jdbc source
Client <-- innerJoin <---<                                | 
                    \     \ ---- sort -- innerJoin ----- /  <--- rollup <---- Collection A
                     \     \                            /  
                      \     \--- sort -- innerJoin ----/
                       \
                        \
                         \ <--- jdbc source
{code}



I imagine partition expression would look something like this

{code}
partition(
  inputA=<source stream A>,
  inputB=<source stream B>,
  work=<stream for the workers>,
  over="fieldA,fieldB",
  workers=6,
  zkHost=<zk connection string>
)
{code}

for example

{code}
innerJoin(
  partition(
    inputA=jdbc(database1),
    inputB=rollup(
      search(collectionA, ...),
      ...
    ),
    work=sort(
      innerJoin(
        inputA,
        inputB,
        on="fieldA,fieldB"
      ),
      by="jdbcFieldC asc, collectionAFieldB desc"
    ),
    workers=6,
    zkHost=localhost:12345
  ),
  jdbc(database2),
  on="fieldZ"
)
{code}




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org