You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Dennis Gove (JIRA)" <ji...@apache.org> on 2016/05/09 22:47:12 UTC
[jira] [Updated] (SOLR-9096) Add PartitionStream to Streaming
Expressions
[ https://issues.apache.org/jira/browse/SOLR-9096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dennis Gove updated SOLR-9096:
------------------------------
Description:
The basic idea of a PartitionStream is to take a one or more input streams of tuples, partition them out to a set of workers such that each worker can work with a subset of the tuples, and then bring them all back into a single stream. This differs from a ParallelStream because in ParallelStream the data is partitioned at the source whereas with a PartitionStream one can take an existing stream and spread it out across workers.
The use-case here is for when one has a source stream (or more) which cannot be parallelized at the source but which can be parallelized after some level of processing. I see this being used most for parallelized sort, rollups, or graph searches.
{code}
/--- sort ----\
/ \ /--- Collection A
/ ---- sort ----- \ /
Client <--- rollup <----< <----- innerJoin <
\ ---- sort ----- / \
\ / \--- Collection B
\--- sort ----/
{code}
{code}
/--- sort -- rollup ----\
/ \ /--- Collection A
/ ---- sort -- rollup ----- \ /
Client <-- innerJoin <---< <----- innerJoin <
\ \ ---- sort -- rollup ----- / \
\ \ / \--- Collection B
\ \--- sort -- rollup ----/
\
\
\ <--- jdbc source
{code}
{code}
/--- sort -- innerJoin ----\
/ \
/ ---- sort -- innerJoin ----- \ <--- jdbc source
Client <-- innerJoin <---< |
\ \ ---- sort -- innerJoin ----- / <--- rollup <---- Collection A
\ \ /
\ \--- sort -- innerJoin ----/
\
\
\ <--- jdbc source
{code}
I imagine partition expression would look something like this
{code}
partition(
inputA=<source stream A>,
inputB=<source stream B>,
work=<stream for the workers>,
over="fieldA,fieldB",
workers=6,
zkHost=<zk connection string>
)
{code}
for example
{code}
innerJoin(
partition(
inputA=jdbc(database1),
inputB=rollup(
search(collectionA, ...),
...
),
work=sort(
innerJoin(
inputA,
inputB,
on="fieldA,fieldB"
),
by="jdbcFieldC asc, collectionAFieldB desc"
),
workers=6,
zkHost=localhost:12345
),
jdbc(database2),
on="fieldZ"
)
{code}
was:
The basic idea of a PartitionStream is to take a one or more input streams of tuples, partition them out to a set of workers such that each worker can work with a subset of the tuples, and then bring them all back into a single stream. This differs from a ParallelStream because in ParallelStream the data is partitioned at the source whereas with a PartitionStream one can take an existing stream and spread it out across workers.
{code}
/--- sort ----\
/ \ /--- Collection A
/ ---- sort ----- \ /
Client <--- rollup <----< <----- innerJoin <
\ ---- sort ----- / \
\ / \--- Collection B
\--- sort ----/
{code}
{code}
/--- sort -- rollup ----\
/ \ /--- Collection A
/ ---- sort -- rollup ----- \ /
Client <-- innerJoin <---< <----- innerJoin <
\ \ ---- sort -- rollup ----- / \
\ \ / \--- Collection B
\ \--- sort -- rollup ----/
\
\
\ <--- jdbc source
{code}
{code}
/--- sort -- innerJoin ----\
/ \
/ ---- sort -- innerJoin ----- \ <--- jdbc source
Client <-- innerJoin <---< |
\ \ ---- sort -- innerJoin ----- / <--- rollup <---- Collection A
\ \ /
\ \--- sort -- innerJoin ----/
\
\
\ <--- jdbc source
{code}
I imagine partition expression would look something like this
{code}
partition(
inputA=<source stream A>,
inputB=<source stream B>,
work=<stream for the workers>,
over="fieldA,fieldB",
workers=6,
zkHost=<zk connection string>
)
{code}
for example
{code}
innerJoin(
partition(
inputA=jdbc(database1),
inputB=rollup(
search(collectionA, ...),
...
),
work=sort(
innerJoin(
inputA,
inputB,
on="fieldA,fieldB"
),
by="jdbcFieldC asc, collectionAFieldB desc"
),
workers=6,
zkHost=localhost:12345
),
jdbc(database2),
on="fieldZ"
)
{code}
> Add PartitionStream to Streaming Expressions
> --------------------------------------------
>
> Key: SOLR-9096
> URL: https://issues.apache.org/jira/browse/SOLR-9096
> Project: Solr
> Issue Type: New Feature
> Reporter: Dennis Gove
>
> The basic idea of a PartitionStream is to take a one or more input streams of tuples, partition them out to a set of workers such that each worker can work with a subset of the tuples, and then bring them all back into a single stream. This differs from a ParallelStream because in ParallelStream the data is partitioned at the source whereas with a PartitionStream one can take an existing stream and spread it out across workers.
> The use-case here is for when one has a source stream (or more) which cannot be parallelized at the source but which can be parallelized after some level of processing. I see this being used most for parallelized sort, rollups, or graph searches.
> {code}
> /--- sort ----\
> / \ /--- Collection A
> / ---- sort ----- \ /
> Client <--- rollup <----< <----- innerJoin <
> \ ---- sort ----- / \
> \ / \--- Collection B
> \--- sort ----/
> {code}
> {code}
> /--- sort -- rollup ----\
> / \ /--- Collection A
> / ---- sort -- rollup ----- \ /
> Client <-- innerJoin <---< <----- innerJoin <
> \ \ ---- sort -- rollup ----- / \
> \ \ / \--- Collection B
> \ \--- sort -- rollup ----/
> \
> \
> \ <--- jdbc source
> {code}
> {code}
> /--- sort -- innerJoin ----\
> / \
> / ---- sort -- innerJoin ----- \ <--- jdbc source
> Client <-- innerJoin <---< |
> \ \ ---- sort -- innerJoin ----- / <--- rollup <---- Collection A
> \ \ /
> \ \--- sort -- innerJoin ----/
> \
> \
> \ <--- jdbc source
> {code}
> I imagine partition expression would look something like this
> {code}
> partition(
> inputA=<source stream A>,
> inputB=<source stream B>,
> work=<stream for the workers>,
> over="fieldA,fieldB",
> workers=6,
> zkHost=<zk connection string>
> )
> {code}
> for example
> {code}
> innerJoin(
> partition(
> inputA=jdbc(database1),
> inputB=rollup(
> search(collectionA, ...),
> ...
> ),
> work=sort(
> innerJoin(
> inputA,
> inputB,
> on="fieldA,fieldB"
> ),
> by="jdbcFieldC asc, collectionAFieldB desc"
> ),
> workers=6,
> zkHost=localhost:12345
> ),
> jdbc(database2),
> on="fieldZ"
> )
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org