You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@mrql.apache.org by "Leonidas Fegaras (JIRA)" <ji...@apache.org> on 2016/07/15 22:14:20 UTC

[jira] [Created] (MRQL-92) Use outer-joins for incremental queries in Spark streaming mode

Leonidas Fegaras created MRQL-92:
------------------------------------

             Summary: Use outer-joins for incremental queries in Spark streaming mode
                 Key: MRQL-92
                 URL: https://issues.apache.org/jira/browse/MRQL-92
             Project: MRQL
          Issue Type: Improvement
          Components: Run-Time/Spark, Streaming
    Affects Versions: 0.9.8
            Reporter: Leonidas Fegaras
            Assignee: Leonidas Fegaras
            Priority: Critical


Currently, incremental queries use Spark's coGroup to merge the current state with the results of processing the new data in the stream. With this patch, the merge is done with a special outer join that doesn't shuffle the state again (it only shuffles the results from the new data).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)