You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mrql.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2015/11/03 18:21:27 UTC
[jira] [Commented] (MRQL-79) Add support for incremental query
processing
[ https://issues.apache.org/jira/browse/MRQL-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14987664#comment-14987664 ]
ASF GitHub Bot commented on MRQL-79:
------------------------------------
GitHub user fegaras opened a pull request:
https://github.com/apache/incubator-mrql/pull/11
[MRQL-79] Add support for incremental query processing
The framework for incremental stream processing is described at [streams15.pdf](http://lambda.uta.edu/streams15.pdf). Most of the changes are at core/src/main/java/org/apache/mrql/Streaming.gen. The queries queries/incremental-*.mrql describe various examples. For example, to run k-mean clustering in incremental mode, first create the data:
`bin/mrql.spark -local queries/points.mrql 1000`
Then, process the data incrementally:
`bin/mrql.spark -local -stream 1000 queries/incremental-kmeans.mrql`
In a separate terminal, use `touch tmp/points.bin/part-00000` to change the timestamp of the file to process the file again.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/fegaras/incubator-mrql MRQL-79
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-mrql/pull/11.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11
----
commit 1cf73e2c8aa332a536548221aacc878e93174189
Author: fegaras <fe...@cse.uta.edu>
Date: 2015-11-03T17:03:35Z
[MRQL-79] Add support for incremental query processing
----
> Add support for incremental query processing
> --------------------------------------------
>
> Key: MRQL-79
> URL: https://issues.apache.org/jira/browse/MRQL-79
> Project: MRQL
> Issue Type: New Feature
> Components: Run-Time/Spark, Streaming
> Affects Versions: 0.9.6
> Reporter: Leonidas Fegaras
> Assignee: Leonidas Fegaras
>
> This is a new feature for MRQL streaming: its task is to convert any stream-based MRQL query to an incremental query that merges the previous query results with the results of applying the query to the new data batches only. For example, it will be able to convert the MRQL PageRank query to an incremental PageRank query automatically. The basic idea was presented at ApacheCon'15 (page 28 in http://lambda.uta.edu/mrql-apachecon15.pdf ) as a future plan for MRQL. It will work on Spark Streaming mode for now, but later it will support Flink Streaming too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)