[jira] [Commented] (MRQL-66) Add support for MRQL streaming in Flink streaming mode

["Robert Metzger (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MRQL-66?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14307588#comment-14307588 ] 

Robert Metzger commented on MRQL-66:
------------------------------------

Cool! 
I've forwarded the JIRA to the Flink dev list. Maybe somebody from Flink is interested in helping or giving advice.

> Add support for MRQL streaming in Flink streaming mode
> ------------------------------------------------------
>
>                 Key: MRQL-66
>                 URL: https://issues.apache.org/jira/browse/MRQL-66
>             Project: MRQL
>          Issue Type: New Feature
>          Components: Run-Time/Flink, Streaming
>    Affects Versions: 0.9.6
>            Reporter: Leonidas Fegaras
>            Priority: Critical
>
> The new extension, MRQL Streaming, works fine with Spark Streaming (see MRQL-63) but it would be nice if we make it work with Flink Streaming too. It was easy to make it work with Spark Streaming: Data in one sliding window in a Spark's DStream is viewed as an RDD. So a DStream can be viewed as a continuous sequence of RDDs. A DStream has a method foreachRDD that applies a function to each RDD in the stream. So to implement MRQL Streaming, we just had to use the MRQL Spark evaluator (a function from RDD to RDD) as an argument to foreachRDD. For Flink Streaming, the implementation will be more complicated. A Flink Streaming DataStream doesn't provide a hook to a DataSet object. I am guessing that this is because Flink Streaming is far more general than Spark Streaming (it's not just sliding windows) and because Flink Streaming needs to do special optimizations. So we need to copy the FlinkEvaluator class into a new class FlinkStreaming and change all methods to be on DataStream instead of DataSet. Many DataSet methods have an equivalent in DataStream but some are missing. I have already provided the input formats for streaming (method FlinkStreaming.stream_source) but we need to write a stream evaluator for MRQL plans.
> Any volunteer?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)