[jira] [Commented] (MRQL-55) Add support for Hadoop Sequence input format in flink mode

["Leonidas Fegaras (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MRQL-55?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14176330#comment-14176330 ] 

Leonidas Fegaras commented on MRQL-55:
--------------------------------------

Here are some performance results (in secs) on a small Yarn cluster with 12 nodes (48 cores):

# PageRank (6 steps) 1M nodes, 10M edges
# K-means clustering (5 steps) 10M points
# DBLP XML PageRank (12 steps) 1.5GB
# matrix multiplication 500x500

{noformat}
     Map-Reduce    Spark      Flink
-----------------------------------------------
1       591.8       145.1     145.3
2      1068.1       184.0     516.4
3       994.2       149.4     181.6
4        78.7        83.2      94.9
{noformat}

k-means is slower in Flink mode than in Spark mode because MRQL doesn't use Flink iterations for k-means (but it does use Flink iterations for pagerank).

> Add support for Hadoop Sequence input format in flink mode
> ----------------------------------------------------------
>
>                 Key: MRQL-55
>                 URL: https://issues.apache.org/jira/browse/MRQL-55
>             Project: MRQL
>          Issue Type: Improvement
>          Components: Run-Time/Flink
>    Affects Versions: 0.9.4
>            Reporter: Leonidas Fegaras
>            Assignee: Leonidas Fegaras
>            Priority: Minor
>         Attachments: MRQL-55.patch
>
>
> The following patch adds support for hadoop Sequence input format in flink mode. Before this, we used the flink binary input format to read/write binary files, which was not compatible with other MRQL evaluation modes. The patch also fixes the mrql.flink script to get the flink job manager from conf/.yarn-properties instead of conf/.yarn-jobmanager.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)