You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@mrql.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2016/10/19 23:46:58 UTC

[jira] [Commented] (MRQL-98) Improve Data Serialization in Spark Evaluation

    [ https://issues.apache.org/jira/browse/MRQL-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15590268#comment-15590268 ] 

ASF GitHub Bot commented on MRQL-98:
------------------------------------

GitHub user fegaras opened a pull request:

    https://github.com/apache/incubator-mrql/pull/28

    [MRQL-98] Improve Data Serialization in Spark Evaluation

    

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/fegaras/incubator-mrql MRQL-98

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-mrql/pull/28.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #28
    
----
commit ab6bac73711b0d5fa5f05fa2c9b4c558c0576042
Author: Leonidas Fegaras <fe...@cse.uta.edu>
Date:   2016-10-19T23:45:07Z

    [MRQL-98] Improve Data Serialization in Spark Evaluation

----


> Improve Data Serialization in Spark Evaluation
> ----------------------------------------------
>
>                 Key: MRQL-98
>                 URL: https://issues.apache.org/jira/browse/MRQL-98
>             Project: MRQL
>          Issue Type: Improvement
>          Components: Run-Time/Spark
>    Affects Versions: 0.9.8
>            Reporter: Leonidas Fegaras
>            Assignee: Leonidas Fegaras
>            Priority: Critical
>
> MRQL data (MRData) are serialized as Writable (for Hadoop Map-Reduce), Java Serializable (for Spark), and CopyableValue (for Flink). Until now, the Spark MRQL engine was using a wrapper for MRData (called MRContainer) to serialize data using the Writable methods. Some data used in Spark mode though were left unwrapped, so Spark was using the default Java serialization, which was inefficient. With this patch, MRData becomes Serializable with custom serialization methods that are very efficient. My performance evaluation of the Pagerank query over 10 millions links run on a cluster with 16 cores gives 38% improvement compared to the old Spark evaluation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)