You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@systemml.apache.org by "Matthias Boehm (JIRA)" <ji...@apache.org> on 2016/03/28 01:48:25 UTC
[jira] [Updated] (SYSTEMML-560) Distributed frame representation
[ https://issues.apache.org/jira/browse/SYSTEMML-560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matthias Boehm updated SYSTEMML-560:
------------------------------------
Description:
The major design goals for our distributed binary block frame representation are twofold:
* Seamless integration: First, we aim for a seamless integration with (1) Spark's DataFrame and DataSet representations, (2) csv text formats, and (3) SystemML's binary block matrix representations.
* Memory efficiency: Second, we are still interested in a block representation to exploit the column-wise native array storage of FrameBlocks.
As a good compromise with regard to both design goals, the initial design proposal is
{code}
FRAME := JavaPairRDD<Long, FrameBlock>
{code}
where the keys represents the row offsets of frameblock values, a frameblock value covers one or multiple rows and all columns of the frame, and most importantly frameblocks values do not exhibit a fixed block size.
> Distributed frame representation
> --------------------------------
>
> Key: SYSTEMML-560
> URL: https://issues.apache.org/jira/browse/SYSTEMML-560
> Project: SystemML
> Issue Type: Task
> Reporter: Matthias Boehm
>
> The major design goals for our distributed binary block frame representation are twofold:
> * Seamless integration: First, we aim for a seamless integration with (1) Spark's DataFrame and DataSet representations, (2) csv text formats, and (3) SystemML's binary block matrix representations.
> * Memory efficiency: Second, we are still interested in a block representation to exploit the column-wise native array storage of FrameBlocks.
> As a good compromise with regard to both design goals, the initial design proposal is
> {code}
> FRAME := JavaPairRDD<Long, FrameBlock>
> {code}
> where the keys represents the row offsets of frameblock values, a frameblock value covers one or multiple rows and all columns of the frame, and most importantly frameblocks values do not exhibit a fixed block size.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)