You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@crunch.apache.org by "Ioannis Kerkinos (JIRA)" <ji...@apache.org> on 2015/03/31 13:26:52 UTC

[jira] [Created] (CRUNCH-505) Store intermediate data in memory only using Tachyon

Ioannis Kerkinos created CRUNCH-505:
---------------------------------------

             Summary: Store intermediate data in memory only using Tachyon
                 Key: CRUNCH-505
                 URL: https://issues.apache.org/jira/browse/CRUNCH-505
             Project: Crunch
          Issue Type: Improvement
          Components: Core
    Affects Versions: 0.12.0
            Reporter: Ioannis Kerkinos
            Assignee: Josh Wills


Tachyon is a memory-centric distributed storage system that enables reliable data sharing at memory-speed. If used as the storage for intermediate data (between MR jobs) it should improve performance as you won't have to go to HDFS. In order to do so, the MUST_CACHE write type of Tachyon can be used. This will enable data to be persisted in memory only without going to HDFS. So the intermediate data will be read/written at memory-speed and only the final result will be written in HDFS.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)