You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@reef.apache.org by "Yunseong Lee (JIRA)" <ji...@apache.org> on 2015/12/20 13:52:46 UTC

[jira] [Created] (REEF-1100) Introduce Vortex caching

Yunseong Lee created REEF-1100:
----------------------------------

             Summary: Introduce Vortex caching
                 Key: REEF-1100
                 URL: https://issues.apache.org/jira/browse/REEF-1100
             Project: REEF
          Issue Type: New Feature
          Components: Vortex
            Reporter: Yunseong Lee


In a Vortex application, many Tasklets can access the same input data. For example, in our {{IdentityMatMul}} example, the matrix on the right side is equivalent in all Tasklets. Also, machine learning algorithms like {{LogisticRegression}}, the same model is shared by all Tasklets in one iteration and the same training data is used in the next iterations. In such cases, caching data in Workers can reduce the huge amount of redundant I/O cost.

In a high-level, the cache works as following:
On the master-side,
1) User calls {{VortexThreadPool.cache(data)}} which keeps data in Master's memory, and returns a {{CacheKey}}.
2) As {{TInput}}, the {{CacheKey}} is shipped instead of the actual body of the data.

On the worker-side, {{VortexCache.get(cacheKey)}} can be called inside user code to access the cached data.
1) If the data is loaded, the data is returned immediately.
2) Otherwise, the Master is requested to send the data. The execution is blocked until the data is loaded into cache. Concurrent request for the same data is managed by Cache, so only a single remote fetch occurs per worker.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)