You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2016/03/25 22:03:25 UTC
[jira] [Commented] (MAHOUT-1817) Implement caching in Flink
Bindings
[ https://issues.apache.org/jira/browse/MAHOUT-1817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15212404#comment-15212404 ]
ASF GitHub Bot commented on MAHOUT-1817:
----------------------------------------
GitHub user andrewpalumbo opened a pull request:
https://github.com/apache/mahout/pull/203
MAHOUT-1817 Implement caching in Flink Bindings
As a temporary measure, use this method to persist the `DataSet` to the filesystem when caching rather that drmDfsRead()/Write.
Todo:
1. Break up into `persist` and `readPersistedDataset` methods and only read a persisted dataset if it is already cached.
2. Use a property setting for the base dir.
3. Check to make sure that this method maintains parallelism deg for the dataset, if not set the new parallelism degree to the original
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/andrewpalumbo/mahout MAHOUT-1817
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/mahout/pull/203.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #203
----
commit a1cf7cf56e036ce12d616f8aea0af1e9dcdf2cb6
Author: Andrew Palumbo <ap...@apache.org>
Date: 2016-03-25T20:56:20Z
Small change addressing DL's comment on apache/mahout#200, also a small fix
----
> Implement caching in Flink Bindings
> -----------------------------------
>
> Key: MAHOUT-1817
> URL: https://issues.apache.org/jira/browse/MAHOUT-1817
> Project: Mahout
> Issue Type: New Feature
> Components: Flink
> Affects Versions: 0.11.2
> Reporter: Andrew Palumbo
> Assignee: Andrew Palumbo
> Priority: Blocker
> Fix For: 0.12.0
>
>
> Flink does not have in-memory caching analogous to that of Spark. We need find a way to honour the {{checkpoint()}} contract in Flink Bindings.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)