You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2017/01/05 13:44:58 UTC
[jira] [Commented] (BEAM-649) Cache any PCollection implementation
if accessed more than once
[ https://issues.apache.org/jira/browse/BEAM-649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15801381#comment-15801381 ]
ASF GitHub Bot commented on BEAM-649:
-------------------------------------
GitHub user jbonofre opened a pull request:
https://github.com/apache/beam/pull/1739
[BEAM-649] Analyse DAG to determine if RDD/DStream has to be cached or not
Be sure to do all of the following to help us incorporate your contribution
quickly and easily:
- [X] Make sure the PR title is formatted like:
`[BEAM-<Jira issue #>] Description of pull request`
- [X] Make sure tests pass via `mvn clean verify`. (Even better, enable
Travis-CI on your fork and ensure the whole test matrix passes).
- [X] Replace `<Jira issue #>` in the title with the actual Jira issue
number, if there is one.
- [X] If this contribution is large, please file an Apache
[Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt).
---
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jbonofre/beam BEAM-649
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/beam/pull/1739.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1739
----
commit fbde8ec0126ea3ba645726551f39feadc2169080
Author: Jean-Baptiste Onofré <jb...@apache.org>
Date: 2017-01-04T20:23:17Z
[BEAM-649] Analyse DAG to determine if RDD/DStream has to be cached or not
----
> Cache any PCollection implementation if accessed more than once
> ---------------------------------------------------------------
>
> Key: BEAM-649
> URL: https://issues.apache.org/jira/browse/BEAM-649
> Project: Beam
> Issue Type: Improvement
> Components: runner-spark
> Reporter: Amit Sela
> Assignee: Jean-Baptiste Onofré
>
> Currently, the runner will cache any {{PCollection}} implementation - {{RDD}} or {{DStream}} - if accessed for the second time.
> This can be further optimized to cache after the first evaluation, if accessed again, and also solve issues in BEAM-1206.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)