You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@crunch.apache.org by "Ryan Brush (JIRA)" <ji...@apache.org> on 2015/02/02 18:00:39 UTC
[jira] [Commented] (CRUNCH-438) Visualizations of some important
internal/intermediate pipeline planning states
[ https://issues.apache.org/jira/browse/CRUNCH-438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14301459#comment-14301459 ]
Ryan Brush commented on CRUNCH-438:
-----------------------------------
Unless I'm missing something, this change appears to have removed the logic done in CRUNCH-481 to deal with multiple output committers. (Notice in the commit tagged below, the CrunchOutputFormat and related changes are deleted.) I'm guessing this was an error as part of this merge and the updates for CRUNCH-481 should be re-added?
https://github.com/apache/crunch/commit/6543ded556dbbd4e8e25e59730f7f5fa8e2c3c26
> Visualizations of some important internal/intermediate pipeline planning states
> -------------------------------------------------------------------------------
>
> Key: CRUNCH-438
> URL: https://issues.apache.org/jira/browse/CRUNCH-438
> Project: Crunch
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.10.0, 0.8.3
> Reporter: Christian Tzolov
> Assignee: Christian Tzolov
> Attachments: CRUNCH-438.2.patch, CRUNCH-438.3.patch, CRUNCH-438.4.patch, CRUNCH-438.5.patch, CRUNCH-438.6.patch, CRUNCH-438.7.patch, CRUNCH-438.8.patch, CRUNCH-438.patch
>
>
> To improve the understability of the pipeline planning stages it would help to visualize some intermediate planning states like:
> - PCollection lineage. (visualizing the output-pcollection-targets structure)
> - MSCRPlanner's planning Graphs before and after the split up of dependent GBK nodes
> - RTNode hierarchy along with the Input and Output configurations as persistent in the Configuration before the execution of the pipeline.
> Most of the information can be intercepted in the MSCRPlanner#plan() method.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)