[jira] [Commented] (APEXCORE-586) Enhancement to persist logical and physical plan snapshots in HDFS

["Sanjay M Pujare (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/APEXCORE-586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15723879#comment-15723879 ] 

Sanjay M Pujare commented on APEXCORE-586:
------------------------------------------

The following is the implementation plan. We will soon start the implementation so pls send your feedback as soon as possible.

We will refactor code to move the logic in

com.datatorrent.stram.webapp.StramWebServices.getLogicalPlan(String)

and

com.datatorrent.stram.webapp.StramWebServices.getPhysicalPlan()

to the class

com.datatorrent.stram.StreamingContainerManager

So these 2 methods will return JSONObject representation of the logical or physical plan.

2 new methods will be added to StreamingContainerManager:

writePhysicalPlan()
writeLogicalPlan()

The first method will create an HDFS file with the name physicalPlan_NNN_MMM.json under the current app's app directory (where NNN is the current app attempt id and MMM is	a running count (maintained in the StreamingContainerManager object) that	is incremented on every call to the method. The method will call getPhysicalPlan() to get the JSON object and write it out to the HDFS file. writeLogicalPlan() will be similarly written to write the logical plan to the file logicalPlan_NNN_MMM.json. There is a separate count for the MMM for logical plans.

The method com.datatorrent.stram.plan.physical.PhysicalPlan.onStatusUpdate(PTOperator) will be modified to accept a parameter for the StreamingContainerManager object. Inside this method after redoPartition() is	called, writePhysicalPlan() will be called on the StreamingContainerManager object which will write a new physicalPlan_NNN_MMM.json file.

Similarly in com.datatorrent.stram.StreamingContainerManager.LogicalPlanChangeRunnable.call() at the end writeLogicalPlan() will be called which will write a new logicalPlan_NNN_MMM.json file.

Unit tests will be written to cover existing functionality and new functionality to minimize the possibility of regressions and to cover the new code.


> Enhancement to persist logical and physical plan snapshots in HDFS
> ------------------------------------------------------------------
>
>                 Key: APEXCORE-586
>                 URL: https://issues.apache.org/jira/browse/APEXCORE-586
>             Project: Apache Apex Core
>          Issue Type: Improvement
>            Reporter: Sanjay M Pujare
>            Assignee: Sanjay M Pujare
>
> Pls refer to the discussion on dev@apex 
> http://apache-apex-developers-list.78491.x6.nabble.com/Proposing-a-new-feature-to-persist-logical-and-physical-plan-snapshots-in-HDFS-td11592.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)