You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by Sanjay Pujare <sa...@datatorrent.com> on 2016/11/23 22:40:45 UTC

Proposing a new feature to persist logical and physical plan snapshots in HDFS

To help Apex developers/users with debugging or troubleshooting “dead” applications, I am proposing a new feature to persist logical and physical plan snapshots in HDFS.

 

Similar to how the Apex engine persists container data per application attempt in HDFS as containers_NNN.json (where NNN is 1 for first app attempt, 2 for the second app attempt and so on), we will create 2 more sets of files under the …/apps/{appId} directory for an application:

 

logicalPlan_NNN_MMM.json

physicalPlan_NNN_MMM.json

 

where NNN stands for the app attempt index (similar to NNN above 1, 2, 3 and so on) and MMM is a running index starting at 1 which stands for a snapshot within an app attempt. Note that a logical or physical plan may change within an app attempt for any number of reasons.

 

The StreamingContainerManager class maintains the current logical/physical plans in the “plan” member variable. New methods will be added in StreamingContainerManager to save the logical or physical plan as JSON representations in the app directory (as described above). The logic is similar to com.datatorrent.stram.webapp.StramWebServices.getLogicalPlan(String) and com.datatorrent.stram.webapp.StramWebServices.getPhysicalPlan() used inside the Stram Web service. There will be running indexes in StreamingContainerManager to keep track of MMM for the logical plan and physical plan. The appropriate save method will be called on the occurrence of any event that updates the logical or physical plan for example:

 

inside com.datatorrent.stram.StreamingContainerManager.LogicalPlanChangeRunnable.call()  for logical plan change event

 

inside com.datatorrent.stram.plan.physical.PhysicalPlan.redoPartitions(PMapping, String) for physical plan change event (i.e. redoing partitioning)

 

Once these files are created, any user or a tool (such as the Apex CLI or the DT Gateway) can look up these files for troubleshooting/researching of “dead” applications and significant events in their lifetime in terms of logical or physical plan changes. Pls send me your feedback.

 

Sanjay

Re: Proposing a new feature to persist logical and physical plan snapshots in HDFS

Posted by Amol Kekre <am...@datatorrent.com>.
Persisted plan on DFS is good. I am +1 for it. This could be both of the
following

1. Attribute : If set, then upon change in plan persist to DFS
2. On demand

Thks
Amol


On Wed, Nov 23, 2016 at 4:15 PM, Sanjay Pujare <sa...@datatorrent.com>
wrote:

> Okay, but this “state” is gone after the app is “dead” isn’t that true?
> Also the reason for this enhancement is debuggability/troubleshooting of
> Apex apps so it is good to have separate explicit user visible files that
> contain the plan information instead of overloading the state for this
> purpose (in my opinion).
>
> In terms of on-demand, it sounds like a good idea - I didn’t think of it.
> But I would like to drill down the use cases. In most cases,
> logical/physical plan changes are spontaneous or rather internal to the app
> so an external entity making a REST call to save the plan on demand might
> not sync up with when the plan changes took place inside the app. So saving
> the plan JSON files on events described previously seems to be the most
> efficient thing to do (as discussed with @Ashwin Putta) but if there are
> use cases I think it is a good idea to do it on demand as well.
>
> On 11/23/16, 3:00 PM, "Amol Kekre" <am...@datatorrent.com> wrote:
>
>     Good idea. Stram does save state, and maybe a script that translates
> may
>     work. But explicit plan saving is also a good idea. Could this be "on
>     demand"? a rest call that writes out the plan(s) to specifid hdfs
> files?
>
>     We could do both (write on any change/set call) and/or on-demand.
>
>     Thks
>     Amol
>
>
>     On Wed, Nov 23, 2016 at 2:40 PM, Sanjay Pujare <sanjay@datatorrent.com
> >
>     wrote:
>
>     > To help Apex developers/users with debugging or troubleshooting
> “dead”
>     > applications, I am proposing a new feature to persist logical and
> physical
>     > plan snapshots in HDFS.
>     >
>     >
>     >
>     > Similar to how the Apex engine persists container data per
> application
>     > attempt in HDFS as containers_NNN.json (where NNN is 1 for first app
>     > attempt, 2 for the second app attempt and so on), we will create 2
> more
>     > sets of files under the …/apps/{appId} directory for an application:
>     >
>     >
>     >
>     > logicalPlan_NNN_MMM.json
>     >
>     > physicalPlan_NNN_MMM.json
>     >
>     >
>     >
>     > where NNN stands for the app attempt index (similar to NNN above 1,
> 2, 3
>     > and so on) and MMM is a running index starting at 1 which stands for
> a
>     > snapshot within an app attempt. Note that a logical or physical plan
> may
>     > change within an app attempt for any number of reasons.
>     >
>     >
>     >
>     > The StreamingContainerManager class maintains the current
> logical/physical
>     > plans in the “plan” member variable. New methods will be added in
>     > StreamingContainerManager to save the logical or physical plan as
> JSON
>     > representations in the app directory (as described above). The logic
> is
>     > similar to com.datatorrent.stram.webapp.StramWebServices.
> getLogicalPlan(String)
>     > and com.datatorrent.stram.webapp.StramWebServices.getPhysicalPlan()
> used
>     > inside the Stram Web service. There will be running indexes in
>     > StreamingContainerManager to keep track of MMM for the logical plan
> and
>     > physical plan. The appropriate save method will be called on the
> occurrence
>     > of any event that updates the logical or physical plan for example:
>     >
>     >
>     >
>     > inside com.datatorrent.stram.StreamingContainerManager.
>     > LogicalPlanChangeRunnable.call()  for logical plan change event
>     >
>     >
>     >
>     > inside com.datatorrent.stram.plan.physical.PhysicalPlan.
> redoPartitions(PMapping,
>     > String) for physical plan change event (i.e. redoing partitioning)
>     >
>     >
>     >
>     > Once these files are created, any user or a tool (such as the Apex
> CLI or
>     > the DT Gateway) can look up these files for
> troubleshooting/researching of
>     > “dead” applications and significant events in their lifetime in
> terms of
>     > logical or physical plan changes. Pls send me your feedback.
>     >
>     >
>     >
>     > Sanjay
>     >
>     >
>     >
>     >
>
>
>
>

Re: Proposing a new feature to persist logical and physical plan snapshots in HDFS

Posted by Sanjay Pujare <sa...@datatorrent.com>.
Okay, but this “state” is gone after the app is “dead” isn’t that true? Also the reason for this enhancement is debuggability/troubleshooting of Apex apps so it is good to have separate explicit user visible files that contain the plan information instead of overloading the state for this purpose (in my opinion).

In terms of on-demand, it sounds like a good idea - I didn’t think of it. But I would like to drill down the use cases. In most cases, logical/physical plan changes are spontaneous or rather internal to the app so an external entity making a REST call to save the plan on demand might not sync up with when the plan changes took place inside the app. So saving the plan JSON files on events described previously seems to be the most efficient thing to do (as discussed with @Ashwin Putta) but if there are use cases I think it is a good idea to do it on demand as well.

On 11/23/16, 3:00 PM, "Amol Kekre" <am...@datatorrent.com> wrote:

    Good idea. Stram does save state, and maybe a script that translates may
    work. But explicit plan saving is also a good idea. Could this be "on
    demand"? a rest call that writes out the plan(s) to specifid hdfs files?
    
    We could do both (write on any change/set call) and/or on-demand.
    
    Thks
    Amol
    
    
    On Wed, Nov 23, 2016 at 2:40 PM, Sanjay Pujare <sa...@datatorrent.com>
    wrote:
    
    > To help Apex developers/users with debugging or troubleshooting “dead”
    > applications, I am proposing a new feature to persist logical and physical
    > plan snapshots in HDFS.
    >
    >
    >
    > Similar to how the Apex engine persists container data per application
    > attempt in HDFS as containers_NNN.json (where NNN is 1 for first app
    > attempt, 2 for the second app attempt and so on), we will create 2 more
    > sets of files under the …/apps/{appId} directory for an application:
    >
    >
    >
    > logicalPlan_NNN_MMM.json
    >
    > physicalPlan_NNN_MMM.json
    >
    >
    >
    > where NNN stands for the app attempt index (similar to NNN above 1, 2, 3
    > and so on) and MMM is a running index starting at 1 which stands for a
    > snapshot within an app attempt. Note that a logical or physical plan may
    > change within an app attempt for any number of reasons.
    >
    >
    >
    > The StreamingContainerManager class maintains the current logical/physical
    > plans in the “plan” member variable. New methods will be added in
    > StreamingContainerManager to save the logical or physical plan as JSON
    > representations in the app directory (as described above). The logic is
    > similar to com.datatorrent.stram.webapp.StramWebServices.getLogicalPlan(String)
    > and com.datatorrent.stram.webapp.StramWebServices.getPhysicalPlan() used
    > inside the Stram Web service. There will be running indexes in
    > StreamingContainerManager to keep track of MMM for the logical plan and
    > physical plan. The appropriate save method will be called on the occurrence
    > of any event that updates the logical or physical plan for example:
    >
    >
    >
    > inside com.datatorrent.stram.StreamingContainerManager.
    > LogicalPlanChangeRunnable.call()  for logical plan change event
    >
    >
    >
    > inside com.datatorrent.stram.plan.physical.PhysicalPlan.redoPartitions(PMapping,
    > String) for physical plan change event (i.e. redoing partitioning)
    >
    >
    >
    > Once these files are created, any user or a tool (such as the Apex CLI or
    > the DT Gateway) can look up these files for troubleshooting/researching of
    > “dead” applications and significant events in their lifetime in terms of
    > logical or physical plan changes. Pls send me your feedback.
    >
    >
    >
    > Sanjay
    >
    >
    >
    >

Re: Proposing a new feature to persist logical and physical plan snapshots in HDFS

Posted by Amol Kekre <am...@datatorrent.com>.
Good idea. Stram does save state, and maybe a script that translates may
work. But explicit plan saving is also a good idea. Could this be "on
demand"? a rest call that writes out the plan(s) to specifid hdfs files?

We could do both (write on any change/set call) and/or on-demand.

Thks
Amol


On Wed, Nov 23, 2016 at 2:40 PM, Sanjay Pujare <sa...@datatorrent.com>
wrote:

> To help Apex developers/users with debugging or troubleshooting “dead”
> applications, I am proposing a new feature to persist logical and physical
> plan snapshots in HDFS.
>
>
>
> Similar to how the Apex engine persists container data per application
> attempt in HDFS as containers_NNN.json (where NNN is 1 for first app
> attempt, 2 for the second app attempt and so on), we will create 2 more
> sets of files under the …/apps/{appId} directory for an application:
>
>
>
> logicalPlan_NNN_MMM.json
>
> physicalPlan_NNN_MMM.json
>
>
>
> where NNN stands for the app attempt index (similar to NNN above 1, 2, 3
> and so on) and MMM is a running index starting at 1 which stands for a
> snapshot within an app attempt. Note that a logical or physical plan may
> change within an app attempt for any number of reasons.
>
>
>
> The StreamingContainerManager class maintains the current logical/physical
> plans in the “plan” member variable. New methods will be added in
> StreamingContainerManager to save the logical or physical plan as JSON
> representations in the app directory (as described above). The logic is
> similar to com.datatorrent.stram.webapp.StramWebServices.getLogicalPlan(String)
> and com.datatorrent.stram.webapp.StramWebServices.getPhysicalPlan() used
> inside the Stram Web service. There will be running indexes in
> StreamingContainerManager to keep track of MMM for the logical plan and
> physical plan. The appropriate save method will be called on the occurrence
> of any event that updates the logical or physical plan for example:
>
>
>
> inside com.datatorrent.stram.StreamingContainerManager.
> LogicalPlanChangeRunnable.call()  for logical plan change event
>
>
>
> inside com.datatorrent.stram.plan.physical.PhysicalPlan.redoPartitions(PMapping,
> String) for physical plan change event (i.e. redoing partitioning)
>
>
>
> Once these files are created, any user or a tool (such as the Apex CLI or
> the DT Gateway) can look up these files for troubleshooting/researching of
> “dead” applications and significant events in their lifetime in terms of
> logical or physical plan changes. Pls send me your feedback.
>
>
>
> Sanjay
>
>
>
>