[jira] [Updated] (OOZIE-547) build workflow progress information in Oozie

["Alejandro Abdelnur (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/OOZIE-547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alejandro Abdelnur updated OOZIE-547:
-------------------------------------

    Assignee: zhu jin wei  (was: Roman Shaposhnik)
     Summary: build workflow progress information in Oozie  (was: OOZIE-130: build workflow progress information in Oozie)

Zhu Jin Wei, I've just assigned the JIRA to you (this JIRA comes from the migration from the non-apache JIRA and the scripts doing that were run by Roman and all those JIRAs got assigned to Roman).

Regarding the JPA changes, the patch changes the schema and the way beans are serialized. After introducing these changes, pre-existing jobs will fail. This means we have to do a database upgrade. We can do this only as part of mayor releases.

I've run test-patch locally (skipping tests) and I didn't see any javac warnings either. I'll look into test-patch to provide more explicit info about the javac warnings.

Also, it seems testcases didn't run correctly, i'll trigger a rerun of test-patch.


Would you mind explaining exactly how the patch measures progress? AI think is OK for now to consider all actions opaque.

                
> build workflow progress information in Oozie
> --------------------------------------------
>
>                 Key: OOZIE-547
>                 URL: https://issues.apache.org/jira/browse/OOZIE-547
>             Project: Oozie
>          Issue Type: New Feature
>            Reporter: Hadoop QA
>            Assignee: zhu jin wei
>         Attachments: oozie-547.patch
>
>
> For a user, knowing progress of her workflow is always desirable. This ticket is to introduce this support to Oozie.
> I know it's a hard problem. For my initial effort, I plan to start with simple workflows that do not contain decision nodes or fork/join nodes, i.e., chain type workflows. I plan to use percentage of finished actions as the overall wf progress estimate.
> Going forward we can improve the estimation by:
> 1) handle general workflows that contain decision, fork/join nodes;
> 2) incorporate the action level progress into wf level progress estimation to make the estimate better. To be more specific:
> In the case of "opaque" actions like pig/hive/jaql where the status can only be 0% or 100% (or failure) we plug that value into the overall DAG status of 0-100%. If a DAG had say 4 opaque actions, the progress would move in discrete steps 0, 25, 50, 75, 100%.  For the m/r actions where the JobTracker
> gives values between 0-100% for an action then the overall progress will be smoother. We can do same thing for pig/hive/jaql actions as well if they expose their own progress info.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira