You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by Abhishek Bafna <ba...@gmail.com> on 2016/06/22 08:51:32 UTC
Review Request 49074: OOZIE-2582: Populating external child Ids for
action failures
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49074/
-----------------------------------------------------------
Review request for oozie.
Bugs: OOZIE-2582
https://issues.apache.org/jira/browse/OOZIE-2582
Repository: oozie-git
Description
-------
Currently Oozie external child ids are populated into workflow bean, when the job/action completes successfully. It should populate external child ids in case of job failures as well.
Diffs
-----
sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java cb0172d
sharelib/hive/src/test/java/org/apache/oozie/action/hadoop/TestHiveMain.java d72e298
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java fb190d6
sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java 360cf1e
Diff: https://reviews.apache.org/r/49074/diff/
Testing
-------
Thanks,
Abhishek Bafna
Re: Review Request 49074: OOZIE-2582: Populating external child Ids
for action failures
Posted by Abhishek Bafna <ba...@gmail.com>.
> On July 7, 2016, 8:57 p.m., Satish Saley wrote:
> > sharelib/distcp/src/main/java/org/apache/oozie/action/hadoop/DistcpMain.java, line 41
> > <https://reviews.apache.org/r/49074/diff/2/?file=1438356#file1438356line41>
> >
> > We don't need "Job complete" pattern. It will end up with fetching child job id twice and in UI will contain duplicate urls. Only "Submitted application" is sufficient.
I agree with you. To prevent the duplicates, I added a conditional check in the LauncherMain, which checks the ids first and if not present alreadt, then appends. I have removed the above patterns. I will marking other two also as Fixed. Thanks.
> On July 7, 2016, 8:57 p.m., Satish Saley wrote:
> > sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java, line 96
> > <https://reviews.apache.org/r/49074/diff/2/?file=1438360#file1438360line96>
> >
> > It became clear after discussing with Rohini, we will write only to ACTION_DATA_EXTERNAL_CHILD_IDS.
> >
> > And for backward compatibility, we need to make changes in the JavaActionExecutor to read from ACTION_DATA_OUTPUT_PROPS as well. Also, we need to remove getCaptureOutput() and getActionData() methods from executors.
> > I will fix those things as part of OOZIE-2561
As per the changes here, I think it is not braeking any backward compatibility things. I do observe that, currently with the successful job, external child ids are set twice, once using the ACTION_DATA_EXTERNAL_CHILD_IDS and later in the getActionData();. I guess, that is what you wanted to say for, getActionData(). I did not get the getCaptureOutput() output part in your comment. Thanks.
- Abhishek
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49074/#review141204
-----------------------------------------------------------
On July 7, 2016, 5:41 a.m., Abhishek Bafna wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49074/
> -----------------------------------------------------------
>
> (Updated July 7, 2016, 5:41 a.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-2582
> https://issues.apache.org/jira/browse/OOZIE-2582
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Currently Oozie external child ids are populated into workflow bean, when the job/action completes successfully. It should populate external child ids in case of job failures as well.
>
>
> Diffs
> -----
>
> examples/src/main/apps/hive/script.q 3abc757
> sharelib/distcp/src/main/java/org/apache/oozie/action/hadoop/DistcpMain.java 6ac5ad6
> sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java cb0172d
> sharelib/hive/src/test/java/org/apache/oozie/action/hadoop/TestHiveMain.java d72e298
> sharelib/hive2/src/test/java/org/apache/oozie/action/hadoop/TestHive2ActionExecutor.java 5963e42
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java fb190d6
> sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java 360cf1e
>
> Diff: https://reviews.apache.org/r/49074/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Abhishek Bafna
>
>
Re: Review Request 49074: OOZIE-2582: Populating external child Ids
for action failures
Posted by Satish Saley <sa...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49074/#review141204
-----------------------------------------------------------
sharelib/distcp/src/main/java/org/apache/oozie/action/hadoop/DistcpMain.java (line 41)
<https://reviews.apache.org/r/49074/#comment206618>
We don't need "Job complete" pattern. It will end up with fetching child job id twice and in UI will contain duplicate urls. Only "Submitted application" is sufficient.
sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java (line 46)
<https://reviews.apache.org/r/49074/#comment206620>
same as above. Only "Submitted application" should be sufficient.
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java (line 96)
<https://reviews.apache.org/r/49074/#comment206623>
It became clear after discussing with Rohini, we will write only to ACTION_DATA_EXTERNAL_CHILD_IDS.
And for backward compatibility, we need to make changes in the JavaActionExecutor to read from ACTION_DATA_OUTPUT_PROPS as well. Also, we need to remove getCaptureOutput() and getActionData() methods from executors.
I will fix those things as part of OOZIE-2561
sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java (line 41)
<https://reviews.apache.org/r/49074/#comment206636>
Only "Submitted application" is sufficient, else will have duplicated links
- Satish Saley
On July 6, 2016, 10:41 p.m., Abhishek Bafna wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49074/
> -----------------------------------------------------------
>
> (Updated July 6, 2016, 10:41 p.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-2582
> https://issues.apache.org/jira/browse/OOZIE-2582
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Currently Oozie external child ids are populated into workflow bean, when the job/action completes successfully. It should populate external child ids in case of job failures as well.
>
>
> Diffs
> -----
>
> examples/src/main/apps/hive/script.q 3abc757
> sharelib/distcp/src/main/java/org/apache/oozie/action/hadoop/DistcpMain.java 6ac5ad6
> sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java cb0172d
> sharelib/hive/src/test/java/org/apache/oozie/action/hadoop/TestHiveMain.java d72e298
> sharelib/hive2/src/test/java/org/apache/oozie/action/hadoop/TestHive2ActionExecutor.java 5963e42
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java fb190d6
> sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java 360cf1e
>
> Diff: https://reviews.apache.org/r/49074/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Abhishek Bafna
>
>
Re: Review Request 49074: OOZIE-2582: Populating external child Ids
for action failures
Posted by Abhishek Bafna <ba...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49074/
-----------------------------------------------------------
(Updated Sept. 24, 2016, 6:42 a.m.)
Review request for oozie.
Bugs: OOZIE-2582
https://issues.apache.org/jira/browse/OOZIE-2582
Repository: oozie-git
Description
-------
Currently Oozie external child ids are populated into workflow bean, when the job/action completes successfully. It should populate external child ids in case of job failures as well.
Diffs (updated)
-----
core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 96726da
core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java b5b1bf9
core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java c74e9e6
core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java ad07b57
core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 15a641b
core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 6813a37
core/src/test/java/org/apache/oozie/action/hadoop/TestDistcpMain.java 098995e
examples/src/main/apps/hive/script.q 3abc757
sharelib/distcp/src/main/java/org/apache/oozie/action/hadoop/DistcpMain.java 6ac5ad6
sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java 8de3766
sharelib/hive/src/test/java/org/apache/oozie/action/hadoop/TestHiveActionExecutor.java b966d4b
sharelib/hive/src/test/java/org/apache/oozie/action/hadoop/TestHiveMain.java d72e298
sharelib/hive2/src/main/java/org/apache/oozie/action/hadoop/Hive2Main.java e122608
sharelib/hive2/src/test/java/org/apache/oozie/action/hadoop/TestHive2ActionExecutor.java 5963e42
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java 054f8ea
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java 5e80d00
sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java 44debbd
sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMainWithOldAPI.java a5291d9
sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java 38e8e8c
sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java 623fd2e
sharelib/sqoop/src/test/java/org/apache/oozie/action/hadoop/TestSqoopActionExecutor.java 6474092
Diff: https://reviews.apache.org/r/49074/diff/
Testing
-------
Thanks,
Abhishek Bafna
Re: Review Request 49074: OOZIE-2582: Populating external child Ids
for action failures
Posted by Abhishek Bafna <ba...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49074/
-----------------------------------------------------------
(Updated Sept. 23, 2016, 4:16 p.m.)
Review request for oozie.
Bugs: OOZIE-2582
https://issues.apache.org/jira/browse/OOZIE-2582
Repository: oozie-git
Description
-------
Currently Oozie external child ids are populated into workflow bean, when the job/action completes successfully. It should populate external child ids in case of job failures as well.
Diffs (updated)
-----
core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 96726da
core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java b5b1bf9
core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java c74e9e6
core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java ad07b57
core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 15a641b
core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 6813a37
core/src/test/java/org/apache/oozie/action/hadoop/TestDistcpMain.java 098995e
examples/src/main/apps/hive/script.q 3abc757
sharelib/distcp/src/main/java/org/apache/oozie/action/hadoop/DistcpMain.java 6ac5ad6
sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java 8de3766
sharelib/hive/src/test/java/org/apache/oozie/action/hadoop/TestHiveActionExecutor.java b966d4b
sharelib/hive/src/test/java/org/apache/oozie/action/hadoop/TestHiveMain.java d72e298
sharelib/hive2/src/test/java/org/apache/oozie/action/hadoop/TestHive2ActionExecutor.java 5963e42
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java 054f8ea
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java 5e80d00
sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java 44debbd
sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMainWithOldAPI.java a5291d9
sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java 38e8e8c
sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java 623fd2e
sharelib/sqoop/src/test/java/org/apache/oozie/action/hadoop/TestSqoopActionExecutor.java 6474092
Diff: https://reviews.apache.org/r/49074/diff/
Testing
-------
Thanks,
Abhishek Bafna
Re: Review Request 49074: OOZIE-2582: Populating external child Ids
for action failures
Posted by Abhishek Bafna <ba...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49074/
-----------------------------------------------------------
(Updated Sept. 20, 2016, 7:21 p.m.)
Review request for oozie.
Bugs: OOZIE-2582
https://issues.apache.org/jira/browse/OOZIE-2582
Repository: oozie-git
Description
-------
Currently Oozie external child ids are populated into workflow bean, when the job/action completes successfully. It should populate external child ids in case of job failures as well.
Diffs (updated)
-----
core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 96726da
core/src/main/java/org/apache/oozie/action/hadoop/Hive2ActionExecutor.java b5b1bf9
core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java c74e9e6
core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java ad07b57
core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java 15a641b
core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 6813a37
examples/src/main/apps/hive/script.q 3abc757
sharelib/distcp/src/main/java/org/apache/oozie/action/hadoop/DistcpMain.java 6ac5ad6
sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java 8de3766
sharelib/hive/src/test/java/org/apache/oozie/action/hadoop/TestHiveActionExecutor.java b966d4b
sharelib/hive/src/test/java/org/apache/oozie/action/hadoop/TestHiveMain.java d72e298
sharelib/hive2/src/test/java/org/apache/oozie/action/hadoop/TestHive2ActionExecutor.java 5963e42
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java 054f8ea
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/ShellMain.java 5e80d00
sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java 44debbd
sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMainWithOldAPI.java a5291d9
sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java 38e8e8c
sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java 623fd2e
sharelib/sqoop/src/test/java/org/apache/oozie/action/hadoop/TestSqoopActionExecutor.java 6474092
Diff: https://reviews.apache.org/r/49074/diff/
Testing
-------
Thanks,
Abhishek Bafna
Re: Review Request 49074: OOZIE-2582: Populating external child Ids
for action failures
Posted by Abhishek Bafna <ba...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49074/
-----------------------------------------------------------
(Updated July 28, 2016, 5:09 a.m.)
Review request for oozie.
Bugs: OOZIE-2582
https://issues.apache.org/jira/browse/OOZIE-2582
Repository: oozie-git
Description
-------
Currently Oozie external child ids are populated into workflow bean, when the job/action completes successfully. It should populate external child ids in case of job failures as well.
Diffs (updated)
-----
examples/src/main/apps/hive/script.q 3abc757
sharelib/distcp/src/main/java/org/apache/oozie/action/hadoop/DistcpMain.java 6ac5ad6
sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java d8ce232
sharelib/hive/src/test/java/org/apache/oozie/action/hadoop/TestHiveMain.java d72e298
sharelib/hive2/src/test/java/org/apache/oozie/action/hadoop/TestHive2ActionExecutor.java 5963e42
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java fb190d6
sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java b0c7635
Diff: https://reviews.apache.org/r/49074/diff/
Testing
-------
Thanks,
Abhishek Bafna
Re: Review Request 49074: OOZIE-2582: Populating external child Ids
for action failures
Posted by Abhishek Bafna <ba...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49074/
-----------------------------------------------------------
(Updated July 10, 2016, 3:10 p.m.)
Review request for oozie.
Bugs: OOZIE-2582
https://issues.apache.org/jira/browse/OOZIE-2582
Repository: oozie-git
Description
-------
Currently Oozie external child ids are populated into workflow bean, when the job/action completes successfully. It should populate external child ids in case of job failures as well.
Diffs (updated)
-----
examples/src/main/apps/hive/script.q 3abc757
sharelib/distcp/src/main/java/org/apache/oozie/action/hadoop/DistcpMain.java 6ac5ad6
sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java cb0172d
sharelib/hive/src/test/java/org/apache/oozie/action/hadoop/TestHiveMain.java d72e298
sharelib/hive2/src/test/java/org/apache/oozie/action/hadoop/TestHive2ActionExecutor.java 5963e42
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java fb190d6
sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java 360cf1e
Diff: https://reviews.apache.org/r/49074/diff/
Testing
-------
Thanks,
Abhishek Bafna
Re: Review Request 49074: OOZIE-2582: Populating external child Ids
for action failures
Posted by Abhishek Bafna <ba...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49074/
-----------------------------------------------------------
(Updated July 8, 2016, 9:45 a.m.)
Review request for oozie.
Changes
-------
Uploading patch after incorporating suggestions by Satish.
Bugs: OOZIE-2582
https://issues.apache.org/jira/browse/OOZIE-2582
Repository: oozie-git
Description
-------
Currently Oozie external child ids are populated into workflow bean, when the job/action completes successfully. It should populate external child ids in case of job failures as well.
Diffs (updated)
-----
examples/src/main/apps/hive/script.q 3abc757
sharelib/distcp/src/main/java/org/apache/oozie/action/hadoop/DistcpMain.java 6ac5ad6
sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java cb0172d
sharelib/hive/src/test/java/org/apache/oozie/action/hadoop/TestHiveMain.java d72e298
sharelib/hive2/src/test/java/org/apache/oozie/action/hadoop/TestHive2ActionExecutor.java 5963e42
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java fb190d6
sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java 360cf1e
Diff: https://reviews.apache.org/r/49074/diff/
Testing
-------
Thanks,
Abhishek Bafna
Re: Review Request 49074: OOZIE-2582: Populating external child Ids
for action failures
Posted by Abhishek Bafna <ba...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49074/
-----------------------------------------------------------
(Updated July 7, 2016, 5:41 a.m.)
Review request for oozie.
Bugs: OOZIE-2582
https://issues.apache.org/jira/browse/OOZIE-2582
Repository: oozie-git
Description
-------
Currently Oozie external child ids are populated into workflow bean, when the job/action completes successfully. It should populate external child ids in case of job failures as well.
Diffs (updated)
-----
examples/src/main/apps/hive/script.q 3abc757
sharelib/distcp/src/main/java/org/apache/oozie/action/hadoop/DistcpMain.java 6ac5ad6
sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java cb0172d
sharelib/hive/src/test/java/org/apache/oozie/action/hadoop/TestHiveMain.java d72e298
sharelib/hive2/src/test/java/org/apache/oozie/action/hadoop/TestHive2ActionExecutor.java 5963e42
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java fb190d6
sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java 360cf1e
Diff: https://reviews.apache.org/r/49074/diff/
Testing
-------
Thanks,
Abhishek Bafna