You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by András Piros <an...@cloudera.com> on 2017/04/04 10:37:22 UTC
Review Request 58171: OOZIE-2844 Increase stability of Oozie actions
when log4j.properties is missing or not readable
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58171/
-----------------------------------------------------------
Review request for oozie and Peter Bacsko.
Repository: oozie-git
Description
-------
In several Oozie actions we face the problem that if `log4j.properties` is not readable by current thread, or not present on the classpath, while taking an `URL.openStream()` call afterwards, an `IOException` is thrown and the action won't run.
Such actions are:
* [`DistcpMain`](https://github.com/apache/oozie/blob/master/sharelib/distcp/src/main/java/org/apache/oozie/action/hadoop/DistcpMain.java#L135)
* [`HiveMain`](https://github.com/apache/oozie/blob/master/sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java#L146)
* [`PigMain`](https://github.com/apache/oozie/blob/master/sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java#L173)
* [`PigMainWithOldApi`](https://github.com/apache/oozie/blob/master/sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMainWithOldAPI.java#L160)
* [`SparkMain`](https://github.com/apache/oozie/blob/master/sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java#L430)
* [`SqoopMain`](https://github.com/apache/oozie/blob/master/sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java#L124)
Extend [`LauncherMain.run()`](https://github.com/apache/oozie/blob/master/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java#L62-L65) by a convenience method that loads Hadoop's `log4j.properties` in beforehand. In case `URL.openStream()` would throw `IOException` we resort to a call to a new file `hadoop-default-log4j.properties` on the classpath (packaged along w/ `LauncherMain.class`) that has the same contents as the `log4j.properties` that is not accessible / readable / present.
Diffs
-----
sharelib/distcp/src/main/java/org/apache/oozie/action/hadoop/DistcpMain.java fcaadefd215368565fb126ca92515792e1f1a09b
sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java bc8ab29c25405806f40232d32254ba2a7f5271e4
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java 7ae48e168a310b4d55feda7cbce8b2d29c2ec505
sharelib/oozie/src/main/resources/auxiliary-log4j.properties PRE-CREATION
sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherMain.java PRE-CREATION
sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java ba393eee53e92d144159f83893d4baab00737a44
sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMainWithOldAPI.java 14ab65e7d00c426036ced3e80400c80deea0e282
sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java c24d95c88be0dbf6fe80241d887a70100a5e1b4a
sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java cb8f56e9b66de0eb2799e68bc56d0ee36d4ba317
Diff: https://reviews.apache.org/r/58171/diff/1/
Testing
-------
Relevant unit tests run.
Thanks,
Andr�s Piros
Re: Review Request 58171: OOZIE-2844 Increase stability of Oozie
actions when log4j.properties is missing or not readable
Posted by András Piros <an...@cloudera.com>.
> On April 4, 2017, 11:31 a.m., Peter Bacsko wrote:
> > sharelib/oozie/src/main/resources/auxiliary-log4j.properties
> > Lines 1 (patched)
> > <https://reviews.apache.org/r/58171/diff/1/?file=1684286#file1684286line1>
> >
> > Where does the contents of this file come from?
Taken mostly from Hadoop's [`default-log4.properties`](https://github.com/apache/hadoop/blob/f67237cbe7bc48a1b9088e990800b37529f1db2a/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/resources/default-log4j.properties).
- Andr�s
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58171/#review170968
-----------------------------------------------------------
On April 4, 2017, 10:37 a.m., Andr�s Piros wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58171/
> -----------------------------------------------------------
>
> (Updated April 4, 2017, 10:37 a.m.)
>
>
> Review request for oozie and Peter Bacsko.
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> In several Oozie actions we face the problem that if `log4j.properties` is not readable by current thread, or not present on the classpath, while taking an `URL.openStream()` call afterwards, an `IOException` is thrown and the action won't run.
>
> Such actions are:
>
> * [`DistcpMain`](https://github.com/apache/oozie/blob/master/sharelib/distcp/src/main/java/org/apache/oozie/action/hadoop/DistcpMain.java#L135)
> * [`HiveMain`](https://github.com/apache/oozie/blob/master/sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java#L146)
> * [`PigMain`](https://github.com/apache/oozie/blob/master/sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java#L173)
> * [`PigMainWithOldApi`](https://github.com/apache/oozie/blob/master/sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMainWithOldAPI.java#L160)
> * [`SparkMain`](https://github.com/apache/oozie/blob/master/sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java#L430)
> * [`SqoopMain`](https://github.com/apache/oozie/blob/master/sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java#L124)
>
> Extend [`LauncherMain.run()`](https://github.com/apache/oozie/blob/master/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java#L62-L65) by a convenience method that loads Hadoop's `log4j.properties` in beforehand. In case `URL.openStream()` would throw `IOException` we resort to a call to a new file `hadoop-default-log4j.properties` on the classpath (packaged along w/ `LauncherMain.class`) that has the same contents as the `log4j.properties` that is not accessible / readable / present.
>
>
> Diffs
> -----
>
> sharelib/distcp/src/main/java/org/apache/oozie/action/hadoop/DistcpMain.java fcaadefd215368565fb126ca92515792e1f1a09b
> sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java bc8ab29c25405806f40232d32254ba2a7f5271e4
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java 7ae48e168a310b4d55feda7cbce8b2d29c2ec505
> sharelib/oozie/src/main/resources/auxiliary-log4j.properties PRE-CREATION
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherMain.java PRE-CREATION
> sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java ba393eee53e92d144159f83893d4baab00737a44
> sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMainWithOldAPI.java 14ab65e7d00c426036ced3e80400c80deea0e282
> sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java c24d95c88be0dbf6fe80241d887a70100a5e1b4a
> sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java cb8f56e9b66de0eb2799e68bc56d0ee36d4ba317
>
>
> Diff: https://reviews.apache.org/r/58171/diff/1/
>
>
> Testing
> -------
>
> Relevant unit tests run.
>
>
> Thanks,
>
> Andr�s Piros
>
>
Re: Review Request 58171: OOZIE-2844 Increase stability of Oozie
actions when log4j.properties is missing or not readable
Posted by Peter Bacsko <pb...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58171/#review170968
-----------------------------------------------------------
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java
Line 31 (original), 31 (patched)
<https://reviews.apache.org/r/58171/#comment243806>
Please avoid "*" imports
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java
Lines 59 (patched)
<https://reviews.apache.org/r/58171/#comment243807>
Could you please add a comment here about what this properties actually is and why is it used?
Also, for me it's not evident what "auxiliary" means in this context. What about a name like "log4j-default.properties" ?
sharelib/oozie/src/main/resources/auxiliary-log4j.properties
Lines 1 (patched)
<https://reviews.apache.org/r/58171/#comment243808>
Where does the contents of this file come from?
- Peter Bacsko
On �pr. 4, 2017, 10:37 de, Andr�s Piros wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58171/
> -----------------------------------------------------------
>
> (Updated �pr. 4, 2017, 10:37 de)
>
>
> Review request for oozie and Peter Bacsko.
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> In several Oozie actions we face the problem that if `log4j.properties` is not readable by current thread, or not present on the classpath, while taking an `URL.openStream()` call afterwards, an `IOException` is thrown and the action won't run.
>
> Such actions are:
>
> * [`DistcpMain`](https://github.com/apache/oozie/blob/master/sharelib/distcp/src/main/java/org/apache/oozie/action/hadoop/DistcpMain.java#L135)
> * [`HiveMain`](https://github.com/apache/oozie/blob/master/sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java#L146)
> * [`PigMain`](https://github.com/apache/oozie/blob/master/sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java#L173)
> * [`PigMainWithOldApi`](https://github.com/apache/oozie/blob/master/sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMainWithOldAPI.java#L160)
> * [`SparkMain`](https://github.com/apache/oozie/blob/master/sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java#L430)
> * [`SqoopMain`](https://github.com/apache/oozie/blob/master/sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java#L124)
>
> Extend [`LauncherMain.run()`](https://github.com/apache/oozie/blob/master/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java#L62-L65) by a convenience method that loads Hadoop's `log4j.properties` in beforehand. In case `URL.openStream()` would throw `IOException` we resort to a call to a new file `hadoop-default-log4j.properties` on the classpath (packaged along w/ `LauncherMain.class`) that has the same contents as the `log4j.properties` that is not accessible / readable / present.
>
>
> Diffs
> -----
>
> sharelib/distcp/src/main/java/org/apache/oozie/action/hadoop/DistcpMain.java fcaadefd215368565fb126ca92515792e1f1a09b
> sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java bc8ab29c25405806f40232d32254ba2a7f5271e4
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java 7ae48e168a310b4d55feda7cbce8b2d29c2ec505
> sharelib/oozie/src/main/resources/auxiliary-log4j.properties PRE-CREATION
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherMain.java PRE-CREATION
> sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java ba393eee53e92d144159f83893d4baab00737a44
> sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMainWithOldAPI.java 14ab65e7d00c426036ced3e80400c80deea0e282
> sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java c24d95c88be0dbf6fe80241d887a70100a5e1b4a
> sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java cb8f56e9b66de0eb2799e68bc56d0ee36d4ba317
>
>
> Diff: https://reviews.apache.org/r/58171/diff/1/
>
>
> Testing
> -------
>
> Relevant unit tests run.
>
>
> Thanks,
>
> Andr�s Piros
>
>
Re: Review Request 58171: OOZIE-2844 Increase stability of Oozie
actions when log4j.properties is missing or not readable
Posted by Peter Bacsko <pb...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58171/#review171205
-----------------------------------------------------------
Ship it!
Ship It!
- Peter Bacsko
On �pr. 5, 2017, 8:47 de, Andr�s Piros wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58171/
> -----------------------------------------------------------
>
> (Updated �pr. 5, 2017, 8:47 de)
>
>
> Review request for oozie and Peter Bacsko.
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> In several Oozie actions we face the problem that if `log4j.properties` is not readable by current thread, or not present on the classpath, while taking an `URL.openStream()` call afterwards, an `IOException` is thrown and the action won't run.
>
> Such actions are:
>
> * [`DistcpMain`](https://github.com/apache/oozie/blob/master/sharelib/distcp/src/main/java/org/apache/oozie/action/hadoop/DistcpMain.java#L135)
> * [`HiveMain`](https://github.com/apache/oozie/blob/master/sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java#L146)
> * [`PigMain`](https://github.com/apache/oozie/blob/master/sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java#L173)
> * [`PigMainWithOldApi`](https://github.com/apache/oozie/blob/master/sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMainWithOldAPI.java#L160)
> * [`SparkMain`](https://github.com/apache/oozie/blob/master/sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java#L430)
> * [`SqoopMain`](https://github.com/apache/oozie/blob/master/sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java#L124)
>
> Extend [`LauncherMain.run()`](https://github.com/apache/oozie/blob/master/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java#L62-L65) by a convenience method that loads Hadoop's `log4j.properties` in beforehand. In case `URL.openStream()` would throw `IOException` we resort to a call to a new file `hadoop-default-log4j.properties` on the classpath (packaged along w/ `LauncherMain.class`) that has the same contents as the `log4j.properties` that is not accessible / readable / present.
>
>
> Diffs
> -----
>
> sharelib/distcp/src/main/java/org/apache/oozie/action/hadoop/DistcpMain.java fcaadefd215368565fb126ca92515792e1f1a09b
> sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java f314bab3808c41f2576f8f64d9fdf8f34e578042
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java 7ae48e168a310b4d55feda7cbce8b2d29c2ec505
> sharelib/oozie/src/main/resources/default-log4j.properties PRE-CREATION
> sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherMain.java PRE-CREATION
> sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java ba393eee53e92d144159f83893d4baab00737a44
> sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMainWithOldAPI.java 14ab65e7d00c426036ced3e80400c80deea0e282
> sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java 88de5c4c614a5cc33c1f4760d2510928ba4b8ede
> sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java cb8f56e9b66de0eb2799e68bc56d0ee36d4ba317
>
>
> Diff: https://reviews.apache.org/r/58171/diff/2/
>
>
> Testing
> -------
>
> Relevant unit tests run.
>
>
> Thanks,
>
> Andr�s Piros
>
>
Re: Review Request 58171: OOZIE-2844 Increase stability of Oozie
actions when log4j.properties is missing or not readable
Posted by András Piros <an...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58171/
-----------------------------------------------------------
(Updated April 5, 2017, 8:47 a.m.)
Review request for oozie and Peter Bacsko.
Changes
-------
Addressing review comments.
Repository: oozie-git
Description
-------
In several Oozie actions we face the problem that if `log4j.properties` is not readable by current thread, or not present on the classpath, while taking an `URL.openStream()` call afterwards, an `IOException` is thrown and the action won't run.
Such actions are:
* [`DistcpMain`](https://github.com/apache/oozie/blob/master/sharelib/distcp/src/main/java/org/apache/oozie/action/hadoop/DistcpMain.java#L135)
* [`HiveMain`](https://github.com/apache/oozie/blob/master/sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java#L146)
* [`PigMain`](https://github.com/apache/oozie/blob/master/sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java#L173)
* [`PigMainWithOldApi`](https://github.com/apache/oozie/blob/master/sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMainWithOldAPI.java#L160)
* [`SparkMain`](https://github.com/apache/oozie/blob/master/sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java#L430)
* [`SqoopMain`](https://github.com/apache/oozie/blob/master/sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java#L124)
Extend [`LauncherMain.run()`](https://github.com/apache/oozie/blob/master/sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java#L62-L65) by a convenience method that loads Hadoop's `log4j.properties` in beforehand. In case `URL.openStream()` would throw `IOException` we resort to a call to a new file `hadoop-default-log4j.properties` on the classpath (packaged along w/ `LauncherMain.class`) that has the same contents as the `log4j.properties` that is not accessible / readable / present.
Diffs (updated)
-----
sharelib/distcp/src/main/java/org/apache/oozie/action/hadoop/DistcpMain.java fcaadefd215368565fb126ca92515792e1f1a09b
sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java f314bab3808c41f2576f8f64d9fdf8f34e578042
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java 7ae48e168a310b4d55feda7cbce8b2d29c2ec505
sharelib/oozie/src/main/resources/default-log4j.properties PRE-CREATION
sharelib/oozie/src/test/java/org/apache/oozie/action/hadoop/TestLauncherMain.java PRE-CREATION
sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMain.java ba393eee53e92d144159f83893d4baab00737a44
sharelib/pig/src/main/java/org/apache/oozie/action/hadoop/PigMainWithOldAPI.java 14ab65e7d00c426036ced3e80400c80deea0e282
sharelib/spark/src/main/java/org/apache/oozie/action/hadoop/SparkMain.java 88de5c4c614a5cc33c1f4760d2510928ba4b8ede
sharelib/sqoop/src/main/java/org/apache/oozie/action/hadoop/SqoopMain.java cb8f56e9b66de0eb2799e68bc56d0ee36d4ba317
Diff: https://reviews.apache.org/r/58171/diff/2/
Changes: https://reviews.apache.org/r/58171/diff/1-2/
Testing
-------
Relevant unit tests run.
Thanks,
Andr�s Piros