You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by Purshotam Shah <pu...@yahoo-inc.com> on 2016/01/08 02:46:50 UTC
Re: Review Request 38474: OOZIE-1976- Specifying coordinator input
datasets in more logical ways
> On Oct. 9, 2015, 12:53 a.m., Robert Kanter wrote:
> > client/src/main/resources/oozie-coordinator-0.5.xsd, line 110
> > <https://reviews.apache.org/r/38474/diff/1/?file=1076234#file1076234line110>
> >
> > Is there any way to enforce a max depth on nested "and"s and "or"s via the schema?
Not sure. We are not enforcing anything. if its needed we can do it in future.
> On Oct. 9, 2015, 12:53 a.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/coord/dependency/CoordDependenciesInputCheck.java, line 75
> > <https://reviews.apache.org/r/38474/diff/1/?file=1076246#file1076246line75>
> >
> > Is there anyway to reuse this without always creating a new one everywhere? (is it not thread safe?)
Haven't checked that, but OozieJexlEngine is very simple.
public class OozieJexlEngine extends JexlEngine {
OozieJexlInterpreter oozieInterpreter;
public OozieJexlEngine() {
}
- Purshotam
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38474/#review101982
-----------------------------------------------------------
On Dec. 15, 2015, 7:33 p.m., Purshotam Shah wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38474/
> -----------------------------------------------------------
>
> (Updated Dec. 15, 2015, 7:33 p.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-1976
> https://issues.apache.org/jira/browse/OOZIE-1976
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> There are three components in this patch
>
> 1. User interface
> A new tag is added to coordinator.xml
> ex.
> <input-check>
> <or name="test">
> <and>
> <data-in dataset="A"/>"
> <data-in dataset="B"/>
> </and>
> <and>
> <data-in dataset="C"/>
> <data-in dataset="D"/>
> </and>
>
> </or>;
> <input-check>
>
>
> input-check will have nested and/or/combine operation. It can have min and wait at operator or at date-in.
> If input-check tag is missing then it consider to be old approach where all data dependency are needed.
>
> 2. Processing
> input-check is converted into logical expression
> (a&&B)||(c&&d)
> We use jexl to parse the logical expression.
>
> There are three phase in parsing.
> phase 1 : only resolved dataset are parsed ( only current).
> phase 2 : once all current are resolved, then future/latest are parsed.
> phase 3 : Doesn't do any filecheck, just return what is being parsed by phase1 and phase2. Is used for EL functions
>
>
> 3. Storage.
> if inputcheck is enable, push_missing_dependencies and missing_dependencies are serialized and stored in DB.
> If then not then it's old approach, where they are stored in plan text. This is backward compatible.
>
>
> Diffs
> -----
>
> client/src/main/resources/oozie-coordinator-0.5.xsd e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
> core/pom.xml b063dab79415447a86c1a33f5c3f5304e0dffca0
> core/src/main/java/org/apache/oozie/CoordinatorActionBean.java 91bff4dca2ef2dece68ca7260724ba3e43b1a08e
> core/src/main/java/org/apache/oozie/ErrorCode.java 6c1e3997c9a1cb0bb0de39d687a083af0a7b5f04
> core/src/main/java/org/apache/oozie/command/coord/CoordActionInputCheckXCommand.java 742d00dd47aab55392abc8fbc207f8100728b832
> core/src/main/java/org/apache/oozie/command/coord/CoordActionUpdatePushMissingDependency.java 4e1c5b3392358cb6b1e98e16e469310338f27fed
> core/src/main/java/org/apache/oozie/command/coord/CoordCommandUtils.java 58ef483272264e0d391d4a1e6533dc1cab9940da
> core/src/main/java/org/apache/oozie/command/coord/CoordMaterializeTransitionXCommand.java 39e6ac15ce3a3ea7f2ed9178688537f7b1d7842d
> core/src/main/java/org/apache/oozie/command/coord/CoordPushDependencyCheckXCommand.java b05344d89e8df0e11fe69c1aa725d19a18eb0a2b
> core/src/main/java/org/apache/oozie/coord/CoordELConstants.java f010a817fc900821c0e429fc16e1d3902a98d8bb
> core/src/main/java/org/apache/oozie/coord/CoordELEvaluator.java 8b2f4560ae66bbcd707a446382e647663ea67be1
> core/src/main/java/org/apache/oozie/coord/CoordELFunctions.java 5d238663aa94f0dd55a9190b60bfe621439c7b53
> core/src/main/java/org/apache/oozie/coord/CoordUtils.java 94c69740618110ea180b188ab0c5a02db76f8b4d
> core/src/main/java/org/apache/oozie/coord/SyncCoordAction.java 44258eb5be40bf6769e32c8780117e8533d80d7e
> core/src/main/java/org/apache/oozie/coord/input/dependency/AbstractCoordInputDependency.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
> core/src/main/java/org/apache/oozie/coord/input/dependency/CoordInputDependenciesBuilder.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
> core/src/main/java/org/apache/oozie/coord/input/dependency/CoordInputDependenciesChecker.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
> core/src/main/java/org/apache/oozie/coord/input/dependency/CoordInputDependenciesUtil.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
> core/src/main/java/org/apache/oozie/coord/input/dependency/CoordInputDependency.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
> core/src/main/java/org/apache/oozie/coord/input/dependency/CoordInputDependencyCheck.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
> core/src/main/java/org/apache/oozie/coord/input/dependency/CoordInputDependencyCheckPhaseOne.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
> core/src/main/java/org/apache/oozie/coord/input/dependency/CoordInputDependencyCheckPhaseThree.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
> core/src/main/java/org/apache/oozie/coord/input/dependency/CoordInputDependencyCheckPhaseTwo.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
> core/src/main/java/org/apache/oozie/coord/input/dependency/CoordInputDependencyCheckPhaseValidate.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
> core/src/main/java/org/apache/oozie/coord/input/dependency/CoordInputDependencyFactory.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
> core/src/main/java/org/apache/oozie/coord/input/dependency/CoordInputInstance.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
> core/src/main/java/org/apache/oozie/coord/input/dependency/CoordOldInputDependency.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
> core/src/main/java/org/apache/oozie/coord/input/dependency/CoordPullInputDependency.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
> core/src/main/java/org/apache/oozie/coord/input/dependency/CoordPushInputDependency.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
> core/src/main/java/org/apache/oozie/coord/input/dependency/CoordUnResolvedInputDependency.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
> core/src/main/java/org/apache/oozie/coord/input/dependency/InputLogicParser.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
> core/src/main/java/org/apache/oozie/coord/input/dependency/OozieJexlEngine.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
> core/src/main/java/org/apache/oozie/coord/input/dependency/OozieJexlInterpreter.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
> core/src/main/java/org/apache/oozie/dependency/ActionDependency.java c280d1dc2387c9b3df96d4e83ad5563e10b1e1ef
> core/src/main/java/org/apache/oozie/dependency/DependencyChecker.java a5507575a3403103133f85a78c873c9976419857
> core/src/main/java/org/apache/oozie/util/WritableUtils.java 76a68953541125cab05bd05c94aa73b50e5363ff
> core/src/main/resources/oozie-default.xml faf37405757719554a5b6e3671d0c87723eb9959
> core/src/test/java/org/apache/oozie/command/coord/TestCoordActionInputCheckXCommand.java 1fe1b3adb5ceaa83985ef31278e49a3224cde0fb
> core/src/test/java/org/apache/oozie/command/coord/TestCoordCommandUtils.java 7062e697e5162ac25688d9cb62352bd139a7013a
> core/src/test/java/org/apache/oozie/command/coord/TestCoordMaterializeTransitionXCommand.java 29e7ca145612fadee661fe975aa0ea2cab1cbfa3
> core/src/test/java/org/apache/oozie/command/coord/TestCoordSubmitXCommand.java 52eb9dd1fb903ac86294a55d6e2a6918390bf4e9
> core/src/test/java/org/apache/oozie/coord/input/dependency/TestCoordInputCheckPush.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
> core/src/test/java/org/apache/oozie/coord/input/dependency/TestCoordinatorInputCheck.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
> core/src/test/java/org/apache/oozie/coord/input/dependency/TestInputLogicParser.java e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
> core/src/test/resources/coord-action-sla.xml f3f1bc09c51272a35d65edd7271599e7e8ba1ba4
> core/src/test/resources/coord-dataset-offset.xml 5dfbb1b40ee86dfc3336031eea3c5ff3144b9f5a
> core/src/test/resources/coord-inputcheck-combine.xml e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
> core/src/test/resources/coord-inputcheck-hcat.xml e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
> core/src/test/resources/coord-inputcheck-latest.xml e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
> core/src/test/resources/coord-inputcheck-range-latest.xml e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
> core/src/test/resources/coord-inputcheck-range.xml e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
> core/src/test/resources/coord-inputcheck.xml e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
> pom.xml a74ffab081defd15a1c5dbbf15355aa4e0455029
>
> Diff: https://reviews.apache.org/r/38474/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Purshotam Shah
>
>