You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by goun na <go...@gmail.com> on 2016/10/19 07:46:25 UTC
Oozie Job + Version Control Tool
Hi users,
What is the best to manage Oozie jobs? Is there a built-in version control
feature?
Best,
Goun Na
Re: Oozie Job + Version Control Tool
Posted by goun na <go...@gmail.com>.
Thanks for creating OOZIE-2876!
2017-05-31 4:26 GMT+09:00 Peter Cseh <ge...@cloudera.com>:
> I'd like to pour this conversation into
> https://issues.apache.org/jira/browse/OOZIE-2876 and it's subtasks. It
> looks like those action types are trying to solve these issues.
> Please share your thoughts on this approach there.
> Thanks!
>
> gp
>
>
> On Mon, Oct 31, 2016 at 8:12 AM, goun na <go...@gmail.com> wrote:
>
> > maven-assembly-plugin is good to use separate packaging. Thanks. :)
> >
> > 2016-10-20 15:05 GMT+09:00 Per Ullberg <pe...@klarna.com>:
> >
> > > Here's a pom, but I don't think it will tell you much.
> > >
> > >
> > > <?xml version="1.0" encoding="UTF-8"?>
> > > <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="
> > > http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="
> > > http://maven.apache.org/POM/4.0.0
> > > http://maven.apache.org/xsd/maven-4.0.0.xsd">
> > > <parent>
> > > <artifactId>hbase-to-hdfs-parent</artifactId>
> > > <groupId>com.klarna.datavault</groupId>
> > > <version>4.1.1-SNAPSHOT</version>
> > > </parent>
> > > <modelVersion>4.0.0</modelVersion>
> > >
> > > <artifactId>hbase-to-hdfs</artifactId>
> > >
> > > <dependencies>
> > >
> > > ...
> > >
> > > <dependency>
> > > <groupId>com.github.spullara.mustache.java</groupId>
> > > <artifactId>compiler</artifactId>
> > > <version>0.8.18</version>
> > > </dependency>
> > >
> > > <dependency>
> > > <groupId>joda-time</groupId>
> > > <artifactId>joda-time</artifactId>
> > > <version>2.8</version>
> > > <scope>test</scope>
> > > </dependency>
> > > <dependency>
> > > <groupId>com.klarna</groupId>
> > > <artifactId>hiverunner</artifactId>
> > > <version>2.6.0</version>
> > > <scope>test</scope>
> > > <exclusions>
> > > <exclusion>
> > > <artifactId>junit</artifactId>
> > > <groupId>junit</groupId>
> > > </exclusion>
> > > <exclusion>
> > > <groupId>org.apache.calcite</groupId>
> > > <artifactId>calcite-avatica</artifactId>
> > > </exclusion>
> > > <exclusion>
> > > <groupId>org.apache.calcite</groupId>
> > > <artifactId>calcite-core</artifactId>
> > > </exclusion>
> > > </exclusions>
> > > </dependency>
> > > </dependencies>
> > >
> > >
> > > <build>
> > > <resources>
> > > <resource>
> > > <directory>src/main/resources</directory>
> > > <filtering>true</filtering>
> > > </resource>
> > > <resource>
> > > <directory>src/main/templates</directory>
> > > <filtering>true</filtering>
> > > </resource>
> > > </resources>
> > >
> > > <plugins>
> > > <plugin>
> > > <groupId>org.apache.maven.plugins</groupId>
> > > <artifactId>maven-assembly-plugin</artifactId>
> > > <version>2.4</version>
> > > <configuration>
> > > <descriptor>assembly-descriptor.xml</descriptor>
> > > </configuration>
> > > <executions>
> > > <execution>
> > > <phase>package</phase>
> > > <goals>
> > > <goal>single</goal>
> > > </goals>
> > > </execution>
> > > </executions>
> > > </plugin>
> > > </plugins>
> > > </build>
> > >
> > > </project>
> > >
> > >
> > > We use mustache a lot to materialize coordinators, workflows and hql,
> > both
> > > buildtime and runtime. A nice thing we came up with is that some of our
> > > workflows have sub-workflows that gets materialized by the main
> workflow
> > at
> > > runtime. That way, different sub workflows may be generated dependent
> on
> > a
> > > runtime configuration. As an example we have sub workflows that load
> data
> > > from either to postgres or to kafka. These two ways of loading are
> > > described in two separate sub-workflow templates and are materialized
> at
> > > runtime dependent on if the user configured a postgres or a kafka load.
> > >
> > > This way we can build workflows that can be more reactive to the
> current
> > > state of the cluster and configuration.
> > >
> > > Here's an oozie snippet:
> > >
> > > <action name="materialize_load_sub_workflow" cred="hive_credentials">
> > > <java>
> > > <prepare>
> > > <mkdir path="${wf:appPath()}/load/"/>
> > > </prepare>
> > > <main-class>com.klarna.datavault.load.materialize.
> > > MaterializeLoadSubWorkflowMain</main-class>
> > > <!-- Output location for the materialized workflow XML -->
> > > <arg>-o</arg>
> > > <arg>${wf:appPath()}/load/load_sub_workflow-${wf:id()}.
> xml</arg>
> > > <!-- Output location for the materialized kafka.properties
> > > file for a Kafka Load -->
> > > <arg>-k</arg>
> > > <arg>${wf:appPath()}/load/kafka.properties</arg>
> > > <arg>-w</arg>
> > > <arg>${workflowConfigPath}</arg>
> > > <arg>-d</arg>
> > > <arg>${TARGET_DB_LOCATION}</arg>
> > > <arg>-s</arg>
> > > <arg>${wf:appPath()}/load/hql-delta-transform-${wf:id()}.
> > hql</arg>
> > > <arg>-n</arg>
> > > <arg>${TARGET_DB_NAME}</arg>
> > > <arg>-c</arg>
> > > <arg>${LOAD_DIFF_DB_LOCATION}</arg>
> > > <file>${HIVE_SITE_XML}</file>
> > > <capture-output/>
> > > </java>
> > > <ok to="execute_load"/>
> > > <error to="report_failure"/>
> > > </action>
> > >
> > >
> > > <action name="execute_load">
> > > <sub-workflow>
> > > <app-path>${wf:appPath()}/load/load_sub_workflow-${wf:
> > > id()}.xml</app-path>
> > > <propagate-configuration/>
> > > <configuration>
> > > <property>
> > > <name>jobName</name>
> > > <value>${jobName}-load</value>
> > > </property>
> > > <property>
> > > <name>parentWorkflowAppPath</name>
> > > <value>${wf:appPath()}</value>
> > > </property>
> > > <property>
> > > <name>kafkaConfigFilePath</name>
> > > <value>${wf:appPath()}/load/kafka.properties</value>
> > > </property>
> > > <property>
> > > <name>hqlDeltaTransformationPath</name>
> > > <value>load/hql-delta-transform-${wf:id()}.hql</value>
> > > </property>
> > > </configuration>
> > > </sub-workflow>
> > > <ok to="generate_view_script"/>
> > > <error to="report_failing_sub_wf"/>
> > > </action>
> > >
> > >
> > >
> > > Sorry if I'm blabbing
> > > /Pelle
> > >
> > > On Thu, Oct 20, 2016 at 3:52 AM, goun na <go...@gmail.com> wrote:
> > >
> > > > Per Ullberg, a snippet of pom.xml would help us. :)
> > > > Thanks,
> > > >
> > > > 2016-10-20 3:36 GMT+09:00 Per Ullberg <pe...@klarna.com>:
> > > >
> > > > > @goun na: we keep one coordinator per (zip|war|jar)
> > > > >
> > > > > @shiva: I'm happy to share, but it's hard to know what you're in
> need
> > > of.
> > > > > Ask and I will try to answer :)
> > > > >
> > > > > /Pelle
> > > > >
> > > > >
> > > > > On Wednesday, October 19, 2016, Shiva Ramagopal <tr.shiv@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > > Per,
> > > > > >
> > > > > > Your approach seems very interesting. Could you elaborate more on
> > > your
> > > > > > approach?
> > > > > >
> > > > > > Thanks,
> > > > > > Shiva
> > > > > >
> > > > > > On Wed, Oct 19, 2016 at 2:19 PM, Per Ullberg <
> > per.ullberg@klarna.com
> > > > > > <javascript:;>> wrote:
> > > > > >
> > > > > > > We package our oozie jobs with maven and release artifacts to
> > > nexus.
> > > > We
> > > > > > > keep the version number as part of the coordinator name. That
> way
> > > we
> > > > > have
> > > > > > > full traceability between code base and running coordinators.
> > > > > > >
> > > > > > > regards
> > > > > > > /Pelle
> > > > > > >
> > > > > > > On Wed, Oct 19, 2016 at 10:05 AM, Abhishek Bafna <
> > > > bafna.iitr@gmail.com
> > > > > > <javascript:;>>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > Oozie does not have version control for jobs. When you
> submit a
> > > > > > > > workflow/coordinator/bundle to oozie, it stores it into DB
> uses
> > > it
> > > > > from
> > > > > > > > there for further execution.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Abhishek
> > > > > > > > > On Oct 19, 2016, at 1:16 PM, goun na <gounna@gmail.com
> > > > > > <javascript:;>> wrote:
> > > > > > > > >
> > > > > > > > > Hi users,
> > > > > > > > >
> > > > > > > > > What is the best to manage Oozie jobs? Is there a built-in
> > > > version
> > > > > > > > control
> > > > > > > > > feature?
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Goun Na
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > > *Per Ullberg*
> > > > > > > Data Vault Tech Lead
> > > > > > > Odin Uppsala
> > > > > > > +46 701612693 <+46+701612693>
> > > > > > >
> > > > > > > Klarna AB (publ)
> > > > > > > Sveavägen 46, 111 34 Stockholm
> > > > > > > Tel: +46 8 120 120 00 <+46812012000>
> > > > > > > Reg no: 556737-0431
> > > > > > > klarna.com
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > *Per Ullberg*
> > > > > Data Vault Tech Lead
> > > > > Odin Uppsala
> > > > > +46 701612693 <+46+701612693>
> > > > >
> > > > > Klarna AB (publ)
> > > > > Sveavägen 46, 111 34 Stockholm
> > > > > Tel: +46 8 120 120 00 <+46812012000>
> > > > > Reg no: 556737-0431
> > > > > klarna.com
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > *Per Ullberg*
> > > Data Vault Tech Lead
> > > Odin Uppsala
> > > +46 701612693 <+46+701612693>
> > >
> > > Klarna AB (publ)
> > > Sveavägen 46, 111 34 Stockholm
> > > Tel: +46 8 120 120 00 <+46812012000>
> > > Reg no: 556737-0431
> > > klarna.com
> > >
> >
>
>
>
> --
> Peter Cseh
> Software Engineer
> <http://www.cloudera.com>
>
Re: Oozie Job + Version Control Tool
Posted by Peter Cseh <ge...@cloudera.com>.
I'd like to pour this conversation into
https://issues.apache.org/jira/browse/OOZIE-2876 and it's subtasks. It
looks like those action types are trying to solve these issues.
Please share your thoughts on this approach there.
Thanks!
gp
On Mon, Oct 31, 2016 at 8:12 AM, goun na <go...@gmail.com> wrote:
> maven-assembly-plugin is good to use separate packaging. Thanks. :)
>
> 2016-10-20 15:05 GMT+09:00 Per Ullberg <pe...@klarna.com>:
>
> > Here's a pom, but I don't think it will tell you much.
> >
> >
> > <?xml version="1.0" encoding="UTF-8"?>
> > <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="
> > http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="
> > http://maven.apache.org/POM/4.0.0
> > http://maven.apache.org/xsd/maven-4.0.0.xsd">
> > <parent>
> > <artifactId>hbase-to-hdfs-parent</artifactId>
> > <groupId>com.klarna.datavault</groupId>
> > <version>4.1.1-SNAPSHOT</version>
> > </parent>
> > <modelVersion>4.0.0</modelVersion>
> >
> > <artifactId>hbase-to-hdfs</artifactId>
> >
> > <dependencies>
> >
> > ...
> >
> > <dependency>
> > <groupId>com.github.spullara.mustache.java</groupId>
> > <artifactId>compiler</artifactId>
> > <version>0.8.18</version>
> > </dependency>
> >
> > <dependency>
> > <groupId>joda-time</groupId>
> > <artifactId>joda-time</artifactId>
> > <version>2.8</version>
> > <scope>test</scope>
> > </dependency>
> > <dependency>
> > <groupId>com.klarna</groupId>
> > <artifactId>hiverunner</artifactId>
> > <version>2.6.0</version>
> > <scope>test</scope>
> > <exclusions>
> > <exclusion>
> > <artifactId>junit</artifactId>
> > <groupId>junit</groupId>
> > </exclusion>
> > <exclusion>
> > <groupId>org.apache.calcite</groupId>
> > <artifactId>calcite-avatica</artifactId>
> > </exclusion>
> > <exclusion>
> > <groupId>org.apache.calcite</groupId>
> > <artifactId>calcite-core</artifactId>
> > </exclusion>
> > </exclusions>
> > </dependency>
> > </dependencies>
> >
> >
> > <build>
> > <resources>
> > <resource>
> > <directory>src/main/resources</directory>
> > <filtering>true</filtering>
> > </resource>
> > <resource>
> > <directory>src/main/templates</directory>
> > <filtering>true</filtering>
> > </resource>
> > </resources>
> >
> > <plugins>
> > <plugin>
> > <groupId>org.apache.maven.plugins</groupId>
> > <artifactId>maven-assembly-plugin</artifactId>
> > <version>2.4</version>
> > <configuration>
> > <descriptor>assembly-descriptor.xml</descriptor>
> > </configuration>
> > <executions>
> > <execution>
> > <phase>package</phase>
> > <goals>
> > <goal>single</goal>
> > </goals>
> > </execution>
> > </executions>
> > </plugin>
> > </plugins>
> > </build>
> >
> > </project>
> >
> >
> > We use mustache a lot to materialize coordinators, workflows and hql,
> both
> > buildtime and runtime. A nice thing we came up with is that some of our
> > workflows have sub-workflows that gets materialized by the main workflow
> at
> > runtime. That way, different sub workflows may be generated dependent on
> a
> > runtime configuration. As an example we have sub workflows that load data
> > from either to postgres or to kafka. These two ways of loading are
> > described in two separate sub-workflow templates and are materialized at
> > runtime dependent on if the user configured a postgres or a kafka load.
> >
> > This way we can build workflows that can be more reactive to the current
> > state of the cluster and configuration.
> >
> > Here's an oozie snippet:
> >
> > <action name="materialize_load_sub_workflow" cred="hive_credentials">
> > <java>
> > <prepare>
> > <mkdir path="${wf:appPath()}/load/"/>
> > </prepare>
> > <main-class>com.klarna.datavault.load.materialize.
> > MaterializeLoadSubWorkflowMain</main-class>
> > <!-- Output location for the materialized workflow XML -->
> > <arg>-o</arg>
> > <arg>${wf:appPath()}/load/load_sub_workflow-${wf:id()}.xml</arg>
> > <!-- Output location for the materialized kafka.properties
> > file for a Kafka Load -->
> > <arg>-k</arg>
> > <arg>${wf:appPath()}/load/kafka.properties</arg>
> > <arg>-w</arg>
> > <arg>${workflowConfigPath}</arg>
> > <arg>-d</arg>
> > <arg>${TARGET_DB_LOCATION}</arg>
> > <arg>-s</arg>
> > <arg>${wf:appPath()}/load/hql-delta-transform-${wf:id()}.
> hql</arg>
> > <arg>-n</arg>
> > <arg>${TARGET_DB_NAME}</arg>
> > <arg>-c</arg>
> > <arg>${LOAD_DIFF_DB_LOCATION}</arg>
> > <file>${HIVE_SITE_XML}</file>
> > <capture-output/>
> > </java>
> > <ok to="execute_load"/>
> > <error to="report_failure"/>
> > </action>
> >
> >
> > <action name="execute_load">
> > <sub-workflow>
> > <app-path>${wf:appPath()}/load/load_sub_workflow-${wf:
> > id()}.xml</app-path>
> > <propagate-configuration/>
> > <configuration>
> > <property>
> > <name>jobName</name>
> > <value>${jobName}-load</value>
> > </property>
> > <property>
> > <name>parentWorkflowAppPath</name>
> > <value>${wf:appPath()}</value>
> > </property>
> > <property>
> > <name>kafkaConfigFilePath</name>
> > <value>${wf:appPath()}/load/kafka.properties</value>
> > </property>
> > <property>
> > <name>hqlDeltaTransformationPath</name>
> > <value>load/hql-delta-transform-${wf:id()}.hql</value>
> > </property>
> > </configuration>
> > </sub-workflow>
> > <ok to="generate_view_script"/>
> > <error to="report_failing_sub_wf"/>
> > </action>
> >
> >
> >
> > Sorry if I'm blabbing
> > /Pelle
> >
> > On Thu, Oct 20, 2016 at 3:52 AM, goun na <go...@gmail.com> wrote:
> >
> > > Per Ullberg, a snippet of pom.xml would help us. :)
> > > Thanks,
> > >
> > > 2016-10-20 3:36 GMT+09:00 Per Ullberg <pe...@klarna.com>:
> > >
> > > > @goun na: we keep one coordinator per (zip|war|jar)
> > > >
> > > > @shiva: I'm happy to share, but it's hard to know what you're in need
> > of.
> > > > Ask and I will try to answer :)
> > > >
> > > > /Pelle
> > > >
> > > >
> > > > On Wednesday, October 19, 2016, Shiva Ramagopal <tr...@gmail.com>
> > > wrote:
> > > >
> > > > > Per,
> > > > >
> > > > > Your approach seems very interesting. Could you elaborate more on
> > your
> > > > > approach?
> > > > >
> > > > > Thanks,
> > > > > Shiva
> > > > >
> > > > > On Wed, Oct 19, 2016 at 2:19 PM, Per Ullberg <
> per.ullberg@klarna.com
> > > > > <javascript:;>> wrote:
> > > > >
> > > > > > We package our oozie jobs with maven and release artifacts to
> > nexus.
> > > We
> > > > > > keep the version number as part of the coordinator name. That way
> > we
> > > > have
> > > > > > full traceability between code base and running coordinators.
> > > > > >
> > > > > > regards
> > > > > > /Pelle
> > > > > >
> > > > > > On Wed, Oct 19, 2016 at 10:05 AM, Abhishek Bafna <
> > > bafna.iitr@gmail.com
> > > > > <javascript:;>>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > Oozie does not have version control for jobs. When you submit a
> > > > > > > workflow/coordinator/bundle to oozie, it stores it into DB uses
> > it
> > > > from
> > > > > > > there for further execution.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Abhishek
> > > > > > > > On Oct 19, 2016, at 1:16 PM, goun na <gounna@gmail.com
> > > > > <javascript:;>> wrote:
> > > > > > > >
> > > > > > > > Hi users,
> > > > > > > >
> > > > > > > > What is the best to manage Oozie jobs? Is there a built-in
> > > version
> > > > > > > control
> > > > > > > > feature?
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Goun Na
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > *Per Ullberg*
> > > > > > Data Vault Tech Lead
> > > > > > Odin Uppsala
> > > > > > +46 701612693 <+46+701612693>
> > > > > >
> > > > > > Klarna AB (publ)
> > > > > > Sveavägen 46, 111 34 Stockholm
> > > > > > Tel: +46 8 120 120 00 <+46812012000>
> > > > > > Reg no: 556737-0431
> > > > > > klarna.com
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > >
> > > > *Per Ullberg*
> > > > Data Vault Tech Lead
> > > > Odin Uppsala
> > > > +46 701612693 <+46+701612693>
> > > >
> > > > Klarna AB (publ)
> > > > Sveavägen 46, 111 34 Stockholm
> > > > Tel: +46 8 120 120 00 <+46812012000>
> > > > Reg no: 556737-0431
> > > > klarna.com
> > > >
> > >
> >
> >
> >
> > --
> >
> > *Per Ullberg*
> > Data Vault Tech Lead
> > Odin Uppsala
> > +46 701612693 <+46+701612693>
> >
> > Klarna AB (publ)
> > Sveavägen 46, 111 34 Stockholm
> > Tel: +46 8 120 120 00 <+46812012000>
> > Reg no: 556737-0431
> > klarna.com
> >
>
--
Peter Cseh
Software Engineer
<http://www.cloudera.com>
Re: Oozie Job + Version Control Tool
Posted by goun na <go...@gmail.com>.
maven-assembly-plugin is good to use separate packaging. Thanks. :)
2016-10-20 15:05 GMT+09:00 Per Ullberg <pe...@klarna.com>:
> Here's a pom, but I don't think it will tell you much.
>
>
> <?xml version="1.0" encoding="UTF-8"?>
> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="
> http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="
> http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/xsd/maven-4.0.0.xsd">
> <parent>
> <artifactId>hbase-to-hdfs-parent</artifactId>
> <groupId>com.klarna.datavault</groupId>
> <version>4.1.1-SNAPSHOT</version>
> </parent>
> <modelVersion>4.0.0</modelVersion>
>
> <artifactId>hbase-to-hdfs</artifactId>
>
> <dependencies>
>
> ...
>
> <dependency>
> <groupId>com.github.spullara.mustache.java</groupId>
> <artifactId>compiler</artifactId>
> <version>0.8.18</version>
> </dependency>
>
> <dependency>
> <groupId>joda-time</groupId>
> <artifactId>joda-time</artifactId>
> <version>2.8</version>
> <scope>test</scope>
> </dependency>
> <dependency>
> <groupId>com.klarna</groupId>
> <artifactId>hiverunner</artifactId>
> <version>2.6.0</version>
> <scope>test</scope>
> <exclusions>
> <exclusion>
> <artifactId>junit</artifactId>
> <groupId>junit</groupId>
> </exclusion>
> <exclusion>
> <groupId>org.apache.calcite</groupId>
> <artifactId>calcite-avatica</artifactId>
> </exclusion>
> <exclusion>
> <groupId>org.apache.calcite</groupId>
> <artifactId>calcite-core</artifactId>
> </exclusion>
> </exclusions>
> </dependency>
> </dependencies>
>
>
> <build>
> <resources>
> <resource>
> <directory>src/main/resources</directory>
> <filtering>true</filtering>
> </resource>
> <resource>
> <directory>src/main/templates</directory>
> <filtering>true</filtering>
> </resource>
> </resources>
>
> <plugins>
> <plugin>
> <groupId>org.apache.maven.plugins</groupId>
> <artifactId>maven-assembly-plugin</artifactId>
> <version>2.4</version>
> <configuration>
> <descriptor>assembly-descriptor.xml</descriptor>
> </configuration>
> <executions>
> <execution>
> <phase>package</phase>
> <goals>
> <goal>single</goal>
> </goals>
> </execution>
> </executions>
> </plugin>
> </plugins>
> </build>
>
> </project>
>
>
> We use mustache a lot to materialize coordinators, workflows and hql, both
> buildtime and runtime. A nice thing we came up with is that some of our
> workflows have sub-workflows that gets materialized by the main workflow at
> runtime. That way, different sub workflows may be generated dependent on a
> runtime configuration. As an example we have sub workflows that load data
> from either to postgres or to kafka. These two ways of loading are
> described in two separate sub-workflow templates and are materialized at
> runtime dependent on if the user configured a postgres or a kafka load.
>
> This way we can build workflows that can be more reactive to the current
> state of the cluster and configuration.
>
> Here's an oozie snippet:
>
> <action name="materialize_load_sub_workflow" cred="hive_credentials">
> <java>
> <prepare>
> <mkdir path="${wf:appPath()}/load/"/>
> </prepare>
> <main-class>com.klarna.datavault.load.materialize.
> MaterializeLoadSubWorkflowMain</main-class>
> <!-- Output location for the materialized workflow XML -->
> <arg>-o</arg>
> <arg>${wf:appPath()}/load/load_sub_workflow-${wf:id()}.xml</arg>
> <!-- Output location for the materialized kafka.properties
> file for a Kafka Load -->
> <arg>-k</arg>
> <arg>${wf:appPath()}/load/kafka.properties</arg>
> <arg>-w</arg>
> <arg>${workflowConfigPath}</arg>
> <arg>-d</arg>
> <arg>${TARGET_DB_LOCATION}</arg>
> <arg>-s</arg>
> <arg>${wf:appPath()}/load/hql-delta-transform-${wf:id()}.hql</arg>
> <arg>-n</arg>
> <arg>${TARGET_DB_NAME}</arg>
> <arg>-c</arg>
> <arg>${LOAD_DIFF_DB_LOCATION}</arg>
> <file>${HIVE_SITE_XML}</file>
> <capture-output/>
> </java>
> <ok to="execute_load"/>
> <error to="report_failure"/>
> </action>
>
>
> <action name="execute_load">
> <sub-workflow>
> <app-path>${wf:appPath()}/load/load_sub_workflow-${wf:
> id()}.xml</app-path>
> <propagate-configuration/>
> <configuration>
> <property>
> <name>jobName</name>
> <value>${jobName}-load</value>
> </property>
> <property>
> <name>parentWorkflowAppPath</name>
> <value>${wf:appPath()}</value>
> </property>
> <property>
> <name>kafkaConfigFilePath</name>
> <value>${wf:appPath()}/load/kafka.properties</value>
> </property>
> <property>
> <name>hqlDeltaTransformationPath</name>
> <value>load/hql-delta-transform-${wf:id()}.hql</value>
> </property>
> </configuration>
> </sub-workflow>
> <ok to="generate_view_script"/>
> <error to="report_failing_sub_wf"/>
> </action>
>
>
>
> Sorry if I'm blabbing
> /Pelle
>
> On Thu, Oct 20, 2016 at 3:52 AM, goun na <go...@gmail.com> wrote:
>
> > Per Ullberg, a snippet of pom.xml would help us. :)
> > Thanks,
> >
> > 2016-10-20 3:36 GMT+09:00 Per Ullberg <pe...@klarna.com>:
> >
> > > @goun na: we keep one coordinator per (zip|war|jar)
> > >
> > > @shiva: I'm happy to share, but it's hard to know what you're in need
> of.
> > > Ask and I will try to answer :)
> > >
> > > /Pelle
> > >
> > >
> > > On Wednesday, October 19, 2016, Shiva Ramagopal <tr...@gmail.com>
> > wrote:
> > >
> > > > Per,
> > > >
> > > > Your approach seems very interesting. Could you elaborate more on
> your
> > > > approach?
> > > >
> > > > Thanks,
> > > > Shiva
> > > >
> > > > On Wed, Oct 19, 2016 at 2:19 PM, Per Ullberg <per.ullberg@klarna.com
> > > > <javascript:;>> wrote:
> > > >
> > > > > We package our oozie jobs with maven and release artifacts to
> nexus.
> > We
> > > > > keep the version number as part of the coordinator name. That way
> we
> > > have
> > > > > full traceability between code base and running coordinators.
> > > > >
> > > > > regards
> > > > > /Pelle
> > > > >
> > > > > On Wed, Oct 19, 2016 at 10:05 AM, Abhishek Bafna <
> > bafna.iitr@gmail.com
> > > > <javascript:;>>
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > Oozie does not have version control for jobs. When you submit a
> > > > > > workflow/coordinator/bundle to oozie, it stores it into DB uses
> it
> > > from
> > > > > > there for further execution.
> > > > > >
> > > > > > Thanks,
> > > > > > Abhishek
> > > > > > > On Oct 19, 2016, at 1:16 PM, goun na <gounna@gmail.com
> > > > <javascript:;>> wrote:
> > > > > > >
> > > > > > > Hi users,
> > > > > > >
> > > > > > > What is the best to manage Oozie jobs? Is there a built-in
> > version
> > > > > > control
> > > > > > > feature?
> > > > > > >
> > > > > > > Best,
> > > > > > > Goun Na
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > *Per Ullberg*
> > > > > Data Vault Tech Lead
> > > > > Odin Uppsala
> > > > > +46 701612693 <+46+701612693>
> > > > >
> > > > > Klarna AB (publ)
> > > > > Sveavägen 46, 111 34 Stockholm
> > > > > Tel: +46 8 120 120 00 <+46812012000>
> > > > > Reg no: 556737-0431
> > > > > klarna.com
> > > > >
> > > >
> > >
> > >
> > > --
> > >
> > > *Per Ullberg*
> > > Data Vault Tech Lead
> > > Odin Uppsala
> > > +46 701612693 <+46+701612693>
> > >
> > > Klarna AB (publ)
> > > Sveavägen 46, 111 34 Stockholm
> > > Tel: +46 8 120 120 00 <+46812012000>
> > > Reg no: 556737-0431
> > > klarna.com
> > >
> >
>
>
>
> --
>
> *Per Ullberg*
> Data Vault Tech Lead
> Odin Uppsala
> +46 701612693 <+46+701612693>
>
> Klarna AB (publ)
> Sveavägen 46, 111 34 Stockholm
> Tel: +46 8 120 120 00 <+46812012000>
> Reg no: 556737-0431
> klarna.com
>
Re: Oozie Job + Version Control Tool
Posted by Per Ullberg <pe...@klarna.com>.
Here's a pom, but I don't think it will tell you much.
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="
http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="
http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd">
<parent>
<artifactId>hbase-to-hdfs-parent</artifactId>
<groupId>com.klarna.datavault</groupId>
<version>4.1.1-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>hbase-to-hdfs</artifactId>
<dependencies>
...
<dependency>
<groupId>com.github.spullara.mustache.java</groupId>
<artifactId>compiler</artifactId>
<version>0.8.18</version>
</dependency>
<dependency>
<groupId>joda-time</groupId>
<artifactId>joda-time</artifactId>
<version>2.8</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.klarna</groupId>
<artifactId>hiverunner</artifactId>
<version>2.6.0</version>
<scope>test</scope>
<exclusions>
<exclusion>
<artifactId>junit</artifactId>
<groupId>junit</groupId>
</exclusion>
<exclusion>
<groupId>org.apache.calcite</groupId>
<artifactId>calcite-avatica</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.calcite</groupId>
<artifactId>calcite-core</artifactId>
</exclusion>
</exclusions>
</dependency>
</dependencies>
<build>
<resources>
<resource>
<directory>src/main/resources</directory>
<filtering>true</filtering>
</resource>
<resource>
<directory>src/main/templates</directory>
<filtering>true</filtering>
</resource>
</resources>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.4</version>
<configuration>
<descriptor>assembly-descriptor.xml</descriptor>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
We use mustache a lot to materialize coordinators, workflows and hql, both
buildtime and runtime. A nice thing we came up with is that some of our
workflows have sub-workflows that gets materialized by the main workflow at
runtime. That way, different sub workflows may be generated dependent on a
runtime configuration. As an example we have sub workflows that load data
from either to postgres or to kafka. These two ways of loading are
described in two separate sub-workflow templates and are materialized at
runtime dependent on if the user configured a postgres or a kafka load.
This way we can build workflows that can be more reactive to the current
state of the cluster and configuration.
Here's an oozie snippet:
<action name="materialize_load_sub_workflow" cred="hive_credentials">
<java>
<prepare>
<mkdir path="${wf:appPath()}/load/"/>
</prepare>
<main-class>com.klarna.datavault.load.materialize.MaterializeLoadSubWorkflowMain</main-class>
<!-- Output location for the materialized workflow XML -->
<arg>-o</arg>
<arg>${wf:appPath()}/load/load_sub_workflow-${wf:id()}.xml</arg>
<!-- Output location for the materialized kafka.properties
file for a Kafka Load -->
<arg>-k</arg>
<arg>${wf:appPath()}/load/kafka.properties</arg>
<arg>-w</arg>
<arg>${workflowConfigPath}</arg>
<arg>-d</arg>
<arg>${TARGET_DB_LOCATION}</arg>
<arg>-s</arg>
<arg>${wf:appPath()}/load/hql-delta-transform-${wf:id()}.hql</arg>
<arg>-n</arg>
<arg>${TARGET_DB_NAME}</arg>
<arg>-c</arg>
<arg>${LOAD_DIFF_DB_LOCATION}</arg>
<file>${HIVE_SITE_XML}</file>
<capture-output/>
</java>
<ok to="execute_load"/>
<error to="report_failure"/>
</action>
<action name="execute_load">
<sub-workflow>
<app-path>${wf:appPath()}/load/load_sub_workflow-${wf:id()}.xml</app-path>
<propagate-configuration/>
<configuration>
<property>
<name>jobName</name>
<value>${jobName}-load</value>
</property>
<property>
<name>parentWorkflowAppPath</name>
<value>${wf:appPath()}</value>
</property>
<property>
<name>kafkaConfigFilePath</name>
<value>${wf:appPath()}/load/kafka.properties</value>
</property>
<property>
<name>hqlDeltaTransformationPath</name>
<value>load/hql-delta-transform-${wf:id()}.hql</value>
</property>
</configuration>
</sub-workflow>
<ok to="generate_view_script"/>
<error to="report_failing_sub_wf"/>
</action>
Sorry if I'm blabbing
/Pelle
On Thu, Oct 20, 2016 at 3:52 AM, goun na <go...@gmail.com> wrote:
> Per Ullberg, a snippet of pom.xml would help us. :)
> Thanks,
>
> 2016-10-20 3:36 GMT+09:00 Per Ullberg <pe...@klarna.com>:
>
> > @goun na: we keep one coordinator per (zip|war|jar)
> >
> > @shiva: I'm happy to share, but it's hard to know what you're in need of.
> > Ask and I will try to answer :)
> >
> > /Pelle
> >
> >
> > On Wednesday, October 19, 2016, Shiva Ramagopal <tr...@gmail.com>
> wrote:
> >
> > > Per,
> > >
> > > Your approach seems very interesting. Could you elaborate more on your
> > > approach?
> > >
> > > Thanks,
> > > Shiva
> > >
> > > On Wed, Oct 19, 2016 at 2:19 PM, Per Ullberg <per.ullberg@klarna.com
> > > <javascript:;>> wrote:
> > >
> > > > We package our oozie jobs with maven and release artifacts to nexus.
> We
> > > > keep the version number as part of the coordinator name. That way we
> > have
> > > > full traceability between code base and running coordinators.
> > > >
> > > > regards
> > > > /Pelle
> > > >
> > > > On Wed, Oct 19, 2016 at 10:05 AM, Abhishek Bafna <
> bafna.iitr@gmail.com
> > > <javascript:;>>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > Oozie does not have version control for jobs. When you submit a
> > > > > workflow/coordinator/bundle to oozie, it stores it into DB uses it
> > from
> > > > > there for further execution.
> > > > >
> > > > > Thanks,
> > > > > Abhishek
> > > > > > On Oct 19, 2016, at 1:16 PM, goun na <gounna@gmail.com
> > > <javascript:;>> wrote:
> > > > > >
> > > > > > Hi users,
> > > > > >
> > > > > > What is the best to manage Oozie jobs? Is there a built-in
> version
> > > > > control
> > > > > > feature?
> > > > > >
> > > > > > Best,
> > > > > > Goun Na
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > >
> > > > *Per Ullberg*
> > > > Data Vault Tech Lead
> > > > Odin Uppsala
> > > > +46 701612693 <+46+701612693>
> > > >
> > > > Klarna AB (publ)
> > > > Sveavägen 46, 111 34 Stockholm
> > > > Tel: +46 8 120 120 00 <+46812012000>
> > > > Reg no: 556737-0431
> > > > klarna.com
> > > >
> > >
> >
> >
> > --
> >
> > *Per Ullberg*
> > Data Vault Tech Lead
> > Odin Uppsala
> > +46 701612693 <+46+701612693>
> >
> > Klarna AB (publ)
> > Sveavägen 46, 111 34 Stockholm
> > Tel: +46 8 120 120 00 <+46812012000>
> > Reg no: 556737-0431
> > klarna.com
> >
>
--
*Per Ullberg*
Data Vault Tech Lead
Odin Uppsala
+46 701612693 <+46+701612693>
Klarna AB (publ)
Sveavägen 46, 111 34 Stockholm
Tel: +46 8 120 120 00 <+46812012000>
Reg no: 556737-0431
klarna.com
Re: Oozie Job + Version Control Tool
Posted by goun na <go...@gmail.com>.
Per Ullberg, a snippet of pom.xml would help us. :)
Thanks,
2016-10-20 3:36 GMT+09:00 Per Ullberg <pe...@klarna.com>:
> @goun na: we keep one coordinator per (zip|war|jar)
>
> @shiva: I'm happy to share, but it's hard to know what you're in need of.
> Ask and I will try to answer :)
>
> /Pelle
>
>
> On Wednesday, October 19, 2016, Shiva Ramagopal <tr...@gmail.com> wrote:
>
> > Per,
> >
> > Your approach seems very interesting. Could you elaborate more on your
> > approach?
> >
> > Thanks,
> > Shiva
> >
> > On Wed, Oct 19, 2016 at 2:19 PM, Per Ullberg <per.ullberg@klarna.com
> > <javascript:;>> wrote:
> >
> > > We package our oozie jobs with maven and release artifacts to nexus. We
> > > keep the version number as part of the coordinator name. That way we
> have
> > > full traceability between code base and running coordinators.
> > >
> > > regards
> > > /Pelle
> > >
> > > On Wed, Oct 19, 2016 at 10:05 AM, Abhishek Bafna <bafna.iitr@gmail.com
> > <javascript:;>>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > Oozie does not have version control for jobs. When you submit a
> > > > workflow/coordinator/bundle to oozie, it stores it into DB uses it
> from
> > > > there for further execution.
> > > >
> > > > Thanks,
> > > > Abhishek
> > > > > On Oct 19, 2016, at 1:16 PM, goun na <gounna@gmail.com
> > <javascript:;>> wrote:
> > > > >
> > > > > Hi users,
> > > > >
> > > > > What is the best to manage Oozie jobs? Is there a built-in version
> > > > control
> > > > > feature?
> > > > >
> > > > > Best,
> > > > > Goun Na
> > > >
> > > >
> > >
> > >
> > > --
> > >
> > > *Per Ullberg*
> > > Data Vault Tech Lead
> > > Odin Uppsala
> > > +46 701612693 <+46+701612693>
> > >
> > > Klarna AB (publ)
> > > Sveavägen 46, 111 34 Stockholm
> > > Tel: +46 8 120 120 00 <+46812012000>
> > > Reg no: 556737-0431
> > > klarna.com
> > >
> >
>
>
> --
>
> *Per Ullberg*
> Data Vault Tech Lead
> Odin Uppsala
> +46 701612693 <+46+701612693>
>
> Klarna AB (publ)
> Sveavägen 46, 111 34 Stockholm
> Tel: +46 8 120 120 00 <+46812012000>
> Reg no: 556737-0431
> klarna.com
>
Re: Oozie Job + Version Control Tool
Posted by Per Ullberg <pe...@klarna.com>.
@goun na: we keep one coordinator per (zip|war|jar)
@shiva: I'm happy to share, but it's hard to know what you're in need of.
Ask and I will try to answer :)
/Pelle
On Wednesday, October 19, 2016, Shiva Ramagopal <tr...@gmail.com> wrote:
> Per,
>
> Your approach seems very interesting. Could you elaborate more on your
> approach?
>
> Thanks,
> Shiva
>
> On Wed, Oct 19, 2016 at 2:19 PM, Per Ullberg <per.ullberg@klarna.com
> <javascript:;>> wrote:
>
> > We package our oozie jobs with maven and release artifacts to nexus. We
> > keep the version number as part of the coordinator name. That way we have
> > full traceability between code base and running coordinators.
> >
> > regards
> > /Pelle
> >
> > On Wed, Oct 19, 2016 at 10:05 AM, Abhishek Bafna <bafna.iitr@gmail.com
> <javascript:;>>
> > wrote:
> >
> > > Hi,
> > >
> > > Oozie does not have version control for jobs. When you submit a
> > > workflow/coordinator/bundle to oozie, it stores it into DB uses it from
> > > there for further execution.
> > >
> > > Thanks,
> > > Abhishek
> > > > On Oct 19, 2016, at 1:16 PM, goun na <gounna@gmail.com
> <javascript:;>> wrote:
> > > >
> > > > Hi users,
> > > >
> > > > What is the best to manage Oozie jobs? Is there a built-in version
> > > control
> > > > feature?
> > > >
> > > > Best,
> > > > Goun Na
> > >
> > >
> >
> >
> > --
> >
> > *Per Ullberg*
> > Data Vault Tech Lead
> > Odin Uppsala
> > +46 701612693 <+46+701612693>
> >
> > Klarna AB (publ)
> > Sveavägen 46, 111 34 Stockholm
> > Tel: +46 8 120 120 00 <+46812012000>
> > Reg no: 556737-0431
> > klarna.com
> >
>
--
*Per Ullberg*
Data Vault Tech Lead
Odin Uppsala
+46 701612693 <+46+701612693>
Klarna AB (publ)
Sveavägen 46, 111 34 Stockholm
Tel: +46 8 120 120 00 <+46812012000>
Reg no: 556737-0431
klarna.com
Re: Oozie Job + Version Control Tool
Posted by Shiva Ramagopal <tr...@gmail.com>.
Per,
Your approach seems very interesting. Could you elaborate more on your
approach?
Thanks,
Shiva
On Wed, Oct 19, 2016 at 2:19 PM, Per Ullberg <pe...@klarna.com> wrote:
> We package our oozie jobs with maven and release artifacts to nexus. We
> keep the version number as part of the coordinator name. That way we have
> full traceability between code base and running coordinators.
>
> regards
> /Pelle
>
> On Wed, Oct 19, 2016 at 10:05 AM, Abhishek Bafna <ba...@gmail.com>
> wrote:
>
> > Hi,
> >
> > Oozie does not have version control for jobs. When you submit a
> > workflow/coordinator/bundle to oozie, it stores it into DB uses it from
> > there for further execution.
> >
> > Thanks,
> > Abhishek
> > > On Oct 19, 2016, at 1:16 PM, goun na <go...@gmail.com> wrote:
> > >
> > > Hi users,
> > >
> > > What is the best to manage Oozie jobs? Is there a built-in version
> > control
> > > feature?
> > >
> > > Best,
> > > Goun Na
> >
> >
>
>
> --
>
> *Per Ullberg*
> Data Vault Tech Lead
> Odin Uppsala
> +46 701612693 <+46+701612693>
>
> Klarna AB (publ)
> Sveavägen 46, 111 34 Stockholm
> Tel: +46 8 120 120 00 <+46812012000>
> Reg no: 556737-0431
> klarna.com
>
Re: Oozie Job + Version Control Tool
Posted by goun na <go...@gmail.com>.
Thank you for good answers! Adding coordinator version name is also a good
idea that I've never thought of.
Per Ullberg, do you make a package per a coordinator or workflow? I am not
sure putting all coordinator/workflow in a jar/war file.
2016-10-19 17:49 GMT+09:00 Per Ullberg <pe...@klarna.com>:
> We package our oozie jobs with maven and release artifacts to nexus. We
> keep the version number as part of the coordinator name. That way we have
> full traceability between code base and running coordinators.
>
> regards
> /Pelle
>
> On Wed, Oct 19, 2016 at 10:05 AM, Abhishek Bafna <ba...@gmail.com>
> wrote:
>
> > Hi,
> >
> > Oozie does not have version control for jobs. When you submit a
> > workflow/coordinator/bundle to oozie, it stores it into DB uses it from
> > there for further execution.
> >
> > Thanks,
> > Abhishek
> > > On Oct 19, 2016, at 1:16 PM, goun na <go...@gmail.com> wrote:
> > >
> > > Hi users,
> > >
> > > What is the best to manage Oozie jobs? Is there a built-in version
> > control
> > > feature?
> > >
> > > Best,
> > > Goun Na
> >
> >
>
>
> --
>
> *Per Ullberg*
> Data Vault Tech Lead
> Odin Uppsala
> +46 701612693 <+46+701612693>
>
> Klarna AB (publ)
> Sveavägen 46, 111 34 Stockholm
> Tel: +46 8 120 120 00 <+46812012000>
> Reg no: 556737-0431
> klarna.com
>
Re: Oozie Job + Version Control Tool
Posted by Per Ullberg <pe...@klarna.com>.
We package our oozie jobs with maven and release artifacts to nexus. We
keep the version number as part of the coordinator name. That way we have
full traceability between code base and running coordinators.
regards
/Pelle
On Wed, Oct 19, 2016 at 10:05 AM, Abhishek Bafna <ba...@gmail.com>
wrote:
> Hi,
>
> Oozie does not have version control for jobs. When you submit a
> workflow/coordinator/bundle to oozie, it stores it into DB uses it from
> there for further execution.
>
> Thanks,
> Abhishek
> > On Oct 19, 2016, at 1:16 PM, goun na <go...@gmail.com> wrote:
> >
> > Hi users,
> >
> > What is the best to manage Oozie jobs? Is there a built-in version
> control
> > feature?
> >
> > Best,
> > Goun Na
>
>
--
*Per Ullberg*
Data Vault Tech Lead
Odin Uppsala
+46 701612693 <+46+701612693>
Klarna AB (publ)
Sveavägen 46, 111 34 Stockholm
Tel: +46 8 120 120 00 <+46812012000>
Reg no: 556737-0431
klarna.com
Re: Oozie Job + Version Control Tool
Posted by Abhishek Bafna <ba...@gmail.com>.
Hi,
Oozie does not have version control for jobs. When you submit a workflow/coordinator/bundle to oozie, it stores it into DB uses it from there for further execution.
Thanks,
Abhishek
> On Oct 19, 2016, at 1:16 PM, goun na <go...@gmail.com> wrote:
>
> Hi users,
>
> What is the best to manage Oozie jobs? Is there a built-in version control
> feature?
>
> Best,
> Goun Na