You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by Andrew Kenworthy <ad...@yahoo.com> on 2011/09/09 10:37:17 UTC
Rerunning a workflow using resolved variables
Hi,
I am setting up workflow control using the following structure:
a) a Coordinator job that calls a "parent" workflow
b) this parent workflow calls a java action to set various variables and then
c) calls one or more subworkflows, injecting variables from b) as needed
I would like to split the subworkflows in c) such that logical, discrete units are easy to rerun. I had thought that it would then be possible to call "rerun" on one of these workflows in isolation, assuming that the workflow definition - complete with *resolved* variables - would be located in the database and re-executed. But it seems that the saved workflow definition still contains variables like ${myVar}. I know I can call the coordinator again specifying which nodes to omit, or supply all variables on the command line, but would ideally like to just call "rerun", knowing my variables have been persisted. Is this not possible, or am I missing something?
Regards,
Andrew
Re: Rerunning a workflow using resolved variables
Posted by Mohammad Islam <mi...@yahoo.com>.
Hi Andrew,
You are right that the saved config is not applied during rerun.
You can still get the config of the original jobs in 2.3 as follows
1. curl -k <OOZIE_URL>/v1/job/<JobID>
This command will return the whole job information in JSON format.
You can parse it using any JSON parser.
2. Alternatively, you can use Oozie Java API to get the same.
OozieClient.getJobInfo(...).getConf().
Regards,
Mohammad
________________________________
From: Andrew Kenworthy <ad...@yahoo.com>
To: Mohammad Islam <mi...@yahoo.com>
Cc: "oozie-users@incubator.apache.org" <oo...@incubator.apache.org>
Sent: Monday, September 12, 2011 12:41 AM
Subject: Re: Rerunning a workflow using resolved variables
Hi Mohammad,
Thank you for your input - I've had a look in the logs and think I understand things a little better now:
I see for each job that the a) job-definition is saved (with variables unresolved i.e. just the ${...} placeholders), as well as b) the job-configuration (with variables resolved into specific values). But...when I call a "rerun", the config for that job saved in the database is not automatically applied.
For example, if I start the "subwf" example workflow supplied in the oozie examples folder, I see two jobs in the console; the parent workflow (subwf) and the child one (map-reduce). I had expected that if I restart the "map-reduce" child workflow directly, then the job definition would - by default - be merged with the persisted configuration. But in the logs I see:
2011-09-12 09:16:32,702 ERROR V1JobServlet:522 - USER[...] GROUP[...] TOKEN[-] APP[-] JOB[-] ACTION[-] URL[PUT http://localhost:11000/oozie/v1/job/<my-job-id>-oozie-oozi-W?action=rerun] error, null
java.lang.NullPointerException
at java.util.Hashtable.put(Hashtable.java:394)
at java.util.Properties.setProperty(Properties.java:143)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:460)
at org.apache.oozie.servlet.BaseJobServlet.checkAuthorizationForApp(BaseJobServlet.java:165)
at org.apache.oozie.servlet.BaseJobServlet.doPut(BaseJobServlet.java:110)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:640)
at org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:281)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:123)
at com.cloudera.alfredo.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:371)
at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:128)
......
Which as far as I can tell is due to the workflow path not being found (in BaseJobServlet.java:165). Is that expected behaviour?
(I'm working with version 2.3.0 I can't call the config_content option).
Regards,
Andrew
>________________________________
>From: Mohammad Islam <mi...@yahoo.com>
>To: "oozie-users@incubator.apache.org" <oo...@incubator.apache.org>; Andrew Kenworthy <ad...@yahoo.com>
>Sent: Friday, September 9, 2011 10:57 AM
>Subject: Re: Rerunning a workflow using resolved variables
>
>
>Hi Andrew,
>I'm not sure I fully understand the cases. Here is my take ...
>
>
>1. If the issue is variables created in previous Java Action are not available during later actions rerun, please let us know. I think the variables generated in any java actions are stored in DB. So it should not be an issue.
>
>
>2. Alternatively, if you want to reuse the same WF configuration for rerun, the following is an option.
>
>
>Oozie has WS API to get the configuration of already running jobs. You could get it and write to file and use that as rerun config.
>I believe the equivalent command option is: oozie job -config_content <ID>.
>I think it was supported from oozie 3.0. For older versions, you might need to tweak.
>
>
>
>
>Regards,
>Mohammad
>
>
>
>
>
>________________________________
>From: Andrew Kenworthy <ad...@yahoo.com>
>To: "oozie-users@incubator.apache.org" <oo...@incubator.apache.org>
>Sent: Friday, September 9, 2011 1:37 AM
>Subject: Rerunning a workflow using resolved variables
>
>Hi,
>
>I am setting up workflow control using the following structure:
>
>a) a Coordinator job that calls a "parent" workflow
>b)
this parent workflow calls a
java action to set
various variables and then
>c) calls one or more subworkflows, injecting variables from b) as needed
>
>I would like to split the subworkflows in c) such that logical, discrete units are easy to rerun. I had thought that it would then be possible to call "rerun" on one of these workflows in isolation, assuming that the workflow definition - complete with *resolved* variables - would be located in the database and re-executed. But it seems that the saved workflow definition still contains variables like ${myVar}. I know I can call the coordinator again specifying which nodes to omit, or supply all variables on the command line, but would ideally like to just call "rerun", knowing my variables have been persisted. Is this not possible, or am I missing something?
>
>Regards,
>
>Andrew
>
>
>
>
Re: Rerunning a workflow using resolved variables
Posted by Andrew Kenworthy <ad...@yahoo.com>.
Hi Mohammad,
Thank you for your input - I've had a look in the logs and think I understand things a little better now:
I see for each job that the a) job-definition is saved (with variables unresolved i.e. just the ${...} placeholders), as well as b) the job-configuration (with variables resolved into specific values). But...when I call a "rerun", the config for that job saved in the database is not automatically applied.
For example, if I start the "subwf" example workflow supplied in the oozie examples folder, I see two jobs in the console; the parent workflow (subwf) and the child one (map-reduce). I had expected that if I restart the "map-reduce" child workflow directly, then the job definition would - by default - be merged with the persisted configuration. But in the logs I see:
2011-09-12 09:16:32,702 ERROR V1JobServlet:522 - USER[...] GROUP[...] TOKEN[-] APP[-] JOB[-] ACTION[-] URL[PUT http://localhost:11000/oozie/v1/job/<my-job-id>-oozie-oozi-W?action=rerun] error, null
java.lang.NullPointerException
at java.util.Hashtable.put(Hashtable.java:394)
at java.util.Properties.setProperty(Properties.java:143)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:460)
at org.apache.oozie.servlet.BaseJobServlet.checkAuthorizationForApp(BaseJobServlet.java:165)
at org.apache.oozie.servlet.BaseJobServlet.doPut(BaseJobServlet.java:110)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:640)
at org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:281)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:123)
at com.cloudera.alfredo.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:371)
at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:128)
......
Which as far as I can tell is due to the workflow path not being found (in BaseJobServlet.java:165). Is that expected behaviour?
(I'm working with version 2.3.0 I can't call the config_content option).
Regards,
Andrew
>________________________________
>From: Mohammad Islam <mi...@yahoo.com>
>To: "oozie-users@incubator.apache.org" <oo...@incubator.apache.org>; Andrew Kenworthy <ad...@yahoo.com>
>Sent: Friday, September 9, 2011 10:57 AM
>Subject: Re: Rerunning a workflow using resolved variables
>
>
>Hi Andrew,
>I'm not sure I fully understand the cases. Here is my take ...
>
>
>1. If the issue is variables created in previous Java Action are not available during later actions rerun, please let us know. I think the variables generated in any java actions are stored in DB. So it should not be an issue.
>
>
>2. Alternatively, if you want to reuse the same WF configuration for rerun, the following is an option.
>
>
>Oozie has WS API to get the configuration of already running jobs. You could get it and write to file and use that as rerun config.
>I believe the equivalent command option is: oozie job -config_content <ID>.
>I think it was supported from oozie 3.0. For older versions, you might need to tweak.
>
>
>
>
>Regards,
>Mohammad
>
>
>
>
>
>________________________________
>From: Andrew Kenworthy <ad...@yahoo.com>
>To: "oozie-users@incubator.apache.org" <oo...@incubator.apache.org>
>Sent: Friday, September 9, 2011 1:37 AM
>Subject: Rerunning a workflow using resolved variables
>
>Hi,
>
>I am setting up workflow control using the following structure:
>
>a) a Coordinator job that calls a "parent" workflow
>b) this parent workflow calls a
java action to set
various variables and then
>c) calls one or more subworkflows, injecting variables from b) as needed
>
>I would like to split the subworkflows in c) such that logical, discrete units are easy to rerun. I had thought that it would then be possible to call "rerun" on one of these workflows in isolation, assuming that the workflow definition - complete with *resolved* variables - would be located in the database and re-executed. But it seems that the saved workflow definition still contains variables like ${myVar}. I know I can call the coordinator again specifying which nodes to omit, or supply all variables on the command line, but would ideally like to just call "rerun", knowing my variables have been persisted. Is this not possible, or am I missing something?
>
>Regards,
>
>Andrew
>
>
>
>
Re: Rerunning a workflow using resolved variables
Posted by Mohammad Islam <mi...@yahoo.com>.
Hi Andrew,
I'm not sure I fully understand the cases. Here is my take ...
1. If the issue is variables created in previous Java Action are not available during later actions rerun, please let us know. I think the variables generated in any java actions are stored in DB. So it should not be an issue.
2. Alternatively, if you want to reuse the same WF configuration for rerun, the following is an option.
Oozie has WS API to get the configuration of already running jobs. You could get it and write to file and use that as rerun config.
I believe the equivalent command option is: oozie job -config_content <ID>.
I think it was supported from oozie 3.0. For older versions, you might need to tweak.
Regards,
Mohammad
________________________________
From: Andrew Kenworthy <ad...@yahoo.com>
To: "oozie-users@incubator.apache.org" <oo...@incubator.apache.org>
Sent: Friday, September 9, 2011 1:37 AM
Subject: Rerunning a workflow using resolved variables
Hi,
I am setting up workflow control using the following structure:
a) a Coordinator job that calls a "parent" workflow
b) this parent workflow calls a java action to set various variables and then
c) calls one or more subworkflows, injecting variables from b) as needed
I would like to split the subworkflows in c) such that logical, discrete units are easy to rerun. I had thought that it would then be possible to call "rerun" on one of these workflows in isolation, assuming that the workflow definition - complete with *resolved* variables - would be located in the database and re-executed. But it seems that the saved workflow definition still contains variables like ${myVar}. I know I can call the coordinator again specifying which nodes to omit, or supply all variables on the command line, but would ideally like to just call "rerun", knowing my variables have been persisted. Is this not possible, or am I missing something?
Regards,
Andrew