You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by Aaron Beppu <ab...@etsy.com> on 2012/01/10 00:54:40 UTC
schema for new (extension) action type not being picked up
Hi all. Confused oozie neophyte here, hoping for some guidance.
I'm trying to create a new action type, following directions both on InfoQ
[1], and in the official documentation [2]. So I've:
1. created the new action type, along with a schema,
2. built a jar containing the new code and schema,
3. added that jar to ${CATALINA_BASE}/webapps/oozie/WEB-INF/lib (and when
that failed, using ".../bin/oozie-setup.sh -jars mine.jar")
4. added my new action type to oozie.service.ActionService.executor.ext.classes
in oozie-site.xml
5. added my xsd filename to oozie.service.SchemaService.wf.ext.schemas, and
to oozie.service.WorkflowSchemaService.ext.schemas (because the oozie docs
and the InfoQ directions disagreed)
6. restarted my oozie server and
7. attempted to submit a workflow using the new action type.
At this point, whenever I try to run the test workflow with the new action,
I get this error :
Error: E0701 : E0701: XML schema error, cvc-complex-type.2.4.a: Invalid
content was found starting with element 'cascadingjruby'. One of
'{"uri:oozie:workflow:0.2":map-reduce, "uri:oozie:workflow:0.2":pig,
"uri:oozie:workflow:0.2":sub-workflow, "uri:oozie:workflow:0.2":fs,
"uri:oozie:workflow:0.2":java, WC[##other:"uri:oozie:workflow:0.2"]}' is
expected.
A few notes :
1. I know that I'm naming the right .xsd file and that it's being picked up
because earlier I had a typo in the filename in oozie-site.xml, and I got
an *earlier* error when attempting to restart oozie [3], *before* I even
tried to kick a job. So having eliminated that error, and not seeing
anything break until I actually try to start a job, I'm reasonably
confident that I'm not just making a filename problem -- and yet it appears
not to know about my new action type.
2. "Extended" action types which I *can* use successfully (e.g.
Hive/email/sqoop) are *not* on the list enumerated -- so whatever it's
using to enumerate that list of "good" types, it's *not* identical to the
actual set of actions that it's loaded and can execute.
3. The sole comment on that InfoQ article is from someone in October
getting the same error. He didn't get any response from the article
authors. So I'm guessing I'm making a common enough error that someone
around here must know about the problem.
What am I missing? I'm running cdh3u2 (so oozie version 2.3.2). I'm
appending in the relevant chunk of my oozie-site.xml [4].
Thanks so much,
Aaron
[1] http://www.infoq.com/articles/ExtendingOozie
[2] http://archive.cloudera.com/cdh/3/oozie/DG_CustomActionExecutor.html
[3] java.lang.IllegalArgumentException: resource
[cascadingjruby-action-1.0.xsd] not found
at
org.apache.oozie.util.IOUtils.getResourceAsStream(IOUtils.java:100)
at
org.apache.oozie.service.SchemaService.loadSchema(SchemaService.java:69)
at
org.apache.oozie.service.SchemaService.init(SchemaService.java:85)
at
org.apache.oozie.service.Services.setServiceInternal(Services.java:307)
at org.apache.oozie.service.Services.setService(Services.java:293)
at org.apache.oozie.service.Services.init(Services.java:203)
at
org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:37)
at
org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4205)
at
org.apache.catalina.core.StandardContext.start(StandardContext.java:4704)
at
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
at
org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
at
org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
at
org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:675)
at
org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:601)
at
org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:502)
at
org.apache.catalina.startup.HostConfig.start(HostConfig.java:1315)
at
org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
at
org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
at
org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1061)
at
org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
at
org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053)
at
org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
at
org.apache.catalina.core.StandardService.start(StandardService.java:525)
at
org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
[4] <property>
<name>oozie.service.ActionService.executor.ext.classes</name>
<value>
org.apache.oozie.action.hadoop.CascadingJrubyIndependent,
org.apache.oozie.action.hadoop.HiveActionExecutor,
org.apache.oozie.action.hadoop.SqoopActionExecutor,
org.apache.oozie.action.email.EmailActionExecutor
</value>
</property>
<property>
<name>oozie.service.SchemaService.wf.ext.schemas</name>
<value>hive-action-0.2.xsd,sqoop-action-0.2.xsd,email-action-0.1.xsd,distcp-action-0.1.xsd,cascadingjruby-action-1.0.xsd</value>
</property>
<property>
<name>oozie.service.WorkflowSchemaService.ext.schemas</name>
<value>hive-action-0.2.xsd,sqoop-action-0.2.xsd,email-action-0.1.xsd,distcp-action-0.1.xsd,cascadingjruby-action-1.0.xsd</value>
</property>
Re: schema for new (extension) action type not being picked up
Posted by Mohammad Islam <mi...@yahoo.com>.
Hi Aaron,
Would you please help the future open source community by providing your use case and experiences (such as code) in the apache wiki page?
I think anyone could write into (c)wiki after creating an account. If you need help we could help you too.
In general, I would request others if you have a use case (specially with new action), please upload that to share with others. Of course, I assume you would remove the business logic. Moreover, it doesn't need to be very superb code. This is just for reference only.
We can than link or upload that write-ups into Oozie main doc page.
Regards,
Mohammad
________________________________
From: Aaron Beppu <ab...@etsy.com>
To: oozie-users@incubator.apache.org
Sent: Wednesday, January 11, 2012 7:40 AM
Subject: Re: schema for new (extension) action type not being picked up
Hi Ken,
A solution to the original problem *has* been found -- Mohammad was correct
that I had failed to put my namespace when I tried to use my new node in a
workflow (so that node needed to be opened like : " <cascadingjruby
xmlns="uri:oozie:cascadingjruby-action:0.1">). Following that change, I can
launch my jobs, and am now working on fixing the bugs within my particular
action. As a side note, the namespace issue is exactly why the error I
encountered printed only the "core" actions (not hive/sqoop/email) which
are in the "uri:oozie:workflow" namespace.
With respect to oozie + ruby, I'm afraid the name of my new action type may
have mislead you. I'm trying to run jobs built with cascading.jruby (
https://github.com/gmarabout/cascading.jruby), a jruby DSL around
Cascading. As a result, my action is not well suited for getting oozie to
run more generic ruby stuff (and certainly doesn't work if your use MRI).
Best,
Aaron
On Wed, Jan 11, 2012 at 8:17 AM, Ken Ernst <ke...@mobileposse.com> wrote:
> Hi Aaron,
>
> I have been following this thread and was hoping a solution would have
> been found.
> Although we are not using ruby with oozie yet I have some ruby code that I
> would
> definitely like to incorporate into oozie.
>
> If you get this working please post it, it would be a great help!
>
> Thanks.
>
> Ken
>
>
>
>
> On 1/10/12 4:13 PM, "Aaron Beppu" <ab...@etsy.com> wrote:
>
> >Hi Mohammad,
> >
> >Thanks for the help! I played around with this a little, and the only way
> >I
> >got a different (I think better) outcome was by putting the xmlns
> >attribute
> >in the cascadingjruby node, rather than the action node, such that the
> >start of the workflow definition looks like this now :
> >
> ><workflow-app xmlns="uri:oozie:workflow:0.2" name="cj-search-query-count">
> > <start to="cj-node"/>
> > <action name="cj-node">
> > <cascadingjruby xmlns="uri:oozie:cascadingjruby-action:0.1">
> > ...
> >
> >
> >Now the job launches, and provides a totally new array of exciting errors
> >for me to investigate ;)
> >
> >Hopefully this entry in the list archive will be helpful to someone.
> >
> >Thanks,
> >Aaron
> >
> >
> >On Tue, Jan 10, 2012 at 2:51 PM, Mohammad Islam <mi...@yahoo.com>
> >wrote:
> >
> >> Hi Aaron,
> >> think you didn't put the namespace for your action correctly.=
> >>
> >>
> >> Please try with this:
> >>
> >> <action xmlns=3Duri:oozie:cascadingjruby-action:0.1 name=3D"cj-node">
> >>
> >> Regards,
> >> Mohammad
> >>
> >> ----- Original Message -----
> >> From: Aaron Beppu <ab...@etsy.com>
> >> To: oozie-users@incubator.apache.org; Mohammad Islam
> >><mi...@yahoo.com>
> >> Cc:
> >> Sent: Tuesday, January 10, 2012 11:04 AM
> >> Subject: Re: schema for new (extension) action type not being picked up
> >>
> >> Any suggestions? It looks like I'm not missing anything with
> >>minOccurs="1"
> >> and I haven't put in more than the maxOccurs for anything. Is there a
> >>way
> >> get oozie to tell which actions the schema service has loaded?
> >>
> >> On Tue, Jan 10, 2012 at 9:05 AM, Aaron Beppu <ab...@etsy.com> wrote:
> >>
> >> > Mohammad,
> >> >
> >> > Thanks for looking. Yeah, my bad. I guess I assumed because of the
> >>error
> >> > message that the issue was that the action wasn't registered properly
> >> > rather than that I was using it wrong -- but clearly one of my
> >> assumptions
> >> > is going to be violated before this is over. Here's the workflow
> >> definition
> >> > (unchanged except that I've subbed out the names of our s3 buckets) :
> >> >
> >> > <workflow-app xmlns="uri:oozie:workflow:0.2"
> >> name="cj-search-query-count">
> >> > <start to="cj-node"/>
> >> > <action name="cj-node">
> >> > <cascadingjruby>
> >> > <job-tracker>${jobTracker}</job-tracker>
> >> > <name-node>${nameNode}</name-node>
> >> > <prepare>
> >> > <delete
> >> >
> >>
> >>path="${nameNode}/user/${wf:user()}/${examplesRoot}/output-data/bigdata"/
> >>>
> >> > </prepare>
> >> > <configuration>
> >> > <property>
> >> > <name>mapred.job.queue.name</name>
> >> > <value>${queueName}</value>
> >> > </property>
> >> > <property>
> >> > <name>mapred.compress.map.output</name>
> >> > <value>true</value>
> >> > </property>
> >> > </configuration>
> >> > <arg>search_query_count.rb</arg>
> >> > <arg>today_path=2011_12_16</arg>
> >> > <arg>yesterday_path=2011_12_15</arg>
> >> > <arg>time_path=2011_12_16/16_33_31</arg>
> >> > <arg>task_name=analytics_search_query_count</arg>
> >> > <arg>job_signature=secretsecretsecret</arg>
> >> > <arg>end_date=2011_12_10</arg>
> >> > <arg>log_prefix=s3n://secretlogbucket</arg>
> >> > <arg>input_prefix=s3n://secretinputbucket</arg>
> >> > <arg>output_prefix=s3n://secretoutputbucket</arg>
> >> > <arg>index_prefix=s3n://secretindexbucket</arg>
> >> > <arg>hive_prefix=s3n://secrethivebucekt</arg>
> >> > <arg>graphite_prefix=s3n://secretgraphicbucket</arg>
> >> > <arg>splunk_prefix=s3n://secretsplunkbucket</arg>
> >> > <arg>dbdata_prefix=s3n://secretdbdatabucket</arg>
> >> > <arg>instance_type=c1.medium</arg>
> >> > <arg>instance_count=20</arg>
> >> > <arg>master_instance_type=c1.medium</arg>
> >> > </cascadingjruby>
> >> > <ok to="end"/>
> >> > <error to="fail"/>
> >> > </action>
> >> > <kill name="fail">
> >> > <message>BigData java action failed, error
> >> > message[${wf:errorMessage(wf:lastErrorNode())}]</message>
> >> > </kill>
> >> > <end name="end"/>
> >> > </workflow-app>
> >> >
> >> >
> >> >
> >> >
> >> > Thanks,
> >> > Aaron
> >> >
> >> > On Tue, Jan 10, 2012 at 4:46 AM, Mohammad Islam <mislam77@yahoo.com
> >> >wrote:
> >> >
> >> >> Would you please send at least the relevant definition of action from
> >> >> your workflow.xml?
> >> >>
> >> >> Thanks
> >> >> Mohammad
> >> >>
> >> >>
> >> >> ________________________________
> >> >> From: Aaron Beppu <ab...@etsy.com>
> >> >> To: oozie-users@incubator.apache.org
> >> >> Sent: Monday, January 9, 2012 5:04 PM
> >> >> Subject: Re: schema for new (extension) action type not being picked
> >>up
> >> >>
> >> >>
> >> >> Hi Alejandro,
> >> >>
> >> >> Thanks for looking. I'm both attaching and copy-pasting, as I'm not
> >>sure
> >> >> how the list handles attachments. I don't know that all of this is
> >> >> necessary, but I was following hive-action-0.2.xsd and
> >> sqoop-action-0.2.xsd
> >> >> closely as examples.
> >> >>
> >> >> <?xmlversion="1.0"encoding="UTF-8"?>
> >> >> <xs:schemaxmlns:xs="http://www.w3.org/2001/XMLSchema";;
> >> >>
> >> >>
> >>
> >>xmlns:cascadingjruby="uri:oozie:cascadingjruby-action:0.1"elementFormDefa
> >>ult="qualified"
> >> >> targetNamespace="uri:oozie:cascadingjruby-action:0.1">
> >> >>
> >> >> <xs:elementname="cascadingjruby"type="cascadingjruby:ACTION"/>
> >> >>
> >> >> <xs:complexTypename="ACTION">
> >> >> <xs:sequence>
> >> >>
> >> >>
> >>
> >><xs:elementname="job-tracker"type="xs:string"minOccurs="1"maxOccurs="1"/>
> >> >>
> >> >>
> >><xs:elementname="name-node"type="xs:string"minOccurs="1"maxOccurs="1"/>
> >> >>
> >> >>
> >>
> >><xs:elementname="prepare"type="cascadingjruby:PREPARE"minOccurs="0"maxOcc
> >>urs="1"/>
> >> >>
> >> >> <xs:elementname="job-xml"type="xs:string"minOccurs="0"maxOccurs="1"/>
> >> >>
> >> >>
> >>
> >><xs:elementname="configuration"type="cascadingjruby:CONFIGURATION"minOccu
> >>rs="0"maxOccurs="1"/>
> >> >>
> >> >>
> >>
> >><xs:elementname="arg"type="xs:string"minOccurs="0"maxOccurs="unbounded"/>
> >> >>
> >> >>
> >>
> >><xs:elementname="file"type="xs:string"minOccurs="0"maxOccurs="unbounded"/
> >>>
> >> >>
> >> >>
> >>
> >><xs:elementname="archive"type="xs:string"minOccurs="0"maxOccurs="unbounde
> >>d"/>
> >> >> </xs:sequence>
> >> >> </xs:complexType>
> >> >>
> >> >> <xs:complexTypename="CONFIGURATION">
> >> >> <xs:sequence>
> >> >>
> >> <xs:elementname="property"minOccurs="1"maxOccurs="unbounded">
> >> >> <xs:complexType>
> >> >> <xs:sequence>
> >> >> <xs:element name="name" minOccurs="1"
> >> >> maxOccurs="1" type="xs:string"/>
> >> >> <xs:element name="value" minOccurs="1"
> >> >> maxOccurs="1" type="xs:string"/>
> >> >> <xs:element name="description" minOccurs="0"
> >> >> maxOccurs="1" type="xs:string"/>
> >> >> </xs:sequence>
> >> >> </xs:complexType>
> >> >> </xs:element>
> >> >> </xs:sequence>
> >> >> </xs:complexType>
> >> >>
> >> >> <xs:complexTypename="PREPARE">
> >> >> <xs:sequence>
> >> >>
> >> >>
> >>
> >><xs:elementname="delete"type="cascadingjruby:DELETE"minOccurs="0"maxOccur
> >>s="unbounded"/>
> >> >>
> >> >>
> >>
> >><xs:elementname="mkdir"type="cascadingjruby:MKDIR"minOccurs="0"maxOccurs=
> >>"unbounded"/>
> >> >> </xs:sequence>
> >> >> </xs:complexType>
> >> >>
> >> >> <xs:complexTypename="DELETE">
> >> >> <xs:attributename="path"type="xs:string"use="required"/>
> >> >> </xs:complexType>
> >> >>
> >> >> <xs:complexTypename="MKDIR">
> >> >> <xs:attributename="path"type="xs:string"use="required"/>
> >> >> </xs:complexType>
> >> >>
> >> >> </xs:schema>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> On Mon, Jan 9, 2012 at 7:26 PM, Alejandro Abdelnur
> >><tu...@cloudera.com>
> >> >> wrote:
> >> >>
> >> >> Aaron,
> >> >> >
> >> >> >Would you mind attaching your XSD file?
> >> >> >
> >> >> >Thxs.
> >> >> >
> >> >> >Alejandro
> >> >> >
> >> >> >
> >> >> >On Mon, Jan 9, 2012 at 3:54 PM, Aaron Beppu <ab...@etsy.com>
> wrote:
> >> >> >
> >> >> >> Hi all. Confused oozie neophyte here, hoping for some guidance.
> >> >> >>
> >> >> >> I'm trying to create a new action type, following directions both
> >>on
> >> >> InfoQ
> >> >> >> [1], and in the official documentation [2]. So I've:
> >> >> >> 1. created the new action type, along with a schema,
> >> >> >> 2. built a jar containing the new code and schema,
> >> >> >> 3. added that jar to ${CATALINA_BASE}/webapps/oozie/WEB-INF/lib
> >>(and
> >> >> when
> >> >> >> that failed, using ".../bin/oozie-setup.sh -jars mine.jar")
> >> >> >> 4. added my new action type to
> >> >> >> oozie.service.ActionService.executor.ext.classes
> >> >> >> in oozie-site.xml
> >> >> >> 5. added my xsd filename to
> >> >> oozie.service.SchemaService.wf.ext.schemas, and
> >> >> >> to oozie.service.WorkflowSchemaService.ext.schemas (because the
> >>oozie
> >> >> docs
> >> >> >> and the InfoQ directions disagreed)
> >> >> >> 6. restarted my oozie server and
> >> >> >> 7. attempted to submit a workflow using the new action type.
> >> >> >>
> >> >> >> At this point, whenever I try to run the test workflow with the
> >>new
> >> >> action,
> >> >> >> I get this error :
> >> >> >>
> >> >> >> Error: E0701 : E0701: XML schema error, cvc-complex-type.2.4.a:
> >> Invalid
> >> >> >> content was found starting with element 'cascadingjruby'. One of
> >> >> >> '{"uri:oozie:workflow:0.2":map-reduce,
> >>"uri:oozie:workflow:0.2":pig,
> >> >> >> "uri:oozie:workflow:0.2":sub-workflow,
> >>"uri:oozie:workflow:0.2":fs,
> >> >> >> "uri:oozie:workflow:0.2":java,
> >>WC[##other:"uri:oozie:workflow:0.2"]}'
> >> >> is
> >> >> >> expected.
> >> >> >>
> >> >> >> A few notes :
> >> >> >> 1. I know that I'm naming the right .xsd file and that it's being
> >> >> picked up
> >> >> >> because earlier I had a typo in the filename in oozie-site.xml,
> >>and I
> >> >> got
> >> >> >> an *earlier* error when attempting to restart oozie [3], *before*
> >>I
> >> >> even
> >> >> >> tried to kick a job. So having eliminated that error, and not
> >>seeing
> >> >> >> anything break until I actually try to start a job, I'm reasonably
> >> >> >> confident that I'm not just making a filename problem -- and yet
> >>it
> >> >> appears
> >> >> >> not to know about my new action type.
> >> >> >> 2. "Extended" action types which I *can* use successfully (e.g.
> >> >> >> Hive/email/sqoop) are *not* on the list enumerated -- so whatever
> >> it's
> >> >> >> using to enumerate that list of "good" types, it's *not*
> >>identical to
> >> >> the
> >> >> >> actual set of actions that it's loaded and can execute.
> >> >> >> 3. The sole comment on that InfoQ article is from someone in
> >>October
> >> >> >> getting the same error. He didn't get any response from the
> >>article
> >> >> >> authors. So I'm guessing I'm making a common enough error that
> >> someone
> >> >> >> around here must know about the problem.
> >> >> >>
> >> >> >> What am I missing? I'm running cdh3u2 (so oozie version 2.3.2).
> >>I'm
> >> >> >> appending in the relevant chunk of my oozie-site.xml [4].
> >> >> >>
> >> >> >> Thanks so much,
> >> >> >> Aaron
> >> >> >>
> >> >> >> [1] http://www.infoq.com/articles/ExtendingOozie
> >> >> >> [2]
> >> >> http://archive.cloudera.com/cdh/3/oozie/DG_CustomActionExecutor.html
> >> >> >> [3] java.lang.IllegalArgumentException: resource
> >> >> >> [cascadingjruby-action-1.0.xsd] not found
> >> >> >> at
> >> >> >>
> >>org.apache.oozie.util.IOUtils.getResourceAsStream(IOUtils.java:100)
> >> >> >> at
> >> >> >>
> >> >>
> >>org.apache.oozie.service.SchemaService.loadSchema(SchemaService.java:69)
> >> >> >> at
> >> >> >> org.apache.oozie.service.SchemaService.init(SchemaService.java:85)
> >> >> >> at
> >> >> >>
> >> org.apache.oozie.service.Services.setServiceInternal(Services.java:307)
> >> >> >> at
> >> >> org.apache.oozie.service.Services.setService(Services.java:293)
> >> >> >> at
> >>org.apache.oozie.service.Services.init(Services.java:203)
> >> >> >> at
> >> >> >>
> >> >> >>
> >> >>
> >>
> >>org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader
> >>.java:37)
> >> >> >> at
> >> >> >>
> >> >> >>
> >> >>
> >>
> >>org.apache.catalina.core.StandardContext.listenerStart(StandardContext.ja
> >>va:4205)
> >> >> >> at
> >> >> >>
> >> >>
> >>
> >>org.apache.catalina.core.StandardContext.start(StandardContext.java:4704)
> >> >> >> at
> >> >> >>
> >> >> >>
> >> >>
> >>
> >>org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.jav
> >>a:799)
> >> >> >> at
> >> >> >>
> >> org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
> >> >> >> at
> >> >> >>
> >>org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
> >> >> >> at
> >> >> >>
> >> >> >>
> >> >>
> >>
> >>org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:6
> >>75)
> >> >> >> at
> >> >> >>
> >> >> >>
> >> >>
> >>
> >>org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:
> >>601)
> >> >> >> at
> >> >> >>
> >> org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:502)
> >> >> >> at
> >> >> >> org.apache.catalina.startup.HostConfig.start(HostConfig.java:1315)
> >> >> >> at
> >> >> >>
> >> >>
> >>
> >>org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324
> >>)
> >> >> >> at
> >> >> >>
> >> >> >>
> >> >>
> >>
> >>org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSup
> >>port.java:142)
> >> >> >> at
> >> >> >>
> >>org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1061)
> >> >> >> at
> >> >> >> org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
> >> >> >> at
> >> >> >>
> >>org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053)
> >> >> >> at
> >> >> >>
> >> org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
> >> >> >> at
> >> >> >>
> >> >>
> >>org.apache.catalina.core.StandardService.start(StandardService.java:525)
> >> >> >> at
> >> >> >>
> >> org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
> >> >> >> at
> >> org.apache.catalina.startup.Catalina.start(Catalina.java:595)
> >> >> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >>Method)
> >> >> >> at
> >> >> >>
> >> >> >>
> >> >>
> >>
> >>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java
> >>:39)
> >> >> >> at
> >> >> >>
> >> >> >>
> >> >>
> >>
> >>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI
> >>mpl.java:25)
> >> >> >> at java.lang.reflect.Method.invoke(Method.java:597)
> >> >> >> at
> >> >> org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
> >> >> >> at
> >> >> org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
> >> >> >> [4] <property>
> >> >> >>
> >><name>oozie.service.ActionService.executor.ext.classes</name>
> >> >> >> <value>
> >> >> >>
> >>org.apache.oozie.action.hadoop.CascadingJrubyIndependent,
> >> >> >> org.apache.oozie.action.hadoop.HiveActionExecutor,
> >> >> >> org.apache.oozie.action.hadoop.SqoopActionExecutor,
> >> >> >> org.apache.oozie.action.email.EmailActionExecutor
> >> >> >> </value>
> >> >> >> </property>
> >> >> >>
> >> >> >> <property>
> >> >> >> <name>oozie.service.SchemaService.wf.ext.schemas</name>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >>
> >>
> >><value>hive-action-0.2.xsd,sqoop-action-0.2.xsd,email-action-0.1.xsd,dist
> >>cp-action-0.1.xsd,cascadingjruby-action-1.0.xsd</value>
> >> >> >> </property>
> >> >> >> <property>
> >> >> >>
> >><name>oozie.service.WorkflowSchemaService.ext.schemas</name>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >>
> >>
> >><value>hive-action-0.2.xsd,sqoop-action-0.2.xsd,email-action-0.1.xsd,dist
> >>cp-action-0.1.xsd,cascadingjruby-action-1.0.xsd</value>
> >> >> >>
> >> >> >> </property>
> >> >> >>
> >> >> >
> >> >>
> >> >
> >> >
> >>
> >>
>
>
Re: schema for new (extension) action type not being picked up
Posted by Aaron Beppu <ab...@etsy.com>.
Hi Ken,
A solution to the original problem *has* been found -- Mohammad was correct
that I had failed to put my namespace when I tried to use my new node in a
workflow (so that node needed to be opened like : " <cascadingjruby
xmlns="uri:oozie:cascadingjruby-action:0.1">). Following that change, I can
launch my jobs, and am now working on fixing the bugs within my particular
action. As a side note, the namespace issue is exactly why the error I
encountered printed only the "core" actions (not hive/sqoop/email) which
are in the "uri:oozie:workflow" namespace.
With respect to oozie + ruby, I'm afraid the name of my new action type may
have mislead you. I'm trying to run jobs built with cascading.jruby (
https://github.com/gmarabout/cascading.jruby), a jruby DSL around
Cascading. As a result, my action is not well suited for getting oozie to
run more generic ruby stuff (and certainly doesn't work if your use MRI).
Best,
Aaron
On Wed, Jan 11, 2012 at 8:17 AM, Ken Ernst <ke...@mobileposse.com> wrote:
> Hi Aaron,
>
> I have been following this thread and was hoping a solution would have
> been found.
> Although we are not using ruby with oozie yet I have some ruby code that I
> would
> definitely like to incorporate into oozie.
>
> If you get this working please post it, it would be a great help!
>
> Thanks.
>
> Ken
>
>
>
>
> On 1/10/12 4:13 PM, "Aaron Beppu" <ab...@etsy.com> wrote:
>
> >Hi Mohammad,
> >
> >Thanks for the help! I played around with this a little, and the only way
> >I
> >got a different (I think better) outcome was by putting the xmlns
> >attribute
> >in the cascadingjruby node, rather than the action node, such that the
> >start of the workflow definition looks like this now :
> >
> ><workflow-app xmlns="uri:oozie:workflow:0.2" name="cj-search-query-count">
> > <start to="cj-node"/>
> > <action name="cj-node">
> > <cascadingjruby xmlns="uri:oozie:cascadingjruby-action:0.1">
> > ...
> >
> >
> >Now the job launches, and provides a totally new array of exciting errors
> >for me to investigate ;)
> >
> >Hopefully this entry in the list archive will be helpful to someone.
> >
> >Thanks,
> >Aaron
> >
> >
> >On Tue, Jan 10, 2012 at 2:51 PM, Mohammad Islam <mi...@yahoo.com>
> >wrote:
> >
> >> Hi Aaron,
> >> think you didn't put the namespace for your action correctly.=
> >>
> >>
> >> Please try with this:
> >>
> >> <action xmlns=3Duri:oozie:cascadingjruby-action:0.1 name=3D"cj-node">
> >>
> >> Regards,
> >> Mohammad
> >>
> >> ----- Original Message -----
> >> From: Aaron Beppu <ab...@etsy.com>
> >> To: oozie-users@incubator.apache.org; Mohammad Islam
> >><mi...@yahoo.com>
> >> Cc:
> >> Sent: Tuesday, January 10, 2012 11:04 AM
> >> Subject: Re: schema for new (extension) action type not being picked up
> >>
> >> Any suggestions? It looks like I'm not missing anything with
> >>minOccurs="1"
> >> and I haven't put in more than the maxOccurs for anything. Is there a
> >>way
> >> get oozie to tell which actions the schema service has loaded?
> >>
> >> On Tue, Jan 10, 2012 at 9:05 AM, Aaron Beppu <ab...@etsy.com> wrote:
> >>
> >> > Mohammad,
> >> >
> >> > Thanks for looking. Yeah, my bad. I guess I assumed because of the
> >>error
> >> > message that the issue was that the action wasn't registered properly
> >> > rather than that I was using it wrong -- but clearly one of my
> >> assumptions
> >> > is going to be violated before this is over. Here's the workflow
> >> definition
> >> > (unchanged except that I've subbed out the names of our s3 buckets) :
> >> >
> >> > <workflow-app xmlns="uri:oozie:workflow:0.2"
> >> name="cj-search-query-count">
> >> > <start to="cj-node"/>
> >> > <action name="cj-node">
> >> > <cascadingjruby>
> >> > <job-tracker>${jobTracker}</job-tracker>
> >> > <name-node>${nameNode}</name-node>
> >> > <prepare>
> >> > <delete
> >> >
> >>
> >>path="${nameNode}/user/${wf:user()}/${examplesRoot}/output-data/bigdata"/
> >>>
> >> > </prepare>
> >> > <configuration>
> >> > <property>
> >> > <name>mapred.job.queue.name</name>
> >> > <value>${queueName}</value>
> >> > </property>
> >> > <property>
> >> > <name>mapred.compress.map.output</name>
> >> > <value>true</value>
> >> > </property>
> >> > </configuration>
> >> > <arg>search_query_count.rb</arg>
> >> > <arg>today_path=2011_12_16</arg>
> >> > <arg>yesterday_path=2011_12_15</arg>
> >> > <arg>time_path=2011_12_16/16_33_31</arg>
> >> > <arg>task_name=analytics_search_query_count</arg>
> >> > <arg>job_signature=secretsecretsecret</arg>
> >> > <arg>end_date=2011_12_10</arg>
> >> > <arg>log_prefix=s3n://secretlogbucket</arg>
> >> > <arg>input_prefix=s3n://secretinputbucket</arg>
> >> > <arg>output_prefix=s3n://secretoutputbucket</arg>
> >> > <arg>index_prefix=s3n://secretindexbucket</arg>
> >> > <arg>hive_prefix=s3n://secrethivebucekt</arg>
> >> > <arg>graphite_prefix=s3n://secretgraphicbucket</arg>
> >> > <arg>splunk_prefix=s3n://secretsplunkbucket</arg>
> >> > <arg>dbdata_prefix=s3n://secretdbdatabucket</arg>
> >> > <arg>instance_type=c1.medium</arg>
> >> > <arg>instance_count=20</arg>
> >> > <arg>master_instance_type=c1.medium</arg>
> >> > </cascadingjruby>
> >> > <ok to="end"/>
> >> > <error to="fail"/>
> >> > </action>
> >> > <kill name="fail">
> >> > <message>BigData java action failed, error
> >> > message[${wf:errorMessage(wf:lastErrorNode())}]</message>
> >> > </kill>
> >> > <end name="end"/>
> >> > </workflow-app>
> >> >
> >> >
> >> >
> >> >
> >> > Thanks,
> >> > Aaron
> >> >
> >> > On Tue, Jan 10, 2012 at 4:46 AM, Mohammad Islam <mislam77@yahoo.com
> >> >wrote:
> >> >
> >> >> Would you please send at least the relevant definition of action from
> >> >> your workflow.xml?
> >> >>
> >> >> Thanks
> >> >> Mohammad
> >> >>
> >> >>
> >> >> ________________________________
> >> >> From: Aaron Beppu <ab...@etsy.com>
> >> >> To: oozie-users@incubator.apache.org
> >> >> Sent: Monday, January 9, 2012 5:04 PM
> >> >> Subject: Re: schema for new (extension) action type not being picked
> >>up
> >> >>
> >> >>
> >> >> Hi Alejandro,
> >> >>
> >> >> Thanks for looking. I'm both attaching and copy-pasting, as I'm not
> >>sure
> >> >> how the list handles attachments. I don't know that all of this is
> >> >> necessary, but I was following hive-action-0.2.xsd and
> >> sqoop-action-0.2.xsd
> >> >> closely as examples.
> >> >>
> >> >> <?xmlversion="1.0"encoding="UTF-8"?>
> >> >> <xs:schemaxmlns:xs="http://www.w3.org/2001/XMLSchema";;
> >> >>
> >> >>
> >>
> >>xmlns:cascadingjruby="uri:oozie:cascadingjruby-action:0.1"elementFormDefa
> >>ult="qualified"
> >> >> targetNamespace="uri:oozie:cascadingjruby-action:0.1">
> >> >>
> >> >> <xs:elementname="cascadingjruby"type="cascadingjruby:ACTION"/>
> >> >>
> >> >> <xs:complexTypename="ACTION">
> >> >> <xs:sequence>
> >> >>
> >> >>
> >>
> >><xs:elementname="job-tracker"type="xs:string"minOccurs="1"maxOccurs="1"/>
> >> >>
> >> >>
> >><xs:elementname="name-node"type="xs:string"minOccurs="1"maxOccurs="1"/>
> >> >>
> >> >>
> >>
> >><xs:elementname="prepare"type="cascadingjruby:PREPARE"minOccurs="0"maxOcc
> >>urs="1"/>
> >> >>
> >> >> <xs:elementname="job-xml"type="xs:string"minOccurs="0"maxOccurs="1"/>
> >> >>
> >> >>
> >>
> >><xs:elementname="configuration"type="cascadingjruby:CONFIGURATION"minOccu
> >>rs="0"maxOccurs="1"/>
> >> >>
> >> >>
> >>
> >><xs:elementname="arg"type="xs:string"minOccurs="0"maxOccurs="unbounded"/>
> >> >>
> >> >>
> >>
> >><xs:elementname="file"type="xs:string"minOccurs="0"maxOccurs="unbounded"/
> >>>
> >> >>
> >> >>
> >>
> >><xs:elementname="archive"type="xs:string"minOccurs="0"maxOccurs="unbounde
> >>d"/>
> >> >> </xs:sequence>
> >> >> </xs:complexType>
> >> >>
> >> >> <xs:complexTypename="CONFIGURATION">
> >> >> <xs:sequence>
> >> >>
> >> <xs:elementname="property"minOccurs="1"maxOccurs="unbounded">
> >> >> <xs:complexType>
> >> >> <xs:sequence>
> >> >> <xs:element name="name" minOccurs="1"
> >> >> maxOccurs="1" type="xs:string"/>
> >> >> <xs:element name="value" minOccurs="1"
> >> >> maxOccurs="1" type="xs:string"/>
> >> >> <xs:element name="description" minOccurs="0"
> >> >> maxOccurs="1" type="xs:string"/>
> >> >> </xs:sequence>
> >> >> </xs:complexType>
> >> >> </xs:element>
> >> >> </xs:sequence>
> >> >> </xs:complexType>
> >> >>
> >> >> <xs:complexTypename="PREPARE">
> >> >> <xs:sequence>
> >> >>
> >> >>
> >>
> >><xs:elementname="delete"type="cascadingjruby:DELETE"minOccurs="0"maxOccur
> >>s="unbounded"/>
> >> >>
> >> >>
> >>
> >><xs:elementname="mkdir"type="cascadingjruby:MKDIR"minOccurs="0"maxOccurs=
> >>"unbounded"/>
> >> >> </xs:sequence>
> >> >> </xs:complexType>
> >> >>
> >> >> <xs:complexTypename="DELETE">
> >> >> <xs:attributename="path"type="xs:string"use="required"/>
> >> >> </xs:complexType>
> >> >>
> >> >> <xs:complexTypename="MKDIR">
> >> >> <xs:attributename="path"type="xs:string"use="required"/>
> >> >> </xs:complexType>
> >> >>
> >> >> </xs:schema>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> On Mon, Jan 9, 2012 at 7:26 PM, Alejandro Abdelnur
> >><tu...@cloudera.com>
> >> >> wrote:
> >> >>
> >> >> Aaron,
> >> >> >
> >> >> >Would you mind attaching your XSD file?
> >> >> >
> >> >> >Thxs.
> >> >> >
> >> >> >Alejandro
> >> >> >
> >> >> >
> >> >> >On Mon, Jan 9, 2012 at 3:54 PM, Aaron Beppu <ab...@etsy.com>
> wrote:
> >> >> >
> >> >> >> Hi all. Confused oozie neophyte here, hoping for some guidance.
> >> >> >>
> >> >> >> I'm trying to create a new action type, following directions both
> >>on
> >> >> InfoQ
> >> >> >> [1], and in the official documentation [2]. So I've:
> >> >> >> 1. created the new action type, along with a schema,
> >> >> >> 2. built a jar containing the new code and schema,
> >> >> >> 3. added that jar to ${CATALINA_BASE}/webapps/oozie/WEB-INF/lib
> >>(and
> >> >> when
> >> >> >> that failed, using ".../bin/oozie-setup.sh -jars mine.jar")
> >> >> >> 4. added my new action type to
> >> >> >> oozie.service.ActionService.executor.ext.classes
> >> >> >> in oozie-site.xml
> >> >> >> 5. added my xsd filename to
> >> >> oozie.service.SchemaService.wf.ext.schemas, and
> >> >> >> to oozie.service.WorkflowSchemaService.ext.schemas (because the
> >>oozie
> >> >> docs
> >> >> >> and the InfoQ directions disagreed)
> >> >> >> 6. restarted my oozie server and
> >> >> >> 7. attempted to submit a workflow using the new action type.
> >> >> >>
> >> >> >> At this point, whenever I try to run the test workflow with the
> >>new
> >> >> action,
> >> >> >> I get this error :
> >> >> >>
> >> >> >> Error: E0701 : E0701: XML schema error, cvc-complex-type.2.4.a:
> >> Invalid
> >> >> >> content was found starting with element 'cascadingjruby'. One of
> >> >> >> '{"uri:oozie:workflow:0.2":map-reduce,
> >>"uri:oozie:workflow:0.2":pig,
> >> >> >> "uri:oozie:workflow:0.2":sub-workflow,
> >>"uri:oozie:workflow:0.2":fs,
> >> >> >> "uri:oozie:workflow:0.2":java,
> >>WC[##other:"uri:oozie:workflow:0.2"]}'
> >> >> is
> >> >> >> expected.
> >> >> >>
> >> >> >> A few notes :
> >> >> >> 1. I know that I'm naming the right .xsd file and that it's being
> >> >> picked up
> >> >> >> because earlier I had a typo in the filename in oozie-site.xml,
> >>and I
> >> >> got
> >> >> >> an *earlier* error when attempting to restart oozie [3], *before*
> >>I
> >> >> even
> >> >> >> tried to kick a job. So having eliminated that error, and not
> >>seeing
> >> >> >> anything break until I actually try to start a job, I'm reasonably
> >> >> >> confident that I'm not just making a filename problem -- and yet
> >>it
> >> >> appears
> >> >> >> not to know about my new action type.
> >> >> >> 2. "Extended" action types which I *can* use successfully (e.g.
> >> >> >> Hive/email/sqoop) are *not* on the list enumerated -- so whatever
> >> it's
> >> >> >> using to enumerate that list of "good" types, it's *not*
> >>identical to
> >> >> the
> >> >> >> actual set of actions that it's loaded and can execute.
> >> >> >> 3. The sole comment on that InfoQ article is from someone in
> >>October
> >> >> >> getting the same error. He didn't get any response from the
> >>article
> >> >> >> authors. So I'm guessing I'm making a common enough error that
> >> someone
> >> >> >> around here must know about the problem.
> >> >> >>
> >> >> >> What am I missing? I'm running cdh3u2 (so oozie version 2.3.2).
> >>I'm
> >> >> >> appending in the relevant chunk of my oozie-site.xml [4].
> >> >> >>
> >> >> >> Thanks so much,
> >> >> >> Aaron
> >> >> >>
> >> >> >> [1] http://www.infoq.com/articles/ExtendingOozie
> >> >> >> [2]
> >> >> http://archive.cloudera.com/cdh/3/oozie/DG_CustomActionExecutor.html
> >> >> >> [3] java.lang.IllegalArgumentException: resource
> >> >> >> [cascadingjruby-action-1.0.xsd] not found
> >> >> >> at
> >> >> >>
> >>org.apache.oozie.util.IOUtils.getResourceAsStream(IOUtils.java:100)
> >> >> >> at
> >> >> >>
> >> >>
> >>org.apache.oozie.service.SchemaService.loadSchema(SchemaService.java:69)
> >> >> >> at
> >> >> >> org.apache.oozie.service.SchemaService.init(SchemaService.java:85)
> >> >> >> at
> >> >> >>
> >> org.apache.oozie.service.Services.setServiceInternal(Services.java:307)
> >> >> >> at
> >> >> org.apache.oozie.service.Services.setService(Services.java:293)
> >> >> >> at
> >>org.apache.oozie.service.Services.init(Services.java:203)
> >> >> >> at
> >> >> >>
> >> >> >>
> >> >>
> >>
> >>org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader
> >>.java:37)
> >> >> >> at
> >> >> >>
> >> >> >>
> >> >>
> >>
> >>org.apache.catalina.core.StandardContext.listenerStart(StandardContext.ja
> >>va:4205)
> >> >> >> at
> >> >> >>
> >> >>
> >>
> >>org.apache.catalina.core.StandardContext.start(StandardContext.java:4704)
> >> >> >> at
> >> >> >>
> >> >> >>
> >> >>
> >>
> >>org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.jav
> >>a:799)
> >> >> >> at
> >> >> >>
> >> org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
> >> >> >> at
> >> >> >>
> >>org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
> >> >> >> at
> >> >> >>
> >> >> >>
> >> >>
> >>
> >>org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:6
> >>75)
> >> >> >> at
> >> >> >>
> >> >> >>
> >> >>
> >>
> >>org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:
> >>601)
> >> >> >> at
> >> >> >>
> >> org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:502)
> >> >> >> at
> >> >> >> org.apache.catalina.startup.HostConfig.start(HostConfig.java:1315)
> >> >> >> at
> >> >> >>
> >> >>
> >>
> >>org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324
> >>)
> >> >> >> at
> >> >> >>
> >> >> >>
> >> >>
> >>
> >>org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSup
> >>port.java:142)
> >> >> >> at
> >> >> >>
> >>org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1061)
> >> >> >> at
> >> >> >> org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
> >> >> >> at
> >> >> >>
> >>org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053)
> >> >> >> at
> >> >> >>
> >> org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
> >> >> >> at
> >> >> >>
> >> >>
> >>org.apache.catalina.core.StandardService.start(StandardService.java:525)
> >> >> >> at
> >> >> >>
> >> org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
> >> >> >> at
> >> org.apache.catalina.startup.Catalina.start(Catalina.java:595)
> >> >> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >>Method)
> >> >> >> at
> >> >> >>
> >> >> >>
> >> >>
> >>
> >>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java
> >>:39)
> >> >> >> at
> >> >> >>
> >> >> >>
> >> >>
> >>
> >>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI
> >>mpl.java:25)
> >> >> >> at java.lang.reflect.Method.invoke(Method.java:597)
> >> >> >> at
> >> >> org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
> >> >> >> at
> >> >> org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
> >> >> >> [4] <property>
> >> >> >>
> >><name>oozie.service.ActionService.executor.ext.classes</name>
> >> >> >> <value>
> >> >> >>
> >>org.apache.oozie.action.hadoop.CascadingJrubyIndependent,
> >> >> >> org.apache.oozie.action.hadoop.HiveActionExecutor,
> >> >> >> org.apache.oozie.action.hadoop.SqoopActionExecutor,
> >> >> >> org.apache.oozie.action.email.EmailActionExecutor
> >> >> >> </value>
> >> >> >> </property>
> >> >> >>
> >> >> >> <property>
> >> >> >> <name>oozie.service.SchemaService.wf.ext.schemas</name>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >>
> >>
> >><value>hive-action-0.2.xsd,sqoop-action-0.2.xsd,email-action-0.1.xsd,dist
> >>cp-action-0.1.xsd,cascadingjruby-action-1.0.xsd</value>
> >> >> >> </property>
> >> >> >> <property>
> >> >> >>
> >><name>oozie.service.WorkflowSchemaService.ext.schemas</name>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >>
> >>
> >><value>hive-action-0.2.xsd,sqoop-action-0.2.xsd,email-action-0.1.xsd,dist
> >>cp-action-0.1.xsd,cascadingjruby-action-1.0.xsd</value>
> >> >> >>
> >> >> >> </property>
> >> >> >>
> >> >> >
> >> >>
> >> >
> >> >
> >>
> >>
>
>
Re: schema for new (extension) action type not being picked up
Posted by Ken Ernst <ke...@mobileposse.com>.
Hi Aaron,
I have been following this thread and was hoping a solution would have
been found.
Although we are not using ruby with oozie yet I have some ruby code that I
would
definitely like to incorporate into oozie.
If you get this working please post it, it would be a great help!
Thanks.
Ken
On 1/10/12 4:13 PM, "Aaron Beppu" <ab...@etsy.com> wrote:
>Hi Mohammad,
>
>Thanks for the help! I played around with this a little, and the only way
>I
>got a different (I think better) outcome was by putting the xmlns
>attribute
>in the cascadingjruby node, rather than the action node, such that the
>start of the workflow definition looks like this now :
>
><workflow-app xmlns="uri:oozie:workflow:0.2" name="cj-search-query-count">
> <start to="cj-node"/>
> <action name="cj-node">
> <cascadingjruby xmlns="uri:oozie:cascadingjruby-action:0.1">
> ...
>
>
>Now the job launches, and provides a totally new array of exciting errors
>for me to investigate ;)
>
>Hopefully this entry in the list archive will be helpful to someone.
>
>Thanks,
>Aaron
>
>
>On Tue, Jan 10, 2012 at 2:51 PM, Mohammad Islam <mi...@yahoo.com>
>wrote:
>
>> Hi Aaron,
>> think you didn't put the namespace for your action correctly.=
>>
>>
>> Please try with this:
>>
>> <action xmlns=3Duri:oozie:cascadingjruby-action:0.1 name=3D"cj-node">
>>
>> Regards,
>> Mohammad
>>
>> ----- Original Message -----
>> From: Aaron Beppu <ab...@etsy.com>
>> To: oozie-users@incubator.apache.org; Mohammad Islam
>><mi...@yahoo.com>
>> Cc:
>> Sent: Tuesday, January 10, 2012 11:04 AM
>> Subject: Re: schema for new (extension) action type not being picked up
>>
>> Any suggestions? It looks like I'm not missing anything with
>>minOccurs="1"
>> and I haven't put in more than the maxOccurs for anything. Is there a
>>way
>> get oozie to tell which actions the schema service has loaded?
>>
>> On Tue, Jan 10, 2012 at 9:05 AM, Aaron Beppu <ab...@etsy.com> wrote:
>>
>> > Mohammad,
>> >
>> > Thanks for looking. Yeah, my bad. I guess I assumed because of the
>>error
>> > message that the issue was that the action wasn't registered properly
>> > rather than that I was using it wrong -- but clearly one of my
>> assumptions
>> > is going to be violated before this is over. Here's the workflow
>> definition
>> > (unchanged except that I've subbed out the names of our s3 buckets) :
>> >
>> > <workflow-app xmlns="uri:oozie:workflow:0.2"
>> name="cj-search-query-count">
>> > <start to="cj-node"/>
>> > <action name="cj-node">
>> > <cascadingjruby>
>> > <job-tracker>${jobTracker}</job-tracker>
>> > <name-node>${nameNode}</name-node>
>> > <prepare>
>> > <delete
>> >
>>
>>path="${nameNode}/user/${wf:user()}/${examplesRoot}/output-data/bigdata"/
>>>
>> > </prepare>
>> > <configuration>
>> > <property>
>> > <name>mapred.job.queue.name</name>
>> > <value>${queueName}</value>
>> > </property>
>> > <property>
>> > <name>mapred.compress.map.output</name>
>> > <value>true</value>
>> > </property>
>> > </configuration>
>> > <arg>search_query_count.rb</arg>
>> > <arg>today_path=2011_12_16</arg>
>> > <arg>yesterday_path=2011_12_15</arg>
>> > <arg>time_path=2011_12_16/16_33_31</arg>
>> > <arg>task_name=analytics_search_query_count</arg>
>> > <arg>job_signature=secretsecretsecret</arg>
>> > <arg>end_date=2011_12_10</arg>
>> > <arg>log_prefix=s3n://secretlogbucket</arg>
>> > <arg>input_prefix=s3n://secretinputbucket</arg>
>> > <arg>output_prefix=s3n://secretoutputbucket</arg>
>> > <arg>index_prefix=s3n://secretindexbucket</arg>
>> > <arg>hive_prefix=s3n://secrethivebucekt</arg>
>> > <arg>graphite_prefix=s3n://secretgraphicbucket</arg>
>> > <arg>splunk_prefix=s3n://secretsplunkbucket</arg>
>> > <arg>dbdata_prefix=s3n://secretdbdatabucket</arg>
>> > <arg>instance_type=c1.medium</arg>
>> > <arg>instance_count=20</arg>
>> > <arg>master_instance_type=c1.medium</arg>
>> > </cascadingjruby>
>> > <ok to="end"/>
>> > <error to="fail"/>
>> > </action>
>> > <kill name="fail">
>> > <message>BigData java action failed, error
>> > message[${wf:errorMessage(wf:lastErrorNode())}]</message>
>> > </kill>
>> > <end name="end"/>
>> > </workflow-app>
>> >
>> >
>> >
>> >
>> > Thanks,
>> > Aaron
>> >
>> > On Tue, Jan 10, 2012 at 4:46 AM, Mohammad Islam <mislam77@yahoo.com
>> >wrote:
>> >
>> >> Would you please send at least the relevant definition of action from
>> >> your workflow.xml?
>> >>
>> >> Thanks
>> >> Mohammad
>> >>
>> >>
>> >> ________________________________
>> >> From: Aaron Beppu <ab...@etsy.com>
>> >> To: oozie-users@incubator.apache.org
>> >> Sent: Monday, January 9, 2012 5:04 PM
>> >> Subject: Re: schema for new (extension) action type not being picked
>>up
>> >>
>> >>
>> >> Hi Alejandro,
>> >>
>> >> Thanks for looking. I'm both attaching and copy-pasting, as I'm not
>>sure
>> >> how the list handles attachments. I don't know that all of this is
>> >> necessary, but I was following hive-action-0.2.xsd and
>> sqoop-action-0.2.xsd
>> >> closely as examples.
>> >>
>> >> <?xmlversion="1.0"encoding="UTF-8"?>
>> >> <xs:schemaxmlns:xs="http://www.w3.org/2001/XMLSchema";;
>> >>
>> >>
>>
>>xmlns:cascadingjruby="uri:oozie:cascadingjruby-action:0.1"elementFormDefa
>>ult="qualified"
>> >> targetNamespace="uri:oozie:cascadingjruby-action:0.1">
>> >>
>> >> <xs:elementname="cascadingjruby"type="cascadingjruby:ACTION"/>
>> >>
>> >> <xs:complexTypename="ACTION">
>> >> <xs:sequence>
>> >>
>> >>
>>
>><xs:elementname="job-tracker"type="xs:string"minOccurs="1"maxOccurs="1"/>
>> >>
>> >>
>><xs:elementname="name-node"type="xs:string"minOccurs="1"maxOccurs="1"/>
>> >>
>> >>
>>
>><xs:elementname="prepare"type="cascadingjruby:PREPARE"minOccurs="0"maxOcc
>>urs="1"/>
>> >>
>> >> <xs:elementname="job-xml"type="xs:string"minOccurs="0"maxOccurs="1"/>
>> >>
>> >>
>>
>><xs:elementname="configuration"type="cascadingjruby:CONFIGURATION"minOccu
>>rs="0"maxOccurs="1"/>
>> >>
>> >>
>>
>><xs:elementname="arg"type="xs:string"minOccurs="0"maxOccurs="unbounded"/>
>> >>
>> >>
>>
>><xs:elementname="file"type="xs:string"minOccurs="0"maxOccurs="unbounded"/
>>>
>> >>
>> >>
>>
>><xs:elementname="archive"type="xs:string"minOccurs="0"maxOccurs="unbounde
>>d"/>
>> >> </xs:sequence>
>> >> </xs:complexType>
>> >>
>> >> <xs:complexTypename="CONFIGURATION">
>> >> <xs:sequence>
>> >>
>> <xs:elementname="property"minOccurs="1"maxOccurs="unbounded">
>> >> <xs:complexType>
>> >> <xs:sequence>
>> >> <xs:element name="name" minOccurs="1"
>> >> maxOccurs="1" type="xs:string"/>
>> >> <xs:element name="value" minOccurs="1"
>> >> maxOccurs="1" type="xs:string"/>
>> >> <xs:element name="description" minOccurs="0"
>> >> maxOccurs="1" type="xs:string"/>
>> >> </xs:sequence>
>> >> </xs:complexType>
>> >> </xs:element>
>> >> </xs:sequence>
>> >> </xs:complexType>
>> >>
>> >> <xs:complexTypename="PREPARE">
>> >> <xs:sequence>
>> >>
>> >>
>>
>><xs:elementname="delete"type="cascadingjruby:DELETE"minOccurs="0"maxOccur
>>s="unbounded"/>
>> >>
>> >>
>>
>><xs:elementname="mkdir"type="cascadingjruby:MKDIR"minOccurs="0"maxOccurs=
>>"unbounded"/>
>> >> </xs:sequence>
>> >> </xs:complexType>
>> >>
>> >> <xs:complexTypename="DELETE">
>> >> <xs:attributename="path"type="xs:string"use="required"/>
>> >> </xs:complexType>
>> >>
>> >> <xs:complexTypename="MKDIR">
>> >> <xs:attributename="path"type="xs:string"use="required"/>
>> >> </xs:complexType>
>> >>
>> >> </xs:schema>
>> >>
>> >>
>> >>
>> >>
>> >> On Mon, Jan 9, 2012 at 7:26 PM, Alejandro Abdelnur
>><tu...@cloudera.com>
>> >> wrote:
>> >>
>> >> Aaron,
>> >> >
>> >> >Would you mind attaching your XSD file?
>> >> >
>> >> >Thxs.
>> >> >
>> >> >Alejandro
>> >> >
>> >> >
>> >> >On Mon, Jan 9, 2012 at 3:54 PM, Aaron Beppu <ab...@etsy.com> wrote:
>> >> >
>> >> >> Hi all. Confused oozie neophyte here, hoping for some guidance.
>> >> >>
>> >> >> I'm trying to create a new action type, following directions both
>>on
>> >> InfoQ
>> >> >> [1], and in the official documentation [2]. So I've:
>> >> >> 1. created the new action type, along with a schema,
>> >> >> 2. built a jar containing the new code and schema,
>> >> >> 3. added that jar to ${CATALINA_BASE}/webapps/oozie/WEB-INF/lib
>>(and
>> >> when
>> >> >> that failed, using ".../bin/oozie-setup.sh -jars mine.jar")
>> >> >> 4. added my new action type to
>> >> >> oozie.service.ActionService.executor.ext.classes
>> >> >> in oozie-site.xml
>> >> >> 5. added my xsd filename to
>> >> oozie.service.SchemaService.wf.ext.schemas, and
>> >> >> to oozie.service.WorkflowSchemaService.ext.schemas (because the
>>oozie
>> >> docs
>> >> >> and the InfoQ directions disagreed)
>> >> >> 6. restarted my oozie server and
>> >> >> 7. attempted to submit a workflow using the new action type.
>> >> >>
>> >> >> At this point, whenever I try to run the test workflow with the
>>new
>> >> action,
>> >> >> I get this error :
>> >> >>
>> >> >> Error: E0701 : E0701: XML schema error, cvc-complex-type.2.4.a:
>> Invalid
>> >> >> content was found starting with element 'cascadingjruby'. One of
>> >> >> '{"uri:oozie:workflow:0.2":map-reduce,
>>"uri:oozie:workflow:0.2":pig,
>> >> >> "uri:oozie:workflow:0.2":sub-workflow,
>>"uri:oozie:workflow:0.2":fs,
>> >> >> "uri:oozie:workflow:0.2":java,
>>WC[##other:"uri:oozie:workflow:0.2"]}'
>> >> is
>> >> >> expected.
>> >> >>
>> >> >> A few notes :
>> >> >> 1. I know that I'm naming the right .xsd file and that it's being
>> >> picked up
>> >> >> because earlier I had a typo in the filename in oozie-site.xml,
>>and I
>> >> got
>> >> >> an *earlier* error when attempting to restart oozie [3], *before*
>>I
>> >> even
>> >> >> tried to kick a job. So having eliminated that error, and not
>>seeing
>> >> >> anything break until I actually try to start a job, I'm reasonably
>> >> >> confident that I'm not just making a filename problem -- and yet
>>it
>> >> appears
>> >> >> not to know about my new action type.
>> >> >> 2. "Extended" action types which I *can* use successfully (e.g.
>> >> >> Hive/email/sqoop) are *not* on the list enumerated -- so whatever
>> it's
>> >> >> using to enumerate that list of "good" types, it's *not*
>>identical to
>> >> the
>> >> >> actual set of actions that it's loaded and can execute.
>> >> >> 3. The sole comment on that InfoQ article is from someone in
>>October
>> >> >> getting the same error. He didn't get any response from the
>>article
>> >> >> authors. So I'm guessing I'm making a common enough error that
>> someone
>> >> >> around here must know about the problem.
>> >> >>
>> >> >> What am I missing? I'm running cdh3u2 (so oozie version 2.3.2).
>>I'm
>> >> >> appending in the relevant chunk of my oozie-site.xml [4].
>> >> >>
>> >> >> Thanks so much,
>> >> >> Aaron
>> >> >>
>> >> >> [1] http://www.infoq.com/articles/ExtendingOozie
>> >> >> [2]
>> >> http://archive.cloudera.com/cdh/3/oozie/DG_CustomActionExecutor.html
>> >> >> [3] java.lang.IllegalArgumentException: resource
>> >> >> [cascadingjruby-action-1.0.xsd] not found
>> >> >> at
>> >> >>
>>org.apache.oozie.util.IOUtils.getResourceAsStream(IOUtils.java:100)
>> >> >> at
>> >> >>
>> >>
>>org.apache.oozie.service.SchemaService.loadSchema(SchemaService.java:69)
>> >> >> at
>> >> >> org.apache.oozie.service.SchemaService.init(SchemaService.java:85)
>> >> >> at
>> >> >>
>> org.apache.oozie.service.Services.setServiceInternal(Services.java:307)
>> >> >> at
>> >> org.apache.oozie.service.Services.setService(Services.java:293)
>> >> >> at
>>org.apache.oozie.service.Services.init(Services.java:203)
>> >> >> at
>> >> >>
>> >> >>
>> >>
>>
>>org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader
>>.java:37)
>> >> >> at
>> >> >>
>> >> >>
>> >>
>>
>>org.apache.catalina.core.StandardContext.listenerStart(StandardContext.ja
>>va:4205)
>> >> >> at
>> >> >>
>> >>
>>
>>org.apache.catalina.core.StandardContext.start(StandardContext.java:4704)
>> >> >> at
>> >> >>
>> >> >>
>> >>
>>
>>org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.jav
>>a:799)
>> >> >> at
>> >> >>
>> org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
>> >> >> at
>> >> >>
>>org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
>> >> >> at
>> >> >>
>> >> >>
>> >>
>>
>>org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:6
>>75)
>> >> >> at
>> >> >>
>> >> >>
>> >>
>>
>>org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:
>>601)
>> >> >> at
>> >> >>
>> org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:502)
>> >> >> at
>> >> >> org.apache.catalina.startup.HostConfig.start(HostConfig.java:1315)
>> >> >> at
>> >> >>
>> >>
>>
>>org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324
>>)
>> >> >> at
>> >> >>
>> >> >>
>> >>
>>
>>org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSup
>>port.java:142)
>> >> >> at
>> >> >>
>>org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1061)
>> >> >> at
>> >> >> org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
>> >> >> at
>> >> >>
>>org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053)
>> >> >> at
>> >> >>
>> org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
>> >> >> at
>> >> >>
>> >>
>>org.apache.catalina.core.StandardService.start(StandardService.java:525)
>> >> >> at
>> >> >>
>> org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
>> >> >> at
>> org.apache.catalina.startup.Catalina.start(Catalina.java:595)
>> >> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>Method)
>> >> >> at
>> >> >>
>> >> >>
>> >>
>>
>>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java
>>:39)
>> >> >> at
>> >> >>
>> >> >>
>> >>
>>
>>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI
>>mpl.java:25)
>> >> >> at java.lang.reflect.Method.invoke(Method.java:597)
>> >> >> at
>> >> org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
>> >> >> at
>> >> org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
>> >> >> [4] <property>
>> >> >>
>><name>oozie.service.ActionService.executor.ext.classes</name>
>> >> >> <value>
>> >> >>
>>org.apache.oozie.action.hadoop.CascadingJrubyIndependent,
>> >> >> org.apache.oozie.action.hadoop.HiveActionExecutor,
>> >> >> org.apache.oozie.action.hadoop.SqoopActionExecutor,
>> >> >> org.apache.oozie.action.email.EmailActionExecutor
>> >> >> </value>
>> >> >> </property>
>> >> >>
>> >> >> <property>
>> >> >> <name>oozie.service.SchemaService.wf.ext.schemas</name>
>> >> >>
>> >> >>
>> >> >>
>> >>
>>
>><value>hive-action-0.2.xsd,sqoop-action-0.2.xsd,email-action-0.1.xsd,dist
>>cp-action-0.1.xsd,cascadingjruby-action-1.0.xsd</value>
>> >> >> </property>
>> >> >> <property>
>> >> >>
>><name>oozie.service.WorkflowSchemaService.ext.schemas</name>
>> >> >>
>> >> >>
>> >> >>
>> >>
>>
>><value>hive-action-0.2.xsd,sqoop-action-0.2.xsd,email-action-0.1.xsd,dist
>>cp-action-0.1.xsd,cascadingjruby-action-1.0.xsd</value>
>> >> >>
>> >> >> </property>
>> >> >>
>> >> >
>> >>
>> >
>> >
>>
>>
Re: schema for new (extension) action type not being picked up
Posted by Aaron Beppu <ab...@etsy.com>.
Hi Mohammad,
Thanks for the help! I played around with this a little, and the only way I
got a different (I think better) outcome was by putting the xmlns attribute
in the cascadingjruby node, rather than the action node, such that the
start of the workflow definition looks like this now :
<workflow-app xmlns="uri:oozie:workflow:0.2" name="cj-search-query-count">
<start to="cj-node"/>
<action name="cj-node">
<cascadingjruby xmlns="uri:oozie:cascadingjruby-action:0.1">
...
Now the job launches, and provides a totally new array of exciting errors
for me to investigate ;)
Hopefully this entry in the list archive will be helpful to someone.
Thanks,
Aaron
On Tue, Jan 10, 2012 at 2:51 PM, Mohammad Islam <mi...@yahoo.com> wrote:
> Hi Aaron,
> think you didn't put the namespace for your action correctly.=
>
>
> Please try with this:
>
> <action xmlns=3Duri:oozie:cascadingjruby-action:0.1 name=3D"cj-node">
>
> Regards,
> Mohammad
>
> ----- Original Message -----
> From: Aaron Beppu <ab...@etsy.com>
> To: oozie-users@incubator.apache.org; Mohammad Islam <mi...@yahoo.com>
> Cc:
> Sent: Tuesday, January 10, 2012 11:04 AM
> Subject: Re: schema for new (extension) action type not being picked up
>
> Any suggestions? It looks like I'm not missing anything with minOccurs="1"
> and I haven't put in more than the maxOccurs for anything. Is there a way
> get oozie to tell which actions the schema service has loaded?
>
> On Tue, Jan 10, 2012 at 9:05 AM, Aaron Beppu <ab...@etsy.com> wrote:
>
> > Mohammad,
> >
> > Thanks for looking. Yeah, my bad. I guess I assumed because of the error
> > message that the issue was that the action wasn't registered properly
> > rather than that I was using it wrong -- but clearly one of my
> assumptions
> > is going to be violated before this is over. Here's the workflow
> definition
> > (unchanged except that I've subbed out the names of our s3 buckets) :
> >
> > <workflow-app xmlns="uri:oozie:workflow:0.2"
> name="cj-search-query-count">
> > <start to="cj-node"/>
> > <action name="cj-node">
> > <cascadingjruby>
> > <job-tracker>${jobTracker}</job-tracker>
> > <name-node>${nameNode}</name-node>
> > <prepare>
> > <delete
> >
> path="${nameNode}/user/${wf:user()}/${examplesRoot}/output-data/bigdata"/>
> > </prepare>
> > <configuration>
> > <property>
> > <name>mapred.job.queue.name</name>
> > <value>${queueName}</value>
> > </property>
> > <property>
> > <name>mapred.compress.map.output</name>
> > <value>true</value>
> > </property>
> > </configuration>
> > <arg>search_query_count.rb</arg>
> > <arg>today_path=2011_12_16</arg>
> > <arg>yesterday_path=2011_12_15</arg>
> > <arg>time_path=2011_12_16/16_33_31</arg>
> > <arg>task_name=analytics_search_query_count</arg>
> > <arg>job_signature=secretsecretsecret</arg>
> > <arg>end_date=2011_12_10</arg>
> > <arg>log_prefix=s3n://secretlogbucket</arg>
> > <arg>input_prefix=s3n://secretinputbucket</arg>
> > <arg>output_prefix=s3n://secretoutputbucket</arg>
> > <arg>index_prefix=s3n://secretindexbucket</arg>
> > <arg>hive_prefix=s3n://secrethivebucekt</arg>
> > <arg>graphite_prefix=s3n://secretgraphicbucket</arg>
> > <arg>splunk_prefix=s3n://secretsplunkbucket</arg>
> > <arg>dbdata_prefix=s3n://secretdbdatabucket</arg>
> > <arg>instance_type=c1.medium</arg>
> > <arg>instance_count=20</arg>
> > <arg>master_instance_type=c1.medium</arg>
> > </cascadingjruby>
> > <ok to="end"/>
> > <error to="fail"/>
> > </action>
> > <kill name="fail">
> > <message>BigData java action failed, error
> > message[${wf:errorMessage(wf:lastErrorNode())}]</message>
> > </kill>
> > <end name="end"/>
> > </workflow-app>
> >
> >
> >
> >
> > Thanks,
> > Aaron
> >
> > On Tue, Jan 10, 2012 at 4:46 AM, Mohammad Islam <mislam77@yahoo.com
> >wrote:
> >
> >> Would you please send at least the relevant definition of action from
> >> your workflow.xml?
> >>
> >> Thanks
> >> Mohammad
> >>
> >>
> >> ________________________________
> >> From: Aaron Beppu <ab...@etsy.com>
> >> To: oozie-users@incubator.apache.org
> >> Sent: Monday, January 9, 2012 5:04 PM
> >> Subject: Re: schema for new (extension) action type not being picked up
> >>
> >>
> >> Hi Alejandro,
> >>
> >> Thanks for looking. I'm both attaching and copy-pasting, as I'm not sure
> >> how the list handles attachments. I don't know that all of this is
> >> necessary, but I was following hive-action-0.2.xsd and
> sqoop-action-0.2.xsd
> >> closely as examples.
> >>
> >> <?xmlversion="1.0"encoding="UTF-8"?>
> >> <xs:schemaxmlns:xs="http://www.w3.org/2001/XMLSchema";;
> >>
> >>
> xmlns:cascadingjruby="uri:oozie:cascadingjruby-action:0.1"elementFormDefault="qualified"
> >> targetNamespace="uri:oozie:cascadingjruby-action:0.1">
> >>
> >> <xs:elementname="cascadingjruby"type="cascadingjruby:ACTION"/>
> >>
> >> <xs:complexTypename="ACTION">
> >> <xs:sequence>
> >>
> >>
> <xs:elementname="job-tracker"type="xs:string"minOccurs="1"maxOccurs="1"/>
> >>
> >> <xs:elementname="name-node"type="xs:string"minOccurs="1"maxOccurs="1"/>
> >>
> >>
> <xs:elementname="prepare"type="cascadingjruby:PREPARE"minOccurs="0"maxOccurs="1"/>
> >>
> >> <xs:elementname="job-xml"type="xs:string"minOccurs="0"maxOccurs="1"/>
> >>
> >>
> <xs:elementname="configuration"type="cascadingjruby:CONFIGURATION"minOccurs="0"maxOccurs="1"/>
> >>
> >>
> <xs:elementname="arg"type="xs:string"minOccurs="0"maxOccurs="unbounded"/>
> >>
> >>
> <xs:elementname="file"type="xs:string"minOccurs="0"maxOccurs="unbounded"/>
> >>
> >>
> <xs:elementname="archive"type="xs:string"minOccurs="0"maxOccurs="unbounded"/>
> >> </xs:sequence>
> >> </xs:complexType>
> >>
> >> <xs:complexTypename="CONFIGURATION">
> >> <xs:sequence>
> >>
> <xs:elementname="property"minOccurs="1"maxOccurs="unbounded">
> >> <xs:complexType>
> >> <xs:sequence>
> >> <xs:element name="name" minOccurs="1"
> >> maxOccurs="1" type="xs:string"/>
> >> <xs:element name="value" minOccurs="1"
> >> maxOccurs="1" type="xs:string"/>
> >> <xs:element name="description" minOccurs="0"
> >> maxOccurs="1" type="xs:string"/>
> >> </xs:sequence>
> >> </xs:complexType>
> >> </xs:element>
> >> </xs:sequence>
> >> </xs:complexType>
> >>
> >> <xs:complexTypename="PREPARE">
> >> <xs:sequence>
> >>
> >>
> <xs:elementname="delete"type="cascadingjruby:DELETE"minOccurs="0"maxOccurs="unbounded"/>
> >>
> >>
> <xs:elementname="mkdir"type="cascadingjruby:MKDIR"minOccurs="0"maxOccurs="unbounded"/>
> >> </xs:sequence>
> >> </xs:complexType>
> >>
> >> <xs:complexTypename="DELETE">
> >> <xs:attributename="path"type="xs:string"use="required"/>
> >> </xs:complexType>
> >>
> >> <xs:complexTypename="MKDIR">
> >> <xs:attributename="path"type="xs:string"use="required"/>
> >> </xs:complexType>
> >>
> >> </xs:schema>
> >>
> >>
> >>
> >>
> >> On Mon, Jan 9, 2012 at 7:26 PM, Alejandro Abdelnur <tu...@cloudera.com>
> >> wrote:
> >>
> >> Aaron,
> >> >
> >> >Would you mind attaching your XSD file?
> >> >
> >> >Thxs.
> >> >
> >> >Alejandro
> >> >
> >> >
> >> >On Mon, Jan 9, 2012 at 3:54 PM, Aaron Beppu <ab...@etsy.com> wrote:
> >> >
> >> >> Hi all. Confused oozie neophyte here, hoping for some guidance.
> >> >>
> >> >> I'm trying to create a new action type, following directions both on
> >> InfoQ
> >> >> [1], and in the official documentation [2]. So I've:
> >> >> 1. created the new action type, along with a schema,
> >> >> 2. built a jar containing the new code and schema,
> >> >> 3. added that jar to ${CATALINA_BASE}/webapps/oozie/WEB-INF/lib (and
> >> when
> >> >> that failed, using ".../bin/oozie-setup.sh -jars mine.jar")
> >> >> 4. added my new action type to
> >> >> oozie.service.ActionService.executor.ext.classes
> >> >> in oozie-site.xml
> >> >> 5. added my xsd filename to
> >> oozie.service.SchemaService.wf.ext.schemas, and
> >> >> to oozie.service.WorkflowSchemaService.ext.schemas (because the oozie
> >> docs
> >> >> and the InfoQ directions disagreed)
> >> >> 6. restarted my oozie server and
> >> >> 7. attempted to submit a workflow using the new action type.
> >> >>
> >> >> At this point, whenever I try to run the test workflow with the new
> >> action,
> >> >> I get this error :
> >> >>
> >> >> Error: E0701 : E0701: XML schema error, cvc-complex-type.2.4.a:
> Invalid
> >> >> content was found starting with element 'cascadingjruby'. One of
> >> >> '{"uri:oozie:workflow:0.2":map-reduce, "uri:oozie:workflow:0.2":pig,
> >> >> "uri:oozie:workflow:0.2":sub-workflow, "uri:oozie:workflow:0.2":fs,
> >> >> "uri:oozie:workflow:0.2":java, WC[##other:"uri:oozie:workflow:0.2"]}'
> >> is
> >> >> expected.
> >> >>
> >> >> A few notes :
> >> >> 1. I know that I'm naming the right .xsd file and that it's being
> >> picked up
> >> >> because earlier I had a typo in the filename in oozie-site.xml, and I
> >> got
> >> >> an *earlier* error when attempting to restart oozie [3], *before* I
> >> even
> >> >> tried to kick a job. So having eliminated that error, and not seeing
> >> >> anything break until I actually try to start a job, I'm reasonably
> >> >> confident that I'm not just making a filename problem -- and yet it
> >> appears
> >> >> not to know about my new action type.
> >> >> 2. "Extended" action types which I *can* use successfully (e.g.
> >> >> Hive/email/sqoop) are *not* on the list enumerated -- so whatever
> it's
> >> >> using to enumerate that list of "good" types, it's *not* identical to
> >> the
> >> >> actual set of actions that it's loaded and can execute.
> >> >> 3. The sole comment on that InfoQ article is from someone in October
> >> >> getting the same error. He didn't get any response from the article
> >> >> authors. So I'm guessing I'm making a common enough error that
> someone
> >> >> around here must know about the problem.
> >> >>
> >> >> What am I missing? I'm running cdh3u2 (so oozie version 2.3.2). I'm
> >> >> appending in the relevant chunk of my oozie-site.xml [4].
> >> >>
> >> >> Thanks so much,
> >> >> Aaron
> >> >>
> >> >> [1] http://www.infoq.com/articles/ExtendingOozie
> >> >> [2]
> >> http://archive.cloudera.com/cdh/3/oozie/DG_CustomActionExecutor.html
> >> >> [3] java.lang.IllegalArgumentException: resource
> >> >> [cascadingjruby-action-1.0.xsd] not found
> >> >> at
> >> >> org.apache.oozie.util.IOUtils.getResourceAsStream(IOUtils.java:100)
> >> >> at
> >> >>
> >> org.apache.oozie.service.SchemaService.loadSchema(SchemaService.java:69)
> >> >> at
> >> >> org.apache.oozie.service.SchemaService.init(SchemaService.java:85)
> >> >> at
> >> >>
> org.apache.oozie.service.Services.setServiceInternal(Services.java:307)
> >> >> at
> >> org.apache.oozie.service.Services.setService(Services.java:293)
> >> >> at org.apache.oozie.service.Services.init(Services.java:203)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:37)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4205)
> >> >> at
> >> >>
> >>
> org.apache.catalina.core.StandardContext.start(StandardContext.java:4704)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
> >> >> at
> >> >>
> org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
> >> >> at
> >> >> org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:675)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:601)
> >> >> at
> >> >>
> org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:502)
> >> >> at
> >> >> org.apache.catalina.startup.HostConfig.start(HostConfig.java:1315)
> >> >> at
> >> >>
> >>
> org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
> >> >> at
> >> >> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1061)
> >> >> at
> >> >> org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
> >> >> at
> >> >> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053)
> >> >> at
> >> >>
> org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
> >> >> at
> >> >>
> >> org.apache.catalina.core.StandardService.start(StandardService.java:525)
> >> >> at
> >> >>
> org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
> >> >> at
> org.apache.catalina.startup.Catalina.start(Catalina.java:595)
> >> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> >> at
> >> >>
> >> >>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> >> at
> >> >>
> >> >>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >> at java.lang.reflect.Method.invoke(Method.java:597)
> >> >> at
> >> org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
> >> >> at
> >> org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
> >> >> [4] <property>
> >> >> <name>oozie.service.ActionService.executor.ext.classes</name>
> >> >> <value>
> >> >> org.apache.oozie.action.hadoop.CascadingJrubyIndependent,
> >> >> org.apache.oozie.action.hadoop.HiveActionExecutor,
> >> >> org.apache.oozie.action.hadoop.SqoopActionExecutor,
> >> >> org.apache.oozie.action.email.EmailActionExecutor
> >> >> </value>
> >> >> </property>
> >> >>
> >> >> <property>
> >> >> <name>oozie.service.SchemaService.wf.ext.schemas</name>
> >> >>
> >> >>
> >> >>
> >>
> <value>hive-action-0.2.xsd,sqoop-action-0.2.xsd,email-action-0.1.xsd,distcp-action-0.1.xsd,cascadingjruby-action-1.0.xsd</value>
> >> >> </property>
> >> >> <property>
> >> >> <name>oozie.service.WorkflowSchemaService.ext.schemas</name>
> >> >>
> >> >>
> >> >>
> >>
> <value>hive-action-0.2.xsd,sqoop-action-0.2.xsd,email-action-0.1.xsd,distcp-action-0.1.xsd,cascadingjruby-action-1.0.xsd</value>
> >> >>
> >> >> </property>
> >> >>
> >> >
> >>
> >
> >
>
>
Re: schema for new (extension) action type not being picked up
Posted by Mohammad Islam <mi...@yahoo.com>.
Hi Aaron,
think you didn't put the namespace for your action correctly.=
Please try with this:
<action xmlns=3Duri:oozie:cascadingjruby-action:0.1 name=3D"cj-node">
Regards,
Mohammad
----- Original Message -----
From: Aaron Beppu <ab...@etsy.com>
To: oozie-users@incubator.apache.org; Mohammad Islam <mi...@yahoo.com>
Cc:
Sent: Tuesday, January 10, 2012 11:04 AM
Subject: Re: schema for new (extension) action type not being picked up
Any suggestions? It looks like I'm not missing anything with minOccurs="1"
and I haven't put in more than the maxOccurs for anything. Is there a way
get oozie to tell which actions the schema service has loaded?
On Tue, Jan 10, 2012 at 9:05 AM, Aaron Beppu <ab...@etsy.com> wrote:
> Mohammad,
>
> Thanks for looking. Yeah, my bad. I guess I assumed because of the error
> message that the issue was that the action wasn't registered properly
> rather than that I was using it wrong -- but clearly one of my assumptions
> is going to be violated before this is over. Here's the workflow definition
> (unchanged except that I've subbed out the names of our s3 buckets) :
>
> <workflow-app xmlns="uri:oozie:workflow:0.2" name="cj-search-query-count">
> <start to="cj-node"/>
> <action name="cj-node">
> <cascadingjruby>
> <job-tracker>${jobTracker}</job-tracker>
> <name-node>${nameNode}</name-node>
> <prepare>
> <delete
> path="${nameNode}/user/${wf:user()}/${examplesRoot}/output-data/bigdata"/>
> </prepare>
> <configuration>
> <property>
> <name>mapred.job.queue.name</name>
> <value>${queueName}</value>
> </property>
> <property>
> <name>mapred.compress.map.output</name>
> <value>true</value>
> </property>
> </configuration>
> <arg>search_query_count.rb</arg>
> <arg>today_path=2011_12_16</arg>
> <arg>yesterday_path=2011_12_15</arg>
> <arg>time_path=2011_12_16/16_33_31</arg>
> <arg>task_name=analytics_search_query_count</arg>
> <arg>job_signature=secretsecretsecret</arg>
> <arg>end_date=2011_12_10</arg>
> <arg>log_prefix=s3n://secretlogbucket</arg>
> <arg>input_prefix=s3n://secretinputbucket</arg>
> <arg>output_prefix=s3n://secretoutputbucket</arg>
> <arg>index_prefix=s3n://secretindexbucket</arg>
> <arg>hive_prefix=s3n://secrethivebucekt</arg>
> <arg>graphite_prefix=s3n://secretgraphicbucket</arg>
> <arg>splunk_prefix=s3n://secretsplunkbucket</arg>
> <arg>dbdata_prefix=s3n://secretdbdatabucket</arg>
> <arg>instance_type=c1.medium</arg>
> <arg>instance_count=20</arg>
> <arg>master_instance_type=c1.medium</arg>
> </cascadingjruby>
> <ok to="end"/>
> <error to="fail"/>
> </action>
> <kill name="fail">
> <message>BigData java action failed, error
> message[${wf:errorMessage(wf:lastErrorNode())}]</message>
> </kill>
> <end name="end"/>
> </workflow-app>
>
>
>
>
> Thanks,
> Aaron
>
> On Tue, Jan 10, 2012 at 4:46 AM, Mohammad Islam <mi...@yahoo.com>wrote:
>
>> Would you please send at least the relevant definition of action from
>> your workflow.xml?
>>
>> Thanks
>> Mohammad
>>
>>
>> ________________________________
>> From: Aaron Beppu <ab...@etsy.com>
>> To: oozie-users@incubator.apache.org
>> Sent: Monday, January 9, 2012 5:04 PM
>> Subject: Re: schema for new (extension) action type not being picked up
>>
>>
>> Hi Alejandro,
>>
>> Thanks for looking. I'm both attaching and copy-pasting, as I'm not sure
>> how the list handles attachments. I don't know that all of this is
>> necessary, but I was following hive-action-0.2.xsd and sqoop-action-0.2.xsd
>> closely as examples.
>>
>> <?xmlversion="1.0"encoding="UTF-8"?>
>> <xs:schemaxmlns:xs="http://www.w3.org/2001/XMLSchema";;
>>
>> xmlns:cascadingjruby="uri:oozie:cascadingjruby-action:0.1"elementFormDefault="qualified"
>> targetNamespace="uri:oozie:cascadingjruby-action:0.1">
>>
>> <xs:elementname="cascadingjruby"type="cascadingjruby:ACTION"/>
>>
>> <xs:complexTypename="ACTION">
>> <xs:sequence>
>>
>> <xs:elementname="job-tracker"type="xs:string"minOccurs="1"maxOccurs="1"/>
>>
>> <xs:elementname="name-node"type="xs:string"minOccurs="1"maxOccurs="1"/>
>>
>> <xs:elementname="prepare"type="cascadingjruby:PREPARE"minOccurs="0"maxOccurs="1"/>
>>
>> <xs:elementname="job-xml"type="xs:string"minOccurs="0"maxOccurs="1"/>
>>
>> <xs:elementname="configuration"type="cascadingjruby:CONFIGURATION"minOccurs="0"maxOccurs="1"/>
>>
>> <xs:elementname="arg"type="xs:string"minOccurs="0"maxOccurs="unbounded"/>
>>
>> <xs:elementname="file"type="xs:string"minOccurs="0"maxOccurs="unbounded"/>
>>
>> <xs:elementname="archive"type="xs:string"minOccurs="0"maxOccurs="unbounded"/>
>> </xs:sequence>
>> </xs:complexType>
>>
>> <xs:complexTypename="CONFIGURATION">
>> <xs:sequence>
>> <xs:elementname="property"minOccurs="1"maxOccurs="unbounded">
>> <xs:complexType>
>> <xs:sequence>
>> <xs:element name="name" minOccurs="1"
>> maxOccurs="1" type="xs:string"/>
>> <xs:element name="value" minOccurs="1"
>> maxOccurs="1" type="xs:string"/>
>> <xs:element name="description" minOccurs="0"
>> maxOccurs="1" type="xs:string"/>
>> </xs:sequence>
>> </xs:complexType>
>> </xs:element>
>> </xs:sequence>
>> </xs:complexType>
>>
>> <xs:complexTypename="PREPARE">
>> <xs:sequence>
>>
>> <xs:elementname="delete"type="cascadingjruby:DELETE"minOccurs="0"maxOccurs="unbounded"/>
>>
>> <xs:elementname="mkdir"type="cascadingjruby:MKDIR"minOccurs="0"maxOccurs="unbounded"/>
>> </xs:sequence>
>> </xs:complexType>
>>
>> <xs:complexTypename="DELETE">
>> <xs:attributename="path"type="xs:string"use="required"/>
>> </xs:complexType>
>>
>> <xs:complexTypename="MKDIR">
>> <xs:attributename="path"type="xs:string"use="required"/>
>> </xs:complexType>
>>
>> </xs:schema>
>>
>>
>>
>>
>> On Mon, Jan 9, 2012 at 7:26 PM, Alejandro Abdelnur <tu...@cloudera.com>
>> wrote:
>>
>> Aaron,
>> >
>> >Would you mind attaching your XSD file?
>> >
>> >Thxs.
>> >
>> >Alejandro
>> >
>> >
>> >On Mon, Jan 9, 2012 at 3:54 PM, Aaron Beppu <ab...@etsy.com> wrote:
>> >
>> >> Hi all. Confused oozie neophyte here, hoping for some guidance.
>> >>
>> >> I'm trying to create a new action type, following directions both on
>> InfoQ
>> >> [1], and in the official documentation [2]. So I've:
>> >> 1. created the new action type, along with a schema,
>> >> 2. built a jar containing the new code and schema,
>> >> 3. added that jar to ${CATALINA_BASE}/webapps/oozie/WEB-INF/lib (and
>> when
>> >> that failed, using ".../bin/oozie-setup.sh -jars mine.jar")
>> >> 4. added my new action type to
>> >> oozie.service.ActionService.executor.ext.classes
>> >> in oozie-site.xml
>> >> 5. added my xsd filename to
>> oozie.service.SchemaService.wf.ext.schemas, and
>> >> to oozie.service.WorkflowSchemaService.ext.schemas (because the oozie
>> docs
>> >> and the InfoQ directions disagreed)
>> >> 6. restarted my oozie server and
>> >> 7. attempted to submit a workflow using the new action type.
>> >>
>> >> At this point, whenever I try to run the test workflow with the new
>> action,
>> >> I get this error :
>> >>
>> >> Error: E0701 : E0701: XML schema error, cvc-complex-type.2.4.a: Invalid
>> >> content was found starting with element 'cascadingjruby'. One of
>> >> '{"uri:oozie:workflow:0.2":map-reduce, "uri:oozie:workflow:0.2":pig,
>> >> "uri:oozie:workflow:0.2":sub-workflow, "uri:oozie:workflow:0.2":fs,
>> >> "uri:oozie:workflow:0.2":java, WC[##other:"uri:oozie:workflow:0.2"]}'
>> is
>> >> expected.
>> >>
>> >> A few notes :
>> >> 1. I know that I'm naming the right .xsd file and that it's being
>> picked up
>> >> because earlier I had a typo in the filename in oozie-site.xml, and I
>> got
>> >> an *earlier* error when attempting to restart oozie [3], *before* I
>> even
>> >> tried to kick a job. So having eliminated that error, and not seeing
>> >> anything break until I actually try to start a job, I'm reasonably
>> >> confident that I'm not just making a filename problem -- and yet it
>> appears
>> >> not to know about my new action type.
>> >> 2. "Extended" action types which I *can* use successfully (e.g.
>> >> Hive/email/sqoop) are *not* on the list enumerated -- so whatever it's
>> >> using to enumerate that list of "good" types, it's *not* identical to
>> the
>> >> actual set of actions that it's loaded and can execute.
>> >> 3. The sole comment on that InfoQ article is from someone in October
>> >> getting the same error. He didn't get any response from the article
>> >> authors. So I'm guessing I'm making a common enough error that someone
>> >> around here must know about the problem.
>> >>
>> >> What am I missing? I'm running cdh3u2 (so oozie version 2.3.2). I'm
>> >> appending in the relevant chunk of my oozie-site.xml [4].
>> >>
>> >> Thanks so much,
>> >> Aaron
>> >>
>> >> [1] http://www.infoq.com/articles/ExtendingOozie
>> >> [2]
>> http://archive.cloudera.com/cdh/3/oozie/DG_CustomActionExecutor.html
>> >> [3] java.lang.IllegalArgumentException: resource
>> >> [cascadingjruby-action-1.0.xsd] not found
>> >> at
>> >> org.apache.oozie.util.IOUtils.getResourceAsStream(IOUtils.java:100)
>> >> at
>> >>
>> org.apache.oozie.service.SchemaService.loadSchema(SchemaService.java:69)
>> >> at
>> >> org.apache.oozie.service.SchemaService.init(SchemaService.java:85)
>> >> at
>> >> org.apache.oozie.service.Services.setServiceInternal(Services.java:307)
>> >> at
>> org.apache.oozie.service.Services.setService(Services.java:293)
>> >> at org.apache.oozie.service.Services.init(Services.java:203)
>> >> at
>> >>
>> >>
>> org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:37)
>> >> at
>> >>
>> >>
>> org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4205)
>> >> at
>> >>
>> org.apache.catalina.core.StandardContext.start(StandardContext.java:4704)
>> >> at
>> >>
>> >>
>> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
>> >> at
>> >> org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
>> >> at
>> >> org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
>> >> at
>> >>
>> >>
>> org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:675)
>> >> at
>> >>
>> >>
>> org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:601)
>> >> at
>> >> org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:502)
>> >> at
>> >> org.apache.catalina.startup.HostConfig.start(HostConfig.java:1315)
>> >> at
>> >>
>> org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
>> >> at
>> >>
>> >>
>> org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
>> >> at
>> >> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1061)
>> >> at
>> >> org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
>> >> at
>> >> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053)
>> >> at
>> >> org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
>> >> at
>> >>
>> org.apache.catalina.core.StandardService.start(StandardService.java:525)
>> >> at
>> >> org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
>> >> at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
>> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> >> at
>> >>
>> >>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> >> at
>> >>
>> >>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >> at java.lang.reflect.Method.invoke(Method.java:597)
>> >> at
>> org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
>> >> at
>> org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
>> >> [4] <property>
>> >> <name>oozie.service.ActionService.executor.ext.classes</name>
>> >> <value>
>> >> org.apache.oozie.action.hadoop.CascadingJrubyIndependent,
>> >> org.apache.oozie.action.hadoop.HiveActionExecutor,
>> >> org.apache.oozie.action.hadoop.SqoopActionExecutor,
>> >> org.apache.oozie.action.email.EmailActionExecutor
>> >> </value>
>> >> </property>
>> >>
>> >> <property>
>> >> <name>oozie.service.SchemaService.wf.ext.schemas</name>
>> >>
>> >>
>> >>
>> <value>hive-action-0.2.xsd,sqoop-action-0.2.xsd,email-action-0.1.xsd,distcp-action-0.1.xsd,cascadingjruby-action-1.0.xsd</value>
>> >> </property>
>> >> <property>
>> >> <name>oozie.service.WorkflowSchemaService.ext.schemas</name>
>> >>
>> >>
>> >>
>> <value>hive-action-0.2.xsd,sqoop-action-0.2.xsd,email-action-0.1.xsd,distcp-action-0.1.xsd,cascadingjruby-action-1.0.xsd</value>
>> >>
>> >> </property>
>> >>
>> >
>>
>
>
Re: schema for new (extension) action type not being picked up
Posted by Aaron Beppu <ab...@etsy.com>.
Any suggestions? It looks like I'm not missing anything with minOccurs="1"
and I haven't put in more than the maxOccurs for anything. Is there a way
get oozie to tell which actions the schema service has loaded?
On Tue, Jan 10, 2012 at 9:05 AM, Aaron Beppu <ab...@etsy.com> wrote:
> Mohammad,
>
> Thanks for looking. Yeah, my bad. I guess I assumed because of the error
> message that the issue was that the action wasn't registered properly
> rather than that I was using it wrong -- but clearly one of my assumptions
> is going to be violated before this is over. Here's the workflow definition
> (unchanged except that I've subbed out the names of our s3 buckets) :
>
> <workflow-app xmlns="uri:oozie:workflow:0.2" name="cj-search-query-count">
> <start to="cj-node"/>
> <action name="cj-node">
> <cascadingjruby>
> <job-tracker>${jobTracker}</job-tracker>
> <name-node>${nameNode}</name-node>
> <prepare>
> <delete
> path="${nameNode}/user/${wf:user()}/${examplesRoot}/output-data/bigdata"/>
> </prepare>
> <configuration>
> <property>
> <name>mapred.job.queue.name</name>
> <value>${queueName}</value>
> </property>
> <property>
> <name>mapred.compress.map.output</name>
> <value>true</value>
> </property>
> </configuration>
> <arg>search_query_count.rb</arg>
> <arg>today_path=2011_12_16</arg>
> <arg>yesterday_path=2011_12_15</arg>
> <arg>time_path=2011_12_16/16_33_31</arg>
> <arg>task_name=analytics_search_query_count</arg>
> <arg>job_signature=secretsecretsecret</arg>
> <arg>end_date=2011_12_10</arg>
> <arg>log_prefix=s3n://secretlogbucket</arg>
> <arg>input_prefix=s3n://secretinputbucket</arg>
> <arg>output_prefix=s3n://secretoutputbucket</arg>
> <arg>index_prefix=s3n://secretindexbucket</arg>
> <arg>hive_prefix=s3n://secrethivebucekt</arg>
> <arg>graphite_prefix=s3n://secretgraphicbucket</arg>
> <arg>splunk_prefix=s3n://secretsplunkbucket</arg>
> <arg>dbdata_prefix=s3n://secretdbdatabucket</arg>
> <arg>instance_type=c1.medium</arg>
> <arg>instance_count=20</arg>
> <arg>master_instance_type=c1.medium</arg>
> </cascadingjruby>
> <ok to="end"/>
> <error to="fail"/>
> </action>
> <kill name="fail">
> <message>BigData java action failed, error
> message[${wf:errorMessage(wf:lastErrorNode())}]</message>
> </kill>
> <end name="end"/>
> </workflow-app>
>
>
>
>
> Thanks,
> Aaron
>
> On Tue, Jan 10, 2012 at 4:46 AM, Mohammad Islam <mi...@yahoo.com>wrote:
>
>> Would you please send at least the relevant definition of action from
>> your workflow.xml?
>>
>> Thanks
>> Mohammad
>>
>>
>> ________________________________
>> From: Aaron Beppu <ab...@etsy.com>
>> To: oozie-users@incubator.apache.org
>> Sent: Monday, January 9, 2012 5:04 PM
>> Subject: Re: schema for new (extension) action type not being picked up
>>
>>
>> Hi Alejandro,
>>
>> Thanks for looking. I'm both attaching and copy-pasting, as I'm not sure
>> how the list handles attachments. I don't know that all of this is
>> necessary, but I was following hive-action-0.2.xsd and sqoop-action-0.2.xsd
>> closely as examples.
>>
>> <?xmlversion="1.0"encoding="UTF-8"?>
>> <xs:schemaxmlns:xs="http://www.w3.org/2001/XMLSchema";
>>
>> xmlns:cascadingjruby="uri:oozie:cascadingjruby-action:0.1"elementFormDefault="qualified"
>> targetNamespace="uri:oozie:cascadingjruby-action:0.1">
>>
>> <xs:elementname="cascadingjruby"type="cascadingjruby:ACTION"/>
>>
>> <xs:complexTypename="ACTION">
>> <xs:sequence>
>>
>> <xs:elementname="job-tracker"type="xs:string"minOccurs="1"maxOccurs="1"/>
>>
>> <xs:elementname="name-node"type="xs:string"minOccurs="1"maxOccurs="1"/>
>>
>> <xs:elementname="prepare"type="cascadingjruby:PREPARE"minOccurs="0"maxOccurs="1"/>
>>
>> <xs:elementname="job-xml"type="xs:string"minOccurs="0"maxOccurs="1"/>
>>
>> <xs:elementname="configuration"type="cascadingjruby:CONFIGURATION"minOccurs="0"maxOccurs="1"/>
>>
>> <xs:elementname="arg"type="xs:string"minOccurs="0"maxOccurs="unbounded"/>
>>
>> <xs:elementname="file"type="xs:string"minOccurs="0"maxOccurs="unbounded"/>
>>
>> <xs:elementname="archive"type="xs:string"minOccurs="0"maxOccurs="unbounded"/>
>> </xs:sequence>
>> </xs:complexType>
>>
>> <xs:complexTypename="CONFIGURATION">
>> <xs:sequence>
>> <xs:elementname="property"minOccurs="1"maxOccurs="unbounded">
>> <xs:complexType>
>> <xs:sequence>
>> <xs:element name="name" minOccurs="1"
>> maxOccurs="1" type="xs:string"/>
>> <xs:element name="value" minOccurs="1"
>> maxOccurs="1" type="xs:string"/>
>> <xs:element name="description" minOccurs="0"
>> maxOccurs="1" type="xs:string"/>
>> </xs:sequence>
>> </xs:complexType>
>> </xs:element>
>> </xs:sequence>
>> </xs:complexType>
>>
>> <xs:complexTypename="PREPARE">
>> <xs:sequence>
>>
>> <xs:elementname="delete"type="cascadingjruby:DELETE"minOccurs="0"maxOccurs="unbounded"/>
>>
>> <xs:elementname="mkdir"type="cascadingjruby:MKDIR"minOccurs="0"maxOccurs="unbounded"/>
>> </xs:sequence>
>> </xs:complexType>
>>
>> <xs:complexTypename="DELETE">
>> <xs:attributename="path"type="xs:string"use="required"/>
>> </xs:complexType>
>>
>> <xs:complexTypename="MKDIR">
>> <xs:attributename="path"type="xs:string"use="required"/>
>> </xs:complexType>
>>
>> </xs:schema>
>>
>>
>>
>>
>> On Mon, Jan 9, 2012 at 7:26 PM, Alejandro Abdelnur <tu...@cloudera.com>
>> wrote:
>>
>> Aaron,
>> >
>> >Would you mind attaching your XSD file?
>> >
>> >Thxs.
>> >
>> >Alejandro
>> >
>> >
>> >On Mon, Jan 9, 2012 at 3:54 PM, Aaron Beppu <ab...@etsy.com> wrote:
>> >
>> >> Hi all. Confused oozie neophyte here, hoping for some guidance.
>> >>
>> >> I'm trying to create a new action type, following directions both on
>> InfoQ
>> >> [1], and in the official documentation [2]. So I've:
>> >> 1. created the new action type, along with a schema,
>> >> 2. built a jar containing the new code and schema,
>> >> 3. added that jar to ${CATALINA_BASE}/webapps/oozie/WEB-INF/lib (and
>> when
>> >> that failed, using ".../bin/oozie-setup.sh -jars mine.jar")
>> >> 4. added my new action type to
>> >> oozie.service.ActionService.executor.ext.classes
>> >> in oozie-site.xml
>> >> 5. added my xsd filename to
>> oozie.service.SchemaService.wf.ext.schemas, and
>> >> to oozie.service.WorkflowSchemaService.ext.schemas (because the oozie
>> docs
>> >> and the InfoQ directions disagreed)
>> >> 6. restarted my oozie server and
>> >> 7. attempted to submit a workflow using the new action type.
>> >>
>> >> At this point, whenever I try to run the test workflow with the new
>> action,
>> >> I get this error :
>> >>
>> >> Error: E0701 : E0701: XML schema error, cvc-complex-type.2.4.a: Invalid
>> >> content was found starting with element 'cascadingjruby'. One of
>> >> '{"uri:oozie:workflow:0.2":map-reduce, "uri:oozie:workflow:0.2":pig,
>> >> "uri:oozie:workflow:0.2":sub-workflow, "uri:oozie:workflow:0.2":fs,
>> >> "uri:oozie:workflow:0.2":java, WC[##other:"uri:oozie:workflow:0.2"]}'
>> is
>> >> expected.
>> >>
>> >> A few notes :
>> >> 1. I know that I'm naming the right .xsd file and that it's being
>> picked up
>> >> because earlier I had a typo in the filename in oozie-site.xml, and I
>> got
>> >> an *earlier* error when attempting to restart oozie [3], *before* I
>> even
>> >> tried to kick a job. So having eliminated that error, and not seeing
>> >> anything break until I actually try to start a job, I'm reasonably
>> >> confident that I'm not just making a filename problem -- and yet it
>> appears
>> >> not to know about my new action type.
>> >> 2. "Extended" action types which I *can* use successfully (e.g.
>> >> Hive/email/sqoop) are *not* on the list enumerated -- so whatever it's
>> >> using to enumerate that list of "good" types, it's *not* identical to
>> the
>> >> actual set of actions that it's loaded and can execute.
>> >> 3. The sole comment on that InfoQ article is from someone in October
>> >> getting the same error. He didn't get any response from the article
>> >> authors. So I'm guessing I'm making a common enough error that someone
>> >> around here must know about the problem.
>> >>
>> >> What am I missing? I'm running cdh3u2 (so oozie version 2.3.2). I'm
>> >> appending in the relevant chunk of my oozie-site.xml [4].
>> >>
>> >> Thanks so much,
>> >> Aaron
>> >>
>> >> [1] http://www.infoq.com/articles/ExtendingOozie
>> >> [2]
>> http://archive.cloudera.com/cdh/3/oozie/DG_CustomActionExecutor.html
>> >> [3] java.lang.IllegalArgumentException: resource
>> >> [cascadingjruby-action-1.0.xsd] not found
>> >> at
>> >> org.apache.oozie.util.IOUtils.getResourceAsStream(IOUtils.java:100)
>> >> at
>> >>
>> org.apache.oozie.service.SchemaService.loadSchema(SchemaService.java:69)
>> >> at
>> >> org.apache.oozie.service.SchemaService.init(SchemaService.java:85)
>> >> at
>> >> org.apache.oozie.service.Services.setServiceInternal(Services.java:307)
>> >> at
>> org.apache.oozie.service.Services.setService(Services.java:293)
>> >> at org.apache.oozie.service.Services.init(Services.java:203)
>> >> at
>> >>
>> >>
>> org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:37)
>> >> at
>> >>
>> >>
>> org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4205)
>> >> at
>> >>
>> org.apache.catalina.core.StandardContext.start(StandardContext.java:4704)
>> >> at
>> >>
>> >>
>> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
>> >> at
>> >> org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
>> >> at
>> >> org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
>> >> at
>> >>
>> >>
>> org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:675)
>> >> at
>> >>
>> >>
>> org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:601)
>> >> at
>> >> org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:502)
>> >> at
>> >> org.apache.catalina.startup.HostConfig.start(HostConfig.java:1315)
>> >> at
>> >>
>> org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
>> >> at
>> >>
>> >>
>> org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
>> >> at
>> >> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1061)
>> >> at
>> >> org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
>> >> at
>> >> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053)
>> >> at
>> >> org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
>> >> at
>> >>
>> org.apache.catalina.core.StandardService.start(StandardService.java:525)
>> >> at
>> >> org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
>> >> at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
>> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> >> at
>> >>
>> >>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> >> at
>> >>
>> >>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >> at java.lang.reflect.Method.invoke(Method.java:597)
>> >> at
>> org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
>> >> at
>> org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
>> >> [4] <property>
>> >> <name>oozie.service.ActionService.executor.ext.classes</name>
>> >> <value>
>> >> org.apache.oozie.action.hadoop.CascadingJrubyIndependent,
>> >> org.apache.oozie.action.hadoop.HiveActionExecutor,
>> >> org.apache.oozie.action.hadoop.SqoopActionExecutor,
>> >> org.apache.oozie.action.email.EmailActionExecutor
>> >> </value>
>> >> </property>
>> >>
>> >> <property>
>> >> <name>oozie.service.SchemaService.wf.ext.schemas</name>
>> >>
>> >>
>> >>
>> <value>hive-action-0.2.xsd,sqoop-action-0.2.xsd,email-action-0.1.xsd,distcp-action-0.1.xsd,cascadingjruby-action-1.0.xsd</value>
>> >> </property>
>> >> <property>
>> >> <name>oozie.service.WorkflowSchemaService.ext.schemas</name>
>> >>
>> >>
>> >>
>> <value>hive-action-0.2.xsd,sqoop-action-0.2.xsd,email-action-0.1.xsd,distcp-action-0.1.xsd,cascadingjruby-action-1.0.xsd</value>
>> >>
>> >> </property>
>> >>
>> >
>>
>
>
Re: schema for new (extension) action type not being picked up
Posted by Aaron Beppu <ab...@etsy.com>.
Mohammad,
Thanks for looking. Yeah, my bad. I guess I assumed because of the error
message that the issue was that the action wasn't registered properly
rather than that I was using it wrong -- but clearly one of my assumptions
is going to be violated before this is over. Here's the workflow definition
(unchanged except that I've subbed out the names of our s3 buckets) :
<workflow-app xmlns="uri:oozie:workflow:0.2" name="cj-search-query-count">
<start to="cj-node"/>
<action name="cj-node">
<cascadingjruby>
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<prepare>
<delete
path="${nameNode}/user/${wf:user()}/${examplesRoot}/output-data/bigdata"/>
</prepare>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>${queueName}</value>
</property>
<property>
<name>mapred.compress.map.output</name>
<value>true</value>
</property>
</configuration>
<arg>search_query_count.rb</arg>
<arg>today_path=2011_12_16</arg>
<arg>yesterday_path=2011_12_15</arg>
<arg>time_path=2011_12_16/16_33_31</arg>
<arg>task_name=analytics_search_query_count</arg>
<arg>job_signature=secretsecretsecret</arg>
<arg>end_date=2011_12_10</arg>
<arg>log_prefix=s3n://secretlogbucket</arg>
<arg>input_prefix=s3n://secretinputbucket</arg>
<arg>output_prefix=s3n://secretoutputbucket</arg>
<arg>index_prefix=s3n://secretindexbucket</arg>
<arg>hive_prefix=s3n://secrethivebucekt</arg>
<arg>graphite_prefix=s3n://secretgraphicbucket</arg>
<arg>splunk_prefix=s3n://secretsplunkbucket</arg>
<arg>dbdata_prefix=s3n://secretdbdatabucket</arg>
<arg>instance_type=c1.medium</arg>
<arg>instance_count=20</arg>
<arg>master_instance_type=c1.medium</arg>
</cascadingjruby>
<ok to="end"/>
<error to="fail"/>
</action>
<kill name="fail">
<message>BigData java action failed, error
message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name="end"/>
</workflow-app>
Thanks,
Aaron
On Tue, Jan 10, 2012 at 4:46 AM, Mohammad Islam <mi...@yahoo.com> wrote:
> Would you please send at least the relevant definition of action from your
> workflow.xml?
>
> Thanks
> Mohammad
>
>
> ________________________________
> From: Aaron Beppu <ab...@etsy.com>
> To: oozie-users@incubator.apache.org
> Sent: Monday, January 9, 2012 5:04 PM
> Subject: Re: schema for new (extension) action type not being picked up
>
>
> Hi Alejandro,
>
> Thanks for looking. I'm both attaching and copy-pasting, as I'm not sure
> how the list handles attachments. I don't know that all of this is
> necessary, but I was following hive-action-0.2.xsd and sqoop-action-0.2.xsd
> closely as examples.
>
> <?xmlversion="1.0"encoding="UTF-8"?>
> <xs:schemaxmlns:xs="http://www.w3.org/2001/XMLSchema";
>
> xmlns:cascadingjruby="uri:oozie:cascadingjruby-action:0.1"elementFormDefault="qualified"
> targetNamespace="uri:oozie:cascadingjruby-action:0.1">
>
> <xs:elementname="cascadingjruby"type="cascadingjruby:ACTION"/>
>
> <xs:complexTypename="ACTION">
> <xs:sequence>
>
> <xs:elementname="job-tracker"type="xs:string"minOccurs="1"maxOccurs="1"/>
>
> <xs:elementname="name-node"type="xs:string"minOccurs="1"maxOccurs="1"/>
>
> <xs:elementname="prepare"type="cascadingjruby:PREPARE"minOccurs="0"maxOccurs="1"/>
>
> <xs:elementname="job-xml"type="xs:string"minOccurs="0"maxOccurs="1"/>
>
> <xs:elementname="configuration"type="cascadingjruby:CONFIGURATION"minOccurs="0"maxOccurs="1"/>
>
> <xs:elementname="arg"type="xs:string"minOccurs="0"maxOccurs="unbounded"/>
>
> <xs:elementname="file"type="xs:string"minOccurs="0"maxOccurs="unbounded"/>
>
> <xs:elementname="archive"type="xs:string"minOccurs="0"maxOccurs="unbounded"/>
> </xs:sequence>
> </xs:complexType>
>
> <xs:complexTypename="CONFIGURATION">
> <xs:sequence>
> <xs:elementname="property"minOccurs="1"maxOccurs="unbounded">
> <xs:complexType>
> <xs:sequence>
> <xs:element name="name" minOccurs="1"
> maxOccurs="1" type="xs:string"/>
> <xs:element name="value" minOccurs="1"
> maxOccurs="1" type="xs:string"/>
> <xs:element name="description" minOccurs="0"
> maxOccurs="1" type="xs:string"/>
> </xs:sequence>
> </xs:complexType>
> </xs:element>
> </xs:sequence>
> </xs:complexType>
>
> <xs:complexTypename="PREPARE">
> <xs:sequence>
>
> <xs:elementname="delete"type="cascadingjruby:DELETE"minOccurs="0"maxOccurs="unbounded"/>
>
> <xs:elementname="mkdir"type="cascadingjruby:MKDIR"minOccurs="0"maxOccurs="unbounded"/>
> </xs:sequence>
> </xs:complexType>
>
> <xs:complexTypename="DELETE">
> <xs:attributename="path"type="xs:string"use="required"/>
> </xs:complexType>
>
> <xs:complexTypename="MKDIR">
> <xs:attributename="path"type="xs:string"use="required"/>
> </xs:complexType>
>
> </xs:schema>
>
>
>
>
> On Mon, Jan 9, 2012 at 7:26 PM, Alejandro Abdelnur <tu...@cloudera.com>
> wrote:
>
> Aaron,
> >
> >Would you mind attaching your XSD file?
> >
> >Thxs.
> >
> >Alejandro
> >
> >
> >On Mon, Jan 9, 2012 at 3:54 PM, Aaron Beppu <ab...@etsy.com> wrote:
> >
> >> Hi all. Confused oozie neophyte here, hoping for some guidance.
> >>
> >> I'm trying to create a new action type, following directions both on
> InfoQ
> >> [1], and in the official documentation [2]. So I've:
> >> 1. created the new action type, along with a schema,
> >> 2. built a jar containing the new code and schema,
> >> 3. added that jar to ${CATALINA_BASE}/webapps/oozie/WEB-INF/lib (and
> when
> >> that failed, using ".../bin/oozie-setup.sh -jars mine.jar")
> >> 4. added my new action type to
> >> oozie.service.ActionService.executor.ext.classes
> >> in oozie-site.xml
> >> 5. added my xsd filename to oozie.service.SchemaService.wf.ext.schemas,
> and
> >> to oozie.service.WorkflowSchemaService.ext.schemas (because the oozie
> docs
> >> and the InfoQ directions disagreed)
> >> 6. restarted my oozie server and
> >> 7. attempted to submit a workflow using the new action type.
> >>
> >> At this point, whenever I try to run the test workflow with the new
> action,
> >> I get this error :
> >>
> >> Error: E0701 : E0701: XML schema error, cvc-complex-type.2.4.a: Invalid
> >> content was found starting with element 'cascadingjruby'. One of
> >> '{"uri:oozie:workflow:0.2":map-reduce, "uri:oozie:workflow:0.2":pig,
> >> "uri:oozie:workflow:0.2":sub-workflow, "uri:oozie:workflow:0.2":fs,
> >> "uri:oozie:workflow:0.2":java, WC[##other:"uri:oozie:workflow:0.2"]}' is
> >> expected.
> >>
> >> A few notes :
> >> 1. I know that I'm naming the right .xsd file and that it's being
> picked up
> >> because earlier I had a typo in the filename in oozie-site.xml, and I
> got
> >> an *earlier* error when attempting to restart oozie [3], *before* I even
> >> tried to kick a job. So having eliminated that error, and not seeing
> >> anything break until I actually try to start a job, I'm reasonably
> >> confident that I'm not just making a filename problem -- and yet it
> appears
> >> not to know about my new action type.
> >> 2. "Extended" action types which I *can* use successfully (e.g.
> >> Hive/email/sqoop) are *not* on the list enumerated -- so whatever it's
> >> using to enumerate that list of "good" types, it's *not* identical to
> the
> >> actual set of actions that it's loaded and can execute.
> >> 3. The sole comment on that InfoQ article is from someone in October
> >> getting the same error. He didn't get any response from the article
> >> authors. So I'm guessing I'm making a common enough error that someone
> >> around here must know about the problem.
> >>
> >> What am I missing? I'm running cdh3u2 (so oozie version 2.3.2). I'm
> >> appending in the relevant chunk of my oozie-site.xml [4].
> >>
> >> Thanks so much,
> >> Aaron
> >>
> >> [1] http://www.infoq.com/articles/ExtendingOozie
> >> [2]
> http://archive.cloudera.com/cdh/3/oozie/DG_CustomActionExecutor.html
> >> [3] java.lang.IllegalArgumentException: resource
> >> [cascadingjruby-action-1.0.xsd] not found
> >> at
> >> org.apache.oozie.util.IOUtils.getResourceAsStream(IOUtils.java:100)
> >> at
> >> org.apache.oozie.service.SchemaService.loadSchema(SchemaService.java:69)
> >> at
> >> org.apache.oozie.service.SchemaService.init(SchemaService.java:85)
> >> at
> >> org.apache.oozie.service.Services.setServiceInternal(Services.java:307)
> >> at
> org.apache.oozie.service.Services.setService(Services.java:293)
> >> at org.apache.oozie.service.Services.init(Services.java:203)
> >> at
> >>
> >>
> org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:37)
> >> at
> >>
> >>
> org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4205)
> >> at
> >>
> org.apache.catalina.core.StandardContext.start(StandardContext.java:4704)
> >> at
> >>
> >>
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
> >> at
> >> org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
> >> at
> >> org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
> >> at
> >>
> >>
> org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:675)
> >> at
> >>
> >>
> org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:601)
> >> at
> >> org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:502)
> >> at
> >> org.apache.catalina.startup.HostConfig.start(HostConfig.java:1315)
> >> at
> >>
> org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
> >> at
> >>
> >>
> org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
> >> at
> >> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1061)
> >> at
> >> org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
> >> at
> >> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053)
> >> at
> >> org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
> >> at
> >> org.apache.catalina.core.StandardService.start(StandardService.java:525)
> >> at
> >> org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
> >> at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> at
> >>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> at
> >>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> at java.lang.reflect.Method.invoke(Method.java:597)
> >> at
> org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
> >> at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
> >> [4] <property>
> >> <name>oozie.service.ActionService.executor.ext.classes</name>
> >> <value>
> >> org.apache.oozie.action.hadoop.CascadingJrubyIndependent,
> >> org.apache.oozie.action.hadoop.HiveActionExecutor,
> >> org.apache.oozie.action.hadoop.SqoopActionExecutor,
> >> org.apache.oozie.action.email.EmailActionExecutor
> >> </value>
> >> </property>
> >>
> >> <property>
> >> <name>oozie.service.SchemaService.wf.ext.schemas</name>
> >>
> >>
> >>
> <value>hive-action-0.2.xsd,sqoop-action-0.2.xsd,email-action-0.1.xsd,distcp-action-0.1.xsd,cascadingjruby-action-1.0.xsd</value>
> >> </property>
> >> <property>
> >> <name>oozie.service.WorkflowSchemaService.ext.schemas</name>
> >>
> >>
> >>
> <value>hive-action-0.2.xsd,sqoop-action-0.2.xsd,email-action-0.1.xsd,distcp-action-0.1.xsd,cascadingjruby-action-1.0.xsd</value>
> >>
> >> </property>
> >>
> >
>
Re: schema for new (extension) action type not being picked up
Posted by Mohammad Islam <mi...@yahoo.com>.
Would you please send at least the relevant definition of action from your workflow.xml?
Thanks
Mohammad
________________________________
From: Aaron Beppu <ab...@etsy.com>
To: oozie-users@incubator.apache.org
Sent: Monday, January 9, 2012 5:04 PM
Subject: Re: schema for new (extension) action type not being picked up
Hi Alejandro,
Thanks for looking. I'm both attaching and copy-pasting, as I'm not sure how the list handles attachments. I don't know that all of this is necessary, but I was following hive-action-0.2.xsd and sqoop-action-0.2.xsd closely as examples.
<?xmlversion="1.0"encoding="UTF-8"?>
<xs:schemaxmlns:xs="http://www.w3.org/2001/XMLSchema";
xmlns:cascadingjruby="uri:oozie:cascadingjruby-action:0.1"elementFormDefault="qualified"
targetNamespace="uri:oozie:cascadingjruby-action:0.1">
<xs:elementname="cascadingjruby"type="cascadingjruby:ACTION"/>
<xs:complexTypename="ACTION">
<xs:sequence>
<xs:elementname="job-tracker"type="xs:string"minOccurs="1"maxOccurs="1"/>
<xs:elementname="name-node"type="xs:string"minOccurs="1"maxOccurs="1"/>
<xs:elementname="prepare"type="cascadingjruby:PREPARE"minOccurs="0"maxOccurs="1"/>
<xs:elementname="job-xml"type="xs:string"minOccurs="0"maxOccurs="1"/>
<xs:elementname="configuration"type="cascadingjruby:CONFIGURATION"minOccurs="0"maxOccurs="1"/>
<xs:elementname="arg"type="xs:string"minOccurs="0"maxOccurs="unbounded"/>
<xs:elementname="file"type="xs:string"minOccurs="0"maxOccurs="unbounded"/>
<xs:elementname="archive"type="xs:string"minOccurs="0"maxOccurs="unbounded"/>
</xs:sequence>
</xs:complexType>
<xs:complexTypename="CONFIGURATION">
<xs:sequence>
<xs:elementname="property"minOccurs="1"maxOccurs="unbounded">
<xs:complexType>
<xs:sequence>
<xs:element name="name" minOccurs="1" maxOccurs="1" type="xs:string"/>
<xs:element name="value" minOccurs="1" maxOccurs="1" type="xs:string"/>
<xs:element name="description" minOccurs="0" maxOccurs="1" type="xs:string"/>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>
<xs:complexTypename="PREPARE">
<xs:sequence>
<xs:elementname="delete"type="cascadingjruby:DELETE"minOccurs="0"maxOccurs="unbounded"/>
<xs:elementname="mkdir"type="cascadingjruby:MKDIR"minOccurs="0"maxOccurs="unbounded"/>
</xs:sequence>
</xs:complexType>
<xs:complexTypename="DELETE">
<xs:attributename="path"type="xs:string"use="required"/>
</xs:complexType>
<xs:complexTypename="MKDIR">
<xs:attributename="path"type="xs:string"use="required"/>
</xs:complexType>
</xs:schema>
On Mon, Jan 9, 2012 at 7:26 PM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
Aaron,
>
>Would you mind attaching your XSD file?
>
>Thxs.
>
>Alejandro
>
>
>On Mon, Jan 9, 2012 at 3:54 PM, Aaron Beppu <ab...@etsy.com> wrote:
>
>> Hi all. Confused oozie neophyte here, hoping for some guidance.
>>
>> I'm trying to create a new action type, following directions both on InfoQ
>> [1], and in the official documentation [2]. So I've:
>> 1. created the new action type, along with a schema,
>> 2. built a jar containing the new code and schema,
>> 3. added that jar to ${CATALINA_BASE}/webapps/oozie/WEB-INF/lib (and when
>> that failed, using ".../bin/oozie-setup.sh -jars mine.jar")
>> 4. added my new action type to
>> oozie.service.ActionService.executor.ext.classes
>> in oozie-site.xml
>> 5. added my xsd filename to oozie.service.SchemaService.wf.ext.schemas, and
>> to oozie.service.WorkflowSchemaService.ext.schemas (because the oozie docs
>> and the InfoQ directions disagreed)
>> 6. restarted my oozie server and
>> 7. attempted to submit a workflow using the new action type.
>>
>> At this point, whenever I try to run the test workflow with the new action,
>> I get this error :
>>
>> Error: E0701 : E0701: XML schema error, cvc-complex-type.2.4.a: Invalid
>> content was found starting with element 'cascadingjruby'. One of
>> '{"uri:oozie:workflow:0.2":map-reduce, "uri:oozie:workflow:0.2":pig,
>> "uri:oozie:workflow:0.2":sub-workflow, "uri:oozie:workflow:0.2":fs,
>> "uri:oozie:workflow:0.2":java, WC[##other:"uri:oozie:workflow:0.2"]}' is
>> expected.
>>
>> A few notes :
>> 1. I know that I'm naming the right .xsd file and that it's being picked up
>> because earlier I had a typo in the filename in oozie-site.xml, and I got
>> an *earlier* error when attempting to restart oozie [3], *before* I even
>> tried to kick a job. So having eliminated that error, and not seeing
>> anything break until I actually try to start a job, I'm reasonably
>> confident that I'm not just making a filename problem -- and yet it appears
>> not to know about my new action type.
>> 2. "Extended" action types which I *can* use successfully (e.g.
>> Hive/email/sqoop) are *not* on the list enumerated -- so whatever it's
>> using to enumerate that list of "good" types, it's *not* identical to the
>> actual set of actions that it's loaded and can execute.
>> 3. The sole comment on that InfoQ article is from someone in October
>> getting the same error. He didn't get any response from the article
>> authors. So I'm guessing I'm making a common enough error that someone
>> around here must know about the problem.
>>
>> What am I missing? I'm running cdh3u2 (so oozie version 2.3.2). I'm
>> appending in the relevant chunk of my oozie-site.xml [4].
>>
>> Thanks so much,
>> Aaron
>>
>> [1] http://www.infoq.com/articles/ExtendingOozie
>> [2] http://archive.cloudera.com/cdh/3/oozie/DG_CustomActionExecutor.html
>> [3] java.lang.IllegalArgumentException: resource
>> [cascadingjruby-action-1.0.xsd] not found
>> at
>> org.apache.oozie.util.IOUtils.getResourceAsStream(IOUtils.java:100)
>> at
>> org.apache.oozie.service.SchemaService.loadSchema(SchemaService.java:69)
>> at
>> org.apache.oozie.service.SchemaService.init(SchemaService.java:85)
>> at
>> org.apache.oozie.service.Services.setServiceInternal(Services.java:307)
>> at org.apache.oozie.service.Services.setService(Services.java:293)
>> at org.apache.oozie.service.Services.init(Services.java:203)
>> at
>>
>> org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:37)
>> at
>>
>> org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4205)
>> at
>> org.apache.catalina.core.StandardContext.start(StandardContext.java:4704)
>> at
>>
>> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
>> at
>> org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
>> at
>> org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
>> at
>>
>> org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:675)
>> at
>>
>> org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:601)
>> at
>> org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:502)
>> at
>> org.apache.catalina.startup.HostConfig.start(HostConfig.java:1315)
>> at
>> org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
>> at
>>
>> org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
>> at
>> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1061)
>> at
>> org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
>> at
>> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053)
>> at
>> org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
>> at
>> org.apache.catalina.core.StandardService.start(StandardService.java:525)
>> at
>> org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
>> at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at
>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
>> at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
>> [4] <property>
>> <name>oozie.service.ActionService.executor.ext.classes</name>
>> <value>
>> org.apache.oozie.action.hadoop.CascadingJrubyIndependent,
>> org.apache.oozie.action.hadoop.HiveActionExecutor,
>> org.apache.oozie.action.hadoop.SqoopActionExecutor,
>> org.apache.oozie.action.email.EmailActionExecutor
>> </value>
>> </property>
>>
>> <property>
>> <name>oozie.service.SchemaService.wf.ext.schemas</name>
>>
>>
>> <value>hive-action-0.2.xsd,sqoop-action-0.2.xsd,email-action-0.1.xsd,distcp-action-0.1.xsd,cascadingjruby-action-1.0.xsd</value>
>> </property>
>> <property>
>> <name>oozie.service.WorkflowSchemaService.ext.schemas</name>
>>
>>
>> <value>hive-action-0.2.xsd,sqoop-action-0.2.xsd,email-action-0.1.xsd,distcp-action-0.1.xsd,cascadingjruby-action-1.0.xsd</value>
>>
>> </property>
>>
>
Re: schema for new (extension) action type not being picked up
Posted by Aaron Beppu <ab...@etsy.com>.
Hi Alejandro,
Thanks for looking. I'm both attaching and copy-pasting, as I'm not sure
how the list handles attachments. I don't know that all of this is
necessary, but I was following hive-action-0.2.xsd and sqoop-action-0.2.xsd
closely as examples.
<?xml version=*"1.0"* encoding=*"UTF-8"*?>
<xs:schema xmlns:xs=*"http://www.w3.org/2001/XMLSchema"*
* * xmlns:cascadingjruby=*"uri:oozie:cascadingjruby-action:0.1"*
elementFormDefault=*"qualified"*
targetNamespace=*"uri:oozie:cascadingjruby-action:0.1"*>
<xs:element name=*"cascadingjruby"* type=*"cascadingjruby:ACTION"*/>
<xs:complexType name=*"ACTION"*>
<xs:sequence>
<xs:element name=*"job-tracker"* type=*"xs:string"* minOccurs=*
"1"* maxOccurs=*"1"*/>
<xs:element name=*"name-node"* type=*"xs:string"* minOccurs=*"1"
* maxOccurs=*"1"*/>
<xs:element name=*"prepare"* type=*"cascadingjruby:PREPARE"*
minOccurs=*"0"* maxOccurs=*"1"*/>
<xs:element name=*"job-xml"* type=*"xs:string"* minOccurs=*"0"*
maxOccurs=*"1"*/>
<xs:element name=*"configuration"* type=*
"cascadingjruby:CONFIGURATION"* minOccurs=*"0"* maxOccurs=*"1"*/>
<xs:element name=*"arg"* type=*"xs:string"* minOccurs=*"0"*
maxOccurs=*"unbounded"*/>
<xs:element name=*"file"* type=*"xs:string"* minOccurs=*"0"*
maxOccurs=*"unbounded"*/>
<xs:element name=*"archive"* type=*"xs:string"* minOccurs=*"0"*
maxOccurs=*"unbounded"*/>
</xs:sequence>
</xs:complexType>
<xs:complexType name=*"CONFIGURATION"*>
<xs:sequence>
<xs:element name=*"property"* minOccurs=*"1"* maxOccurs=*
"unbounded"*>
<xs:complexType>
<xs:sequence>
<xs:element name=*"name"* minOccurs=*"1"* maxOccurs=
*"1"* type=*"xs:string"*/>
<xs:element name=*"value"* minOccurs=*"1"* maxOccurs
=*"1"* type=*"xs:string"*/>
<xs:element name=*"description"* minOccurs=*"0"*
maxOccurs=*"1"* type=*"xs:string"*/>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>
<xs:complexType name=*"PREPARE"*>
<xs:sequence>
<xs:element name=*"delete"* type=*"cascadingjruby:DELETE"*
minOccurs=*"0"* maxOccurs=*"unbounded"*/>
<xs:element name=*"mkdir"* type=*"cascadingjruby:MKDIR"*
minOccurs=*"0"* maxOccurs=*"unbounded"*/>
</xs:sequence>
</xs:complexType>
<xs:complexType name=*"DELETE"*>
<xs:attribute name=*"path"* type=*"xs:string"* use=*"required"*/>
</xs:complexType>
<xs:complexType name=*"MKDIR"*>
<xs:attribute name=*"path"* type=*"xs:string"* use=*"required"*/>
</xs:complexType>
</xs:schema>
On Mon, Jan 9, 2012 at 7:26 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> Aaron,
>
> Would you mind attaching your XSD file?
>
> Thxs.
>
> Alejandro
>
> On Mon, Jan 9, 2012 at 3:54 PM, Aaron Beppu <ab...@etsy.com> wrote:
>
> > Hi all. Confused oozie neophyte here, hoping for some guidance.
> >
> > I'm trying to create a new action type, following directions both on
> InfoQ
> > [1], and in the official documentation [2]. So I've:
> > 1. created the new action type, along with a schema,
> > 2. built a jar containing the new code and schema,
> > 3. added that jar to ${CATALINA_BASE}/webapps/oozie/WEB-INF/lib (and when
> > that failed, using ".../bin/oozie-setup.sh -jars mine.jar")
> > 4. added my new action type to
> > oozie.service.ActionService.executor.ext.classes
> > in oozie-site.xml
> > 5. added my xsd filename to oozie.service.SchemaService.wf.ext.schemas,
> and
> > to oozie.service.WorkflowSchemaService.ext.schemas (because the oozie
> docs
> > and the InfoQ directions disagreed)
> > 6. restarted my oozie server and
> > 7. attempted to submit a workflow using the new action type.
> >
> > At this point, whenever I try to run the test workflow with the new
> action,
> > I get this error :
> >
> > Error: E0701 : E0701: XML schema error, cvc-complex-type.2.4.a: Invalid
> > content was found starting with element 'cascadingjruby'. One of
> > '{"uri:oozie:workflow:0.2":map-reduce, "uri:oozie:workflow:0.2":pig,
> > "uri:oozie:workflow:0.2":sub-workflow, "uri:oozie:workflow:0.2":fs,
> > "uri:oozie:workflow:0.2":java, WC[##other:"uri:oozie:workflow:0.2"]}' is
> > expected.
> >
> > A few notes :
> > 1. I know that I'm naming the right .xsd file and that it's being picked
> up
> > because earlier I had a typo in the filename in oozie-site.xml, and I got
> > an *earlier* error when attempting to restart oozie [3], *before* I even
> > tried to kick a job. So having eliminated that error, and not seeing
> > anything break until I actually try to start a job, I'm reasonably
> > confident that I'm not just making a filename problem -- and yet it
> appears
> > not to know about my new action type.
> > 2. "Extended" action types which I *can* use successfully (e.g.
> > Hive/email/sqoop) are *not* on the list enumerated -- so whatever it's
> > using to enumerate that list of "good" types, it's *not* identical to the
> > actual set of actions that it's loaded and can execute.
> > 3. The sole comment on that InfoQ article is from someone in October
> > getting the same error. He didn't get any response from the article
> > authors. So I'm guessing I'm making a common enough error that someone
> > around here must know about the problem.
> >
> > What am I missing? I'm running cdh3u2 (so oozie version 2.3.2). I'm
> > appending in the relevant chunk of my oozie-site.xml [4].
> >
> > Thanks so much,
> > Aaron
> >
> > [1] http://www.infoq.com/articles/ExtendingOozie
> > [2] http://archive.cloudera.com/cdh/3/oozie/DG_CustomActionExecutor.html
> > [3] java.lang.IllegalArgumentException: resource
> > [cascadingjruby-action-1.0.xsd] not found
> > at
> > org.apache.oozie.util.IOUtils.getResourceAsStream(IOUtils.java:100)
> > at
> > org.apache.oozie.service.SchemaService.loadSchema(SchemaService.java:69)
> > at
> > org.apache.oozie.service.SchemaService.init(SchemaService.java:85)
> > at
> > org.apache.oozie.service.Services.setServiceInternal(Services.java:307)
> > at org.apache.oozie.service.Services.setService(Services.java:293)
> > at org.apache.oozie.service.Services.init(Services.java:203)
> > at
> >
> >
> org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:37)
> > at
> >
> >
> org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4205)
> > at
> > org.apache.catalina.core.StandardContext.start(StandardContext.java:4704)
> > at
> >
> >
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
> > at
> > org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
> > at
> > org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
> > at
> >
> >
> org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:675)
> > at
> >
> >
> org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:601)
> > at
> > org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:502)
> > at
> > org.apache.catalina.startup.HostConfig.start(HostConfig.java:1315)
> > at
> >
> org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
> > at
> >
> >
> org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
> > at
> > org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1061)
> > at
> > org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
> > at
> > org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053)
> > at
> > org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
> > at
> > org.apache.catalina.core.StandardService.start(StandardService.java:525)
> > at
> > org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
> > at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
> > at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
> > [4] <property>
> > <name>oozie.service.ActionService.executor.ext.classes</name>
> > <value>
> > org.apache.oozie.action.hadoop.CascadingJrubyIndependent,
> > org.apache.oozie.action.hadoop.HiveActionExecutor,
> > org.apache.oozie.action.hadoop.SqoopActionExecutor,
> > org.apache.oozie.action.email.EmailActionExecutor
> > </value>
> > </property>
> >
> > <property>
> > <name>oozie.service.SchemaService.wf.ext.schemas</name>
> >
> >
> >
> <value>hive-action-0.2.xsd,sqoop-action-0.2.xsd,email-action-0.1.xsd,distcp-action-0.1.xsd,cascadingjruby-action-1.0.xsd</value>
> > </property>
> > <property>
> > <name>oozie.service.WorkflowSchemaService.ext.schemas</name>
> >
> >
> >
> <value>hive-action-0.2.xsd,sqoop-action-0.2.xsd,email-action-0.1.xsd,distcp-action-0.1.xsd,cascadingjruby-action-1.0.xsd</value>
> >
> > </property>
> >
>
Re: schema for new (extension) action type not being picked up
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
Aaron,
Would you mind attaching your XSD file?
Thxs.
Alejandro
On Mon, Jan 9, 2012 at 3:54 PM, Aaron Beppu <ab...@etsy.com> wrote:
> Hi all. Confused oozie neophyte here, hoping for some guidance.
>
> I'm trying to create a new action type, following directions both on InfoQ
> [1], and in the official documentation [2]. So I've:
> 1. created the new action type, along with a schema,
> 2. built a jar containing the new code and schema,
> 3. added that jar to ${CATALINA_BASE}/webapps/oozie/WEB-INF/lib (and when
> that failed, using ".../bin/oozie-setup.sh -jars mine.jar")
> 4. added my new action type to
> oozie.service.ActionService.executor.ext.classes
> in oozie-site.xml
> 5. added my xsd filename to oozie.service.SchemaService.wf.ext.schemas, and
> to oozie.service.WorkflowSchemaService.ext.schemas (because the oozie docs
> and the InfoQ directions disagreed)
> 6. restarted my oozie server and
> 7. attempted to submit a workflow using the new action type.
>
> At this point, whenever I try to run the test workflow with the new action,
> I get this error :
>
> Error: E0701 : E0701: XML schema error, cvc-complex-type.2.4.a: Invalid
> content was found starting with element 'cascadingjruby'. One of
> '{"uri:oozie:workflow:0.2":map-reduce, "uri:oozie:workflow:0.2":pig,
> "uri:oozie:workflow:0.2":sub-workflow, "uri:oozie:workflow:0.2":fs,
> "uri:oozie:workflow:0.2":java, WC[##other:"uri:oozie:workflow:0.2"]}' is
> expected.
>
> A few notes :
> 1. I know that I'm naming the right .xsd file and that it's being picked up
> because earlier I had a typo in the filename in oozie-site.xml, and I got
> an *earlier* error when attempting to restart oozie [3], *before* I even
> tried to kick a job. So having eliminated that error, and not seeing
> anything break until I actually try to start a job, I'm reasonably
> confident that I'm not just making a filename problem -- and yet it appears
> not to know about my new action type.
> 2. "Extended" action types which I *can* use successfully (e.g.
> Hive/email/sqoop) are *not* on the list enumerated -- so whatever it's
> using to enumerate that list of "good" types, it's *not* identical to the
> actual set of actions that it's loaded and can execute.
> 3. The sole comment on that InfoQ article is from someone in October
> getting the same error. He didn't get any response from the article
> authors. So I'm guessing I'm making a common enough error that someone
> around here must know about the problem.
>
> What am I missing? I'm running cdh3u2 (so oozie version 2.3.2). I'm
> appending in the relevant chunk of my oozie-site.xml [4].
>
> Thanks so much,
> Aaron
>
> [1] http://www.infoq.com/articles/ExtendingOozie
> [2] http://archive.cloudera.com/cdh/3/oozie/DG_CustomActionExecutor.html
> [3] java.lang.IllegalArgumentException: resource
> [cascadingjruby-action-1.0.xsd] not found
> at
> org.apache.oozie.util.IOUtils.getResourceAsStream(IOUtils.java:100)
> at
> org.apache.oozie.service.SchemaService.loadSchema(SchemaService.java:69)
> at
> org.apache.oozie.service.SchemaService.init(SchemaService.java:85)
> at
> org.apache.oozie.service.Services.setServiceInternal(Services.java:307)
> at org.apache.oozie.service.Services.setService(Services.java:293)
> at org.apache.oozie.service.Services.init(Services.java:203)
> at
>
> org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:37)
> at
>
> org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4205)
> at
> org.apache.catalina.core.StandardContext.start(StandardContext.java:4704)
> at
>
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
> at
> org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
> at
> org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
> at
>
> org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:675)
> at
>
> org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:601)
> at
> org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:502)
> at
> org.apache.catalina.startup.HostConfig.start(HostConfig.java:1315)
> at
> org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
> at
>
> org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
> at
> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1061)
> at
> org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
> at
> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053)
> at
> org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
> at
> org.apache.catalina.core.StandardService.start(StandardService.java:525)
> at
> org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
> at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
> at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
> [4] <property>
> <name>oozie.service.ActionService.executor.ext.classes</name>
> <value>
> org.apache.oozie.action.hadoop.CascadingJrubyIndependent,
> org.apache.oozie.action.hadoop.HiveActionExecutor,
> org.apache.oozie.action.hadoop.SqoopActionExecutor,
> org.apache.oozie.action.email.EmailActionExecutor
> </value>
> </property>
>
> <property>
> <name>oozie.service.SchemaService.wf.ext.schemas</name>
>
>
> <value>hive-action-0.2.xsd,sqoop-action-0.2.xsd,email-action-0.1.xsd,distcp-action-0.1.xsd,cascadingjruby-action-1.0.xsd</value>
> </property>
> <property>
> <name>oozie.service.WorkflowSchemaService.ext.schemas</name>
>
>
> <value>hive-action-0.2.xsd,sqoop-action-0.2.xsd,email-action-0.1.xsd,distcp-action-0.1.xsd,cascadingjruby-action-1.0.xsd</value>
>
> </property>
>