You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by Serega Sheypak <se...@gmail.com> on 2013/10/02 11:06:30 UTC
HowTo configure purge for coordinator materializations?
Hi, we are using Oozie vervsion Oozie BUILD_VERSION [3.3.2-cdh4.3.0]
compiled by [jenkins] on [2013.05.28-04:29:38GMT]
I want to PurgeService work.
I did set these props:
<!-- Purging old materializations--> <!-- Keep them for three months -->
<property> <name>oozie.service.PurgeService.older.than</name>
<value>90</value> </property> <property>
<name>oozie.service.PurgeService.coord.older.than</name> <value>90</value>
</property> <property> <name>oozie.service.PurgeService.purge.limit</name>
<value>100</value> </property> <property>
<name>oozie.service.PurgeService.purge.interval</name> <value>100</value>
</property>
The idea is: keep actions data for 90 dayes or not more than last 100 items.
oozie.service.PurgeService.purge.interval is set to 100 seconds.
Nothing happened after 100 seconds.
I've tried to add this section:
<property> <name>oozie.services</name> <value>
org.apache.oozie.service.PurgeService,
org.apache.oozie.service.SchedulerService </value> </property>
Now Ooize is not starting:
org.apache.oozie.service.ServiceException: E0103: Could not load
service classes, null
at org.apache.oozie.service.Services.loadServices(Services.java:291)
at org.apache.oozie.service.Services.init(Services.java:208)
at org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:39)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4206)
at org.apache.catalina.core.StandardContext.start(StandardContext.java:4705)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:943)
at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:778)
at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:504)
at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317)
at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065)
at org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057)
at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
at org.apache.catalina.core.StandardService.start(StandardService.java:525)
at org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
Caused by: java.lang.NullPointerException
at org.apache.oozie.service.PurgeService.init(PurgeService.java:76)
at org.apache.oozie.service.Services.setServiceInternal(Services.java:368)
at org.apache.oozie.service.Services.setService(Services.java:354)
at org.apache.oozie.service.Services.loadServices(Services.java:287)
... 26 more
What do I do wrong?
Re: HowTo configure purge for coordinator materializations?
Posted by Robert Kanter <rk...@cloudera.com>.
If you turn on DEBUG level in your log configuration, it should print out
whenever the purge service starts and finishes; in the message where it
finishes, it should say how many jobs it deleted. With info level logging,
I don't think it prints anything.
The purge behavior won't purge a workflow/coordinator/bundle until all of
its parents/children are ready to be purged. This is to protect the
consistency of the data. So for example, if you have a long-running
coordinator job launching workflows, none of those workflows will be purged
until the coordinator is ready to be purged. Perhaps this is what you're
seeing?
<!-- We need it because of NPE in PurgeService. It needs all Purge
> properties to be explicitly set-->
> <property>
> <name>oozie.service.PurgeService.bundle.older.than</name>
> <value>14</value>
> <description>
> Completed bundle jobs older than this value, in days, will be
> purged by the PurgeService.
> </description>
> </property>
This shouldn't be the case; if you don't specify this, it should just
default to 7. Can you paste the NPE so I can look into what's causing it?
- Robert
On Mon, Oct 7, 2013 at 1:30 AM, Serega Sheypak <se...@gmail.com>wrote:
> Nothing helps. Looks like I'm missing somthing. Oozie works, but there is
> no purge happens.
>
>
> 2013/10/4 Serega Sheypak <se...@gmail.com>
>
> > purge.executions= 11
> > in counters section, but nothing happens or I don't see i
> >
> >
> > 2013/10/4 Serega Sheypak <se...@gmail.com>
> >
> >> Hi, I've tried this one:
> >>
> >> <property>
> >> <name>oozie.services.ext</name>
> >> <value>
> >> org.apache.oozie.service.PurgeService
> >> </value>
> >> </property>
> >>
> >> <!-- Purging old materializations-->
> >> <!-- Keep them for three months -->
> >> <property>
> >> <name>oozie.service.PurgeService.older.than</name>
> >> <value>90</value>
> >> </property>
> >>
> >> <property>
> >> <name>oozie.service.PurgeService.coord.older.than</name>
> >> <value>90</value>
> >> </property>
> >>
> >> <property>
> >> <name>oozie.service.PurgeService.purge.limit</name>
> >> <value>100</value>
> >> </property>
> >>
> >> <property>
> >> <name>oozie.service.PurgeService.purge.interval</name>
> >> <value>100</value>
> >> </property>
> >>
> >> <!-- We need it because of NPE in PurgeService. It needs all Purge
> >> properties to be explicitly set-->
> >> <property>
> >> <name>oozie.service.PurgeService.bundle.older.than</name>
> >> <value>14</value>
> >> <description>
> >> Completed bundle jobs older than this value, in days, will
> be
> >> purged by the PurgeService.
> >> </description>
> >> </property>
> >>
> >> Oozie is started, cool, I forgot about oozie-default.xml. Thank you.
> >> The problem is here I still see last 900+ coordinator materializations.
> >> Nothing has been deleted.
> >> 100 seconds (oozie.service.PurgeService.purge.interval) did pass.
> >>
> >> There is nothing related to PurgeSerivce in log.
> >>
> >> 1. What do I do wrong?
> >> 2. oozie.service.PurgeService.purge.limit=100 and
> >> oozie.service.PurgeService.older.than=90 (days)
> >> What has more power? Or one of these conditions should trigger
> >> PurgeService to cleanup oozie DB?
> >>
> >>
> >>
> >> 2013/10/2 Robert Kanter <rk...@cloudera.com>
> >>
> >>> CDH does have that property. In both CDH and Apache distributions,
> that
> >>> property (and some others) is specified in oozie-default.xml. There
> >>> should
> >>> be a copy of its somewhere. Oozie has its own internal copy of
> >>> oozie-default.xml, so don't bother editing the one you find; Oozie
> loads
> >>> its oozie-default.xml first and then loads oozie-site.xml with any
> >>> properties you set (Hadoop works the same way with core-default.xml and
> >>> core-site.xml).
> >>>
> >>> The NPE is because you only have the two services, PurgeService and
> >>> SchedulerService,
> >>> loaded because you overrode the oozie.services property. The NPE
> occurs
> >>> because Oozie does need the ConfigurationService for sure to work
> >>> properly;
> >>> and without many of the other services, it will likely have other
> issues.
> >>>
> >>> You should remove your override of oozie.services from oozie-site.xml.
> >>> Check oozie-default.xml and see if it lists PurgeService under
> >>> oozie.services. If it does not, then you can add it in your
> >>> oozie-site.xml
> >>> in the oozie.services.ext property; if it does, then the PurgeService
> is
> >>> enabled.
> >>>
> >>> - Robert
> >>>
> >>>
> >>>
> >>>
> >>> On Wed, Oct 2, 2013 at 10:49 AM, Serega Sheypak <
> >>> serega.sheypak@gmail.com>wrote:
> >>>
> >>> > Hi Robert, thanks for reply.
> >>> >
> >>> > Cloudera distirbution doesn't have property oozie.services
> >>> > 1. I've tried to set all props for purge service with purge period =
> >>> 100
> >>> > (seconds???) Nothing happened for half an hour
> >>> >
> >>> > 2. I've tired to explicitly declare PurgeService and got NPE.
> >>> >
> >>> > I don't understand how it works at all... I don't see expected
> >>> behaviour.
> >>> >
> >>> >
> >>> > 2013/10/2 Robert Kanter <rk...@cloudera.com>
> >>> >
> >>> > > Hi Serega,
> >>> > >
> >>> > > oozie-default.xml specifies a number of oozie services to load via
> >>> the
> >>> > > "oozie.services" property; but specifying
> >>> > > <property> <name>oozie.services</name> <value>
> >>> > > org.apache.oozie.service.PurgeService,
> >>> > > org.apache.oozie.service.SchedulerService </value> </property>
> >>> > > you've now removed all but 2 of them. You should remove that
> >>> property
> >>> > from
> >>> > > your oozie-site.xml.
> >>> > >
> >>> > > If you want to add additional services, you should use the
> >>> > > "oozie.services.ext" property instead. That said, the PurgeService
> >>> > should
> >>> > > already be listed under "oozie.services" so it should be running by
> >>> > > default.
> >>> > >
> >>> > > - Robert
> >>> > >
> >>> > >
> >>> > > On Wed, Oct 2, 2013 at 2:06 AM, Serega Sheypak <
> >>> serega.sheypak@gmail.com
> >>> > > >wrote:
> >>> > >
> >>> > > > Hi, we are using Oozie vervsion Oozie BUILD_VERSION
> >>> [3.3.2-cdh4.3.0]
> >>> > > > compiled by [jenkins] on [2013.05.28-04:29:38GMT]
> >>> > > >
> >>> > > > I want to PurgeService work.
> >>> > > > I did set these props:
> >>> > > > <!-- Purging old materializations--> <!-- Keep them for three
> >>> months
> >>> > -->
> >>> > > > <property> <name>oozie.service.PurgeService.older.than</name>
> >>> > > > <value>90</value> </property> <property>
> >>> > > > <name>oozie.service.PurgeService.coord.older.than</name>
> >>> > > <value>90</value>
> >>> > > > </property> <property>
> >>> > > <name>oozie.service.PurgeService.purge.limit</name>
> >>> > > > <value>100</value> </property> <property>
> >>> > > > <name>oozie.service.PurgeService.purge.interval</name>
> >>> > <value>100</value>
> >>> > > > </property>
> >>> > > >
> >>> > > >
> >>> > > > The idea is: keep actions data for 90 dayes or not more than last
> >>> 100
> >>> > > > items.
> >>> > > > oozie.service.PurgeService.purge.interval is set to 100 seconds.
> >>> > > > Nothing happened after 100 seconds.
> >>> > > >
> >>> > > > I've tried to add this section:
> >>> > > > <property> <name>oozie.services</name> <value>
> >>> > > > org.apache.oozie.service.PurgeService,
> >>> > > > org.apache.oozie.service.SchedulerService </value> </property>
> >>> > > >
> >>> > > > Now Ooize is not starting:
> >>> > > >
> >>> > > >
> >>> > > > org.apache.oozie.service.ServiceException: E0103: Could not load
> >>> > > > service classes, null
> >>> > > > at
> >>> > > > org.apache.oozie.service.Services.loadServices(Services.java:291)
> >>> > > > at
> >>> org.apache.oozie.service.Services.init(Services.java:208)
> >>> > > >
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:39)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4206)
> >>> > > > at
> >>> > > >
> >>> >
> >>>
> org.apache.catalina.core.StandardContext.start(StandardContext.java:4705)
> >>> > > >
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
> >>> > > > at
> >>> > > >
> >>> org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
> >>> > > > at
> >>> > > >
> >>> org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
> >>> > > >
> >>> > > > at
> >>> > > >
> >>> org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:943)
> >>> > > > at
> >>> > > >
> >>> org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:778)
> >>> > > > at
> >>> > > >
> >>> org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:504)
> >>> > > >
> >>> > > > at
> >>> > > >
> org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
> >>> > > >
> >>> > > > at
> >>> > > >
> >>> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065)
> >>> > > > at
> >>> > > >
> org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
> >>> > > > at
> >>> > > >
> >>> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057)
> >>> > > >
> >>> > > > at
> >>> > > >
> >>> org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
> >>> > > > at
> >>> > > >
> >>> >
> >>>
> org.apache.catalina.core.StandardService.start(StandardService.java:525)
> >>> > > > at
> >>> > > >
> >>> org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
> >>> > > >
> >>> > > > at
> >>> > org.apache.catalina.startup.Catalina.start(Catalina.java:595)
> >>> > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >>> Method)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>> > > >
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>> > > > at java.lang.reflect.Method.invoke(Method.java:597)
> >>> > > > at
> >>> > > org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
> >>> > > > at
> >>> > org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
> >>> > > >
> >>> > > > Caused by: java.lang.NullPointerException
> >>> > > > at
> >>> > > org.apache.oozie.service.PurgeService.init(PurgeService.java:76)
> >>> > > > at
> >>> > > >
> >>> org.apache.oozie.service.Services.setServiceInternal(Services.java:368)
> >>> > > > at
> >>> > > org.apache.oozie.service.Services.setService(Services.java:354)
> >>> > > >
> >>> > > > at
> >>> > > > org.apache.oozie.service.Services.loadServices(Services.java:287)
> >>> > > > ... 26 more
> >>> > > >
> >>> > > >
> >>> > > > What do I do wrong?
> >>> > > >
> >>> > >
> >>> >
> >>>
> >>
> >>
> >
>
Re: HowTo configure purge for coordinator materializations?
Posted by Serega Sheypak <se...@gmail.com>.
Nothing helps. Looks like I'm missing somthing. Oozie works, but there is
no purge happens.
2013/10/4 Serega Sheypak <se...@gmail.com>
> purge.executions= 11
> in counters section, but nothing happens or I don't see i
>
>
> 2013/10/4 Serega Sheypak <se...@gmail.com>
>
>> Hi, I've tried this one:
>>
>> <property>
>> <name>oozie.services.ext</name>
>> <value>
>> org.apache.oozie.service.PurgeService
>> </value>
>> </property>
>>
>> <!-- Purging old materializations-->
>> <!-- Keep them for three months -->
>> <property>
>> <name>oozie.service.PurgeService.older.than</name>
>> <value>90</value>
>> </property>
>>
>> <property>
>> <name>oozie.service.PurgeService.coord.older.than</name>
>> <value>90</value>
>> </property>
>>
>> <property>
>> <name>oozie.service.PurgeService.purge.limit</name>
>> <value>100</value>
>> </property>
>>
>> <property>
>> <name>oozie.service.PurgeService.purge.interval</name>
>> <value>100</value>
>> </property>
>>
>> <!-- We need it because of NPE in PurgeService. It needs all Purge
>> properties to be explicitly set-->
>> <property>
>> <name>oozie.service.PurgeService.bundle.older.than</name>
>> <value>14</value>
>> <description>
>> Completed bundle jobs older than this value, in days, will be
>> purged by the PurgeService.
>> </description>
>> </property>
>>
>> Oozie is started, cool, I forgot about oozie-default.xml. Thank you.
>> The problem is here I still see last 900+ coordinator materializations.
>> Nothing has been deleted.
>> 100 seconds (oozie.service.PurgeService.purge.interval) did pass.
>>
>> There is nothing related to PurgeSerivce in log.
>>
>> 1. What do I do wrong?
>> 2. oozie.service.PurgeService.purge.limit=100 and
>> oozie.service.PurgeService.older.than=90 (days)
>> What has more power? Or one of these conditions should trigger
>> PurgeService to cleanup oozie DB?
>>
>>
>>
>> 2013/10/2 Robert Kanter <rk...@cloudera.com>
>>
>>> CDH does have that property. In both CDH and Apache distributions, that
>>> property (and some others) is specified in oozie-default.xml. There
>>> should
>>> be a copy of its somewhere. Oozie has its own internal copy of
>>> oozie-default.xml, so don't bother editing the one you find; Oozie loads
>>> its oozie-default.xml first and then loads oozie-site.xml with any
>>> properties you set (Hadoop works the same way with core-default.xml and
>>> core-site.xml).
>>>
>>> The NPE is because you only have the two services, PurgeService and
>>> SchedulerService,
>>> loaded because you overrode the oozie.services property. The NPE occurs
>>> because Oozie does need the ConfigurationService for sure to work
>>> properly;
>>> and without many of the other services, it will likely have other issues.
>>>
>>> You should remove your override of oozie.services from oozie-site.xml.
>>> Check oozie-default.xml and see if it lists PurgeService under
>>> oozie.services. If it does not, then you can add it in your
>>> oozie-site.xml
>>> in the oozie.services.ext property; if it does, then the PurgeService is
>>> enabled.
>>>
>>> - Robert
>>>
>>>
>>>
>>>
>>> On Wed, Oct 2, 2013 at 10:49 AM, Serega Sheypak <
>>> serega.sheypak@gmail.com>wrote:
>>>
>>> > Hi Robert, thanks for reply.
>>> >
>>> > Cloudera distirbution doesn't have property oozie.services
>>> > 1. I've tried to set all props for purge service with purge period =
>>> 100
>>> > (seconds???) Nothing happened for half an hour
>>> >
>>> > 2. I've tired to explicitly declare PurgeService and got NPE.
>>> >
>>> > I don't understand how it works at all... I don't see expected
>>> behaviour.
>>> >
>>> >
>>> > 2013/10/2 Robert Kanter <rk...@cloudera.com>
>>> >
>>> > > Hi Serega,
>>> > >
>>> > > oozie-default.xml specifies a number of oozie services to load via
>>> the
>>> > > "oozie.services" property; but specifying
>>> > > <property> <name>oozie.services</name> <value>
>>> > > org.apache.oozie.service.PurgeService,
>>> > > org.apache.oozie.service.SchedulerService </value> </property>
>>> > > you've now removed all but 2 of them. You should remove that
>>> property
>>> > from
>>> > > your oozie-site.xml.
>>> > >
>>> > > If you want to add additional services, you should use the
>>> > > "oozie.services.ext" property instead. That said, the PurgeService
>>> > should
>>> > > already be listed under "oozie.services" so it should be running by
>>> > > default.
>>> > >
>>> > > - Robert
>>> > >
>>> > >
>>> > > On Wed, Oct 2, 2013 at 2:06 AM, Serega Sheypak <
>>> serega.sheypak@gmail.com
>>> > > >wrote:
>>> > >
>>> > > > Hi, we are using Oozie vervsion Oozie BUILD_VERSION
>>> [3.3.2-cdh4.3.0]
>>> > > > compiled by [jenkins] on [2013.05.28-04:29:38GMT]
>>> > > >
>>> > > > I want to PurgeService work.
>>> > > > I did set these props:
>>> > > > <!-- Purging old materializations--> <!-- Keep them for three
>>> months
>>> > -->
>>> > > > <property> <name>oozie.service.PurgeService.older.than</name>
>>> > > > <value>90</value> </property> <property>
>>> > > > <name>oozie.service.PurgeService.coord.older.than</name>
>>> > > <value>90</value>
>>> > > > </property> <property>
>>> > > <name>oozie.service.PurgeService.purge.limit</name>
>>> > > > <value>100</value> </property> <property>
>>> > > > <name>oozie.service.PurgeService.purge.interval</name>
>>> > <value>100</value>
>>> > > > </property>
>>> > > >
>>> > > >
>>> > > > The idea is: keep actions data for 90 dayes or not more than last
>>> 100
>>> > > > items.
>>> > > > oozie.service.PurgeService.purge.interval is set to 100 seconds.
>>> > > > Nothing happened after 100 seconds.
>>> > > >
>>> > > > I've tried to add this section:
>>> > > > <property> <name>oozie.services</name> <value>
>>> > > > org.apache.oozie.service.PurgeService,
>>> > > > org.apache.oozie.service.SchedulerService </value> </property>
>>> > > >
>>> > > > Now Ooize is not starting:
>>> > > >
>>> > > >
>>> > > > org.apache.oozie.service.ServiceException: E0103: Could not load
>>> > > > service classes, null
>>> > > > at
>>> > > > org.apache.oozie.service.Services.loadServices(Services.java:291)
>>> > > > at
>>> org.apache.oozie.service.Services.init(Services.java:208)
>>> > > >
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:39)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4206)
>>> > > > at
>>> > > >
>>> >
>>> org.apache.catalina.core.StandardContext.start(StandardContext.java:4705)
>>> > > >
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
>>> > > > at
>>> > > >
>>> org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
>>> > > > at
>>> > > >
>>> org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
>>> > > >
>>> > > > at
>>> > > >
>>> org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:943)
>>> > > > at
>>> > > >
>>> org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:778)
>>> > > > at
>>> > > >
>>> org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:504)
>>> > > >
>>> > > > at
>>> > > > org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
>>> > > >
>>> > > > at
>>> > > >
>>> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065)
>>> > > > at
>>> > > > org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
>>> > > > at
>>> > > >
>>> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057)
>>> > > >
>>> > > > at
>>> > > >
>>> org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
>>> > > > at
>>> > > >
>>> >
>>> org.apache.catalina.core.StandardService.start(StandardService.java:525)
>>> > > > at
>>> > > >
>>> org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
>>> > > >
>>> > > > at
>>> > org.apache.catalina.startup.Catalina.start(Catalina.java:595)
>>> > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>> Method)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> > > >
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > > > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > > > at
>>> > > org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
>>> > > > at
>>> > org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
>>> > > >
>>> > > > Caused by: java.lang.NullPointerException
>>> > > > at
>>> > > org.apache.oozie.service.PurgeService.init(PurgeService.java:76)
>>> > > > at
>>> > > >
>>> org.apache.oozie.service.Services.setServiceInternal(Services.java:368)
>>> > > > at
>>> > > org.apache.oozie.service.Services.setService(Services.java:354)
>>> > > >
>>> > > > at
>>> > > > org.apache.oozie.service.Services.loadServices(Services.java:287)
>>> > > > ... 26 more
>>> > > >
>>> > > >
>>> > > > What do I do wrong?
>>> > > >
>>> > >
>>> >
>>>
>>
>>
>
Re: HowTo configure purge for coordinator materializations?
Posted by Serega Sheypak <se...@gmail.com>.
purge.executions= 11
in counters section, but nothing happens or I don't see i
2013/10/4 Serega Sheypak <se...@gmail.com>
> Hi, I've tried this one:
>
> <property>
> <name>oozie.services.ext</name>
> <value>
> org.apache.oozie.service.PurgeService
> </value>
> </property>
>
> <!-- Purging old materializations-->
> <!-- Keep them for three months -->
> <property>
> <name>oozie.service.PurgeService.older.than</name>
> <value>90</value>
> </property>
>
> <property>
> <name>oozie.service.PurgeService.coord.older.than</name>
> <value>90</value>
> </property>
>
> <property>
> <name>oozie.service.PurgeService.purge.limit</name>
> <value>100</value>
> </property>
>
> <property>
> <name>oozie.service.PurgeService.purge.interval</name>
> <value>100</value>
> </property>
>
> <!-- We need it because of NPE in PurgeService. It needs all Purge
> properties to be explicitly set-->
> <property>
> <name>oozie.service.PurgeService.bundle.older.than</name>
> <value>14</value>
> <description>
> Completed bundle jobs older than this value, in days, will be
> purged by the PurgeService.
> </description>
> </property>
>
> Oozie is started, cool, I forgot about oozie-default.xml. Thank you.
> The problem is here I still see last 900+ coordinator materializations.
> Nothing has been deleted.
> 100 seconds (oozie.service.PurgeService.purge.interval) did pass.
>
> There is nothing related to PurgeSerivce in log.
>
> 1. What do I do wrong?
> 2. oozie.service.PurgeService.purge.limit=100 and
> oozie.service.PurgeService.older.than=90 (days)
> What has more power? Or one of these conditions should trigger
> PurgeService to cleanup oozie DB?
>
>
>
> 2013/10/2 Robert Kanter <rk...@cloudera.com>
>
>> CDH does have that property. In both CDH and Apache distributions, that
>> property (and some others) is specified in oozie-default.xml. There
>> should
>> be a copy of its somewhere. Oozie has its own internal copy of
>> oozie-default.xml, so don't bother editing the one you find; Oozie loads
>> its oozie-default.xml first and then loads oozie-site.xml with any
>> properties you set (Hadoop works the same way with core-default.xml and
>> core-site.xml).
>>
>> The NPE is because you only have the two services, PurgeService and
>> SchedulerService,
>> loaded because you overrode the oozie.services property. The NPE occurs
>> because Oozie does need the ConfigurationService for sure to work
>> properly;
>> and without many of the other services, it will likely have other issues.
>>
>> You should remove your override of oozie.services from oozie-site.xml.
>> Check oozie-default.xml and see if it lists PurgeService under
>> oozie.services. If it does not, then you can add it in your
>> oozie-site.xml
>> in the oozie.services.ext property; if it does, then the PurgeService is
>> enabled.
>>
>> - Robert
>>
>>
>>
>>
>> On Wed, Oct 2, 2013 at 10:49 AM, Serega Sheypak <serega.sheypak@gmail.com
>> >wrote:
>>
>> > Hi Robert, thanks for reply.
>> >
>> > Cloudera distirbution doesn't have property oozie.services
>> > 1. I've tried to set all props for purge service with purge period = 100
>> > (seconds???) Nothing happened for half an hour
>> >
>> > 2. I've tired to explicitly declare PurgeService and got NPE.
>> >
>> > I don't understand how it works at all... I don't see expected
>> behaviour.
>> >
>> >
>> > 2013/10/2 Robert Kanter <rk...@cloudera.com>
>> >
>> > > Hi Serega,
>> > >
>> > > oozie-default.xml specifies a number of oozie services to load via the
>> > > "oozie.services" property; but specifying
>> > > <property> <name>oozie.services</name> <value>
>> > > org.apache.oozie.service.PurgeService,
>> > > org.apache.oozie.service.SchedulerService </value> </property>
>> > > you've now removed all but 2 of them. You should remove that property
>> > from
>> > > your oozie-site.xml.
>> > >
>> > > If you want to add additional services, you should use the
>> > > "oozie.services.ext" property instead. That said, the PurgeService
>> > should
>> > > already be listed under "oozie.services" so it should be running by
>> > > default.
>> > >
>> > > - Robert
>> > >
>> > >
>> > > On Wed, Oct 2, 2013 at 2:06 AM, Serega Sheypak <
>> serega.sheypak@gmail.com
>> > > >wrote:
>> > >
>> > > > Hi, we are using Oozie vervsion Oozie BUILD_VERSION
>> [3.3.2-cdh4.3.0]
>> > > > compiled by [jenkins] on [2013.05.28-04:29:38GMT]
>> > > >
>> > > > I want to PurgeService work.
>> > > > I did set these props:
>> > > > <!-- Purging old materializations--> <!-- Keep them for three months
>> > -->
>> > > > <property> <name>oozie.service.PurgeService.older.than</name>
>> > > > <value>90</value> </property> <property>
>> > > > <name>oozie.service.PurgeService.coord.older.than</name>
>> > > <value>90</value>
>> > > > </property> <property>
>> > > <name>oozie.service.PurgeService.purge.limit</name>
>> > > > <value>100</value> </property> <property>
>> > > > <name>oozie.service.PurgeService.purge.interval</name>
>> > <value>100</value>
>> > > > </property>
>> > > >
>> > > >
>> > > > The idea is: keep actions data for 90 dayes or not more than last
>> 100
>> > > > items.
>> > > > oozie.service.PurgeService.purge.interval is set to 100 seconds.
>> > > > Nothing happened after 100 seconds.
>> > > >
>> > > > I've tried to add this section:
>> > > > <property> <name>oozie.services</name> <value>
>> > > > org.apache.oozie.service.PurgeService,
>> > > > org.apache.oozie.service.SchedulerService </value> </property>
>> > > >
>> > > > Now Ooize is not starting:
>> > > >
>> > > >
>> > > > org.apache.oozie.service.ServiceException: E0103: Could not load
>> > > > service classes, null
>> > > > at
>> > > > org.apache.oozie.service.Services.loadServices(Services.java:291)
>> > > > at org.apache.oozie.service.Services.init(Services.java:208)
>> > > >
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:39)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4206)
>> > > > at
>> > > >
>> >
>> org.apache.catalina.core.StandardContext.start(StandardContext.java:4705)
>> > > >
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
>> > > > at
>> > > >
>> org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
>> > > > at
>> > > >
>> org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
>> > > >
>> > > > at
>> > > >
>> org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:943)
>> > > > at
>> > > >
>> org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:778)
>> > > > at
>> > > >
>> org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:504)
>> > > >
>> > > > at
>> > > > org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
>> > > >
>> > > > at
>> > > >
>> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065)
>> > > > at
>> > > > org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
>> > > > at
>> > > >
>> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057)
>> > > >
>> > > > at
>> > > >
>> org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
>> > > > at
>> > > >
>> > org.apache.catalina.core.StandardService.start(StandardService.java:525)
>> > > > at
>> > > >
>> org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
>> > > >
>> > > > at
>> > org.apache.catalina.startup.Catalina.start(Catalina.java:595)
>> > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)
>> > > > at
>> > > >
>> > >
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > > >
>> > > > at
>> > > >
>> > >
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > > > at java.lang.reflect.Method.invoke(Method.java:597)
>> > > > at
>> > > org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
>> > > > at
>> > org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
>> > > >
>> > > > Caused by: java.lang.NullPointerException
>> > > > at
>> > > org.apache.oozie.service.PurgeService.init(PurgeService.java:76)
>> > > > at
>> > > >
>> org.apache.oozie.service.Services.setServiceInternal(Services.java:368)
>> > > > at
>> > > org.apache.oozie.service.Services.setService(Services.java:354)
>> > > >
>> > > > at
>> > > > org.apache.oozie.service.Services.loadServices(Services.java:287)
>> > > > ... 26 more
>> > > >
>> > > >
>> > > > What do I do wrong?
>> > > >
>> > >
>> >
>>
>
>
Re: HowTo configure purge for coordinator materializations?
Posted by Serega Sheypak <se...@gmail.com>.
Hi, I've tried this one:
<property>
<name>oozie.services.ext</name>
<value>
org.apache.oozie.service.PurgeService
</value>
</property>
<!-- Purging old materializations-->
<!-- Keep them for three months -->
<property>
<name>oozie.service.PurgeService.older.than</name>
<value>90</value>
</property>
<property>
<name>oozie.service.PurgeService.coord.older.than</name>
<value>90</value>
</property>
<property>
<name>oozie.service.PurgeService.purge.limit</name>
<value>100</value>
</property>
<property>
<name>oozie.service.PurgeService.purge.interval</name>
<value>100</value>
</property>
<!-- We need it because of NPE in PurgeService. It needs all Purge
properties to be explicitly set-->
<property>
<name>oozie.service.PurgeService.bundle.older.than</name>
<value>14</value>
<description>
Completed bundle jobs older than this value, in days, will be
purged by the PurgeService.
</description>
</property>
Oozie is started, cool, I forgot about oozie-default.xml. Thank you.
The problem is here I still see last 900+ coordinator materializations.
Nothing has been deleted.
100 seconds (oozie.service.PurgeService.purge.interval) did pass.
There is nothing related to PurgeSerivce in log.
1. What do I do wrong?
2. oozie.service.PurgeService.purge.limit=100 and
oozie.service.PurgeService.older.than=90 (days)
What has more power? Or one of these conditions should trigger PurgeService
to cleanup oozie DB?
2013/10/2 Robert Kanter <rk...@cloudera.com>
> CDH does have that property. In both CDH and Apache distributions, that
> property (and some others) is specified in oozie-default.xml. There should
> be a copy of its somewhere. Oozie has its own internal copy of
> oozie-default.xml, so don't bother editing the one you find; Oozie loads
> its oozie-default.xml first and then loads oozie-site.xml with any
> properties you set (Hadoop works the same way with core-default.xml and
> core-site.xml).
>
> The NPE is because you only have the two services, PurgeService and
> SchedulerService,
> loaded because you overrode the oozie.services property. The NPE occurs
> because Oozie does need the ConfigurationService for sure to work properly;
> and without many of the other services, it will likely have other issues.
>
> You should remove your override of oozie.services from oozie-site.xml.
> Check oozie-default.xml and see if it lists PurgeService under
> oozie.services. If it does not, then you can add it in your oozie-site.xml
> in the oozie.services.ext property; if it does, then the PurgeService is
> enabled.
>
> - Robert
>
>
>
>
> On Wed, Oct 2, 2013 at 10:49 AM, Serega Sheypak <serega.sheypak@gmail.com
> >wrote:
>
> > Hi Robert, thanks for reply.
> >
> > Cloudera distirbution doesn't have property oozie.services
> > 1. I've tried to set all props for purge service with purge period = 100
> > (seconds???) Nothing happened for half an hour
> >
> > 2. I've tired to explicitly declare PurgeService and got NPE.
> >
> > I don't understand how it works at all... I don't see expected behaviour.
> >
> >
> > 2013/10/2 Robert Kanter <rk...@cloudera.com>
> >
> > > Hi Serega,
> > >
> > > oozie-default.xml specifies a number of oozie services to load via the
> > > "oozie.services" property; but specifying
> > > <property> <name>oozie.services</name> <value>
> > > org.apache.oozie.service.PurgeService,
> > > org.apache.oozie.service.SchedulerService </value> </property>
> > > you've now removed all but 2 of them. You should remove that property
> > from
> > > your oozie-site.xml.
> > >
> > > If you want to add additional services, you should use the
> > > "oozie.services.ext" property instead. That said, the PurgeService
> > should
> > > already be listed under "oozie.services" so it should be running by
> > > default.
> > >
> > > - Robert
> > >
> > >
> > > On Wed, Oct 2, 2013 at 2:06 AM, Serega Sheypak <
> serega.sheypak@gmail.com
> > > >wrote:
> > >
> > > > Hi, we are using Oozie vervsion Oozie BUILD_VERSION [3.3.2-cdh4.3.0]
> > > > compiled by [jenkins] on [2013.05.28-04:29:38GMT]
> > > >
> > > > I want to PurgeService work.
> > > > I did set these props:
> > > > <!-- Purging old materializations--> <!-- Keep them for three months
> > -->
> > > > <property> <name>oozie.service.PurgeService.older.than</name>
> > > > <value>90</value> </property> <property>
> > > > <name>oozie.service.PurgeService.coord.older.than</name>
> > > <value>90</value>
> > > > </property> <property>
> > > <name>oozie.service.PurgeService.purge.limit</name>
> > > > <value>100</value> </property> <property>
> > > > <name>oozie.service.PurgeService.purge.interval</name>
> > <value>100</value>
> > > > </property>
> > > >
> > > >
> > > > The idea is: keep actions data for 90 dayes or not more than last 100
> > > > items.
> > > > oozie.service.PurgeService.purge.interval is set to 100 seconds.
> > > > Nothing happened after 100 seconds.
> > > >
> > > > I've tried to add this section:
> > > > <property> <name>oozie.services</name> <value>
> > > > org.apache.oozie.service.PurgeService,
> > > > org.apache.oozie.service.SchedulerService </value> </property>
> > > >
> > > > Now Ooize is not starting:
> > > >
> > > >
> > > > org.apache.oozie.service.ServiceException: E0103: Could not load
> > > > service classes, null
> > > > at
> > > > org.apache.oozie.service.Services.loadServices(Services.java:291)
> > > > at org.apache.oozie.service.Services.init(Services.java:208)
> > > >
> > > > at
> > > >
> > >
> >
> org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:39)
> > > > at
> > > >
> > >
> >
> org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4206)
> > > > at
> > > >
> > org.apache.catalina.core.StandardContext.start(StandardContext.java:4705)
> > > >
> > > > at
> > > >
> > >
> >
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
> > > > at
> > > >
> org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
> > > > at
> > > > org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
> > > >
> > > > at
> > > > org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:943)
> > > > at
> > > >
> org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:778)
> > > > at
> > > >
> org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:504)
> > > >
> > > > at
> > > > org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317)
> > > > at
> > > >
> > >
> >
> org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
> > > > at
> > > >
> > >
> >
> org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
> > > >
> > > > at
> > > > org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065)
> > > > at
> > > > org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
> > > > at
> > > > org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057)
> > > >
> > > > at
> > > >
> org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
> > > > at
> > > >
> > org.apache.catalina.core.StandardService.start(StandardService.java:525)
> > > > at
> > > >
> org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
> > > >
> > > > at
> > org.apache.catalina.startup.Catalina.start(Catalina.java:595)
> > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> > > > at
> > > >
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > > >
> > > > at
> > > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > > > at java.lang.reflect.Method.invoke(Method.java:597)
> > > > at
> > > org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
> > > > at
> > org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
> > > >
> > > > Caused by: java.lang.NullPointerException
> > > > at
> > > org.apache.oozie.service.PurgeService.init(PurgeService.java:76)
> > > > at
> > > >
> org.apache.oozie.service.Services.setServiceInternal(Services.java:368)
> > > > at
> > > org.apache.oozie.service.Services.setService(Services.java:354)
> > > >
> > > > at
> > > > org.apache.oozie.service.Services.loadServices(Services.java:287)
> > > > ... 26 more
> > > >
> > > >
> > > > What do I do wrong?
> > > >
> > >
> >
>
Re: HowTo configure purge for coordinator materializations?
Posted by Robert Kanter <rk...@cloudera.com>.
CDH does have that property. In both CDH and Apache distributions, that
property (and some others) is specified in oozie-default.xml. There should
be a copy of its somewhere. Oozie has its own internal copy of
oozie-default.xml, so don't bother editing the one you find; Oozie loads
its oozie-default.xml first and then loads oozie-site.xml with any
properties you set (Hadoop works the same way with core-default.xml and
core-site.xml).
The NPE is because you only have the two services, PurgeService and
SchedulerService,
loaded because you overrode the oozie.services property. The NPE occurs
because Oozie does need the ConfigurationService for sure to work properly;
and without many of the other services, it will likely have other issues.
You should remove your override of oozie.services from oozie-site.xml.
Check oozie-default.xml and see if it lists PurgeService under
oozie.services. If it does not, then you can add it in your oozie-site.xml
in the oozie.services.ext property; if it does, then the PurgeService is
enabled.
- Robert
On Wed, Oct 2, 2013 at 10:49 AM, Serega Sheypak <se...@gmail.com>wrote:
> Hi Robert, thanks for reply.
>
> Cloudera distirbution doesn't have property oozie.services
> 1. I've tried to set all props for purge service with purge period = 100
> (seconds???) Nothing happened for half an hour
>
> 2. I've tired to explicitly declare PurgeService and got NPE.
>
> I don't understand how it works at all... I don't see expected behaviour.
>
>
> 2013/10/2 Robert Kanter <rk...@cloudera.com>
>
> > Hi Serega,
> >
> > oozie-default.xml specifies a number of oozie services to load via the
> > "oozie.services" property; but specifying
> > <property> <name>oozie.services</name> <value>
> > org.apache.oozie.service.PurgeService,
> > org.apache.oozie.service.SchedulerService </value> </property>
> > you've now removed all but 2 of them. You should remove that property
> from
> > your oozie-site.xml.
> >
> > If you want to add additional services, you should use the
> > "oozie.services.ext" property instead. That said, the PurgeService
> should
> > already be listed under "oozie.services" so it should be running by
> > default.
> >
> > - Robert
> >
> >
> > On Wed, Oct 2, 2013 at 2:06 AM, Serega Sheypak <serega.sheypak@gmail.com
> > >wrote:
> >
> > > Hi, we are using Oozie vervsion Oozie BUILD_VERSION [3.3.2-cdh4.3.0]
> > > compiled by [jenkins] on [2013.05.28-04:29:38GMT]
> > >
> > > I want to PurgeService work.
> > > I did set these props:
> > > <!-- Purging old materializations--> <!-- Keep them for three months
> -->
> > > <property> <name>oozie.service.PurgeService.older.than</name>
> > > <value>90</value> </property> <property>
> > > <name>oozie.service.PurgeService.coord.older.than</name>
> > <value>90</value>
> > > </property> <property>
> > <name>oozie.service.PurgeService.purge.limit</name>
> > > <value>100</value> </property> <property>
> > > <name>oozie.service.PurgeService.purge.interval</name>
> <value>100</value>
> > > </property>
> > >
> > >
> > > The idea is: keep actions data for 90 dayes or not more than last 100
> > > items.
> > > oozie.service.PurgeService.purge.interval is set to 100 seconds.
> > > Nothing happened after 100 seconds.
> > >
> > > I've tried to add this section:
> > > <property> <name>oozie.services</name> <value>
> > > org.apache.oozie.service.PurgeService,
> > > org.apache.oozie.service.SchedulerService </value> </property>
> > >
> > > Now Ooize is not starting:
> > >
> > >
> > > org.apache.oozie.service.ServiceException: E0103: Could not load
> > > service classes, null
> > > at
> > > org.apache.oozie.service.Services.loadServices(Services.java:291)
> > > at org.apache.oozie.service.Services.init(Services.java:208)
> > >
> > > at
> > >
> >
> org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:39)
> > > at
> > >
> >
> org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4206)
> > > at
> > >
> org.apache.catalina.core.StandardContext.start(StandardContext.java:4705)
> > >
> > > at
> > >
> >
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
> > > at
> > > org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
> > > at
> > > org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
> > >
> > > at
> > > org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:943)
> > > at
> > > org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:778)
> > > at
> > > org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:504)
> > >
> > > at
> > > org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317)
> > > at
> > >
> >
> org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
> > > at
> > >
> >
> org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
> > >
> > > at
> > > org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065)
> > > at
> > > org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
> > > at
> > > org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057)
> > >
> > > at
> > > org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
> > > at
> > >
> org.apache.catalina.core.StandardService.start(StandardService.java:525)
> > > at
> > > org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
> > >
> > > at
> org.apache.catalina.startup.Catalina.start(Catalina.java:595)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > at
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > >
> > > at
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > > at java.lang.reflect.Method.invoke(Method.java:597)
> > > at
> > org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
> > > at
> org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
> > >
> > > Caused by: java.lang.NullPointerException
> > > at
> > org.apache.oozie.service.PurgeService.init(PurgeService.java:76)
> > > at
> > > org.apache.oozie.service.Services.setServiceInternal(Services.java:368)
> > > at
> > org.apache.oozie.service.Services.setService(Services.java:354)
> > >
> > > at
> > > org.apache.oozie.service.Services.loadServices(Services.java:287)
> > > ... 26 more
> > >
> > >
> > > What do I do wrong?
> > >
> >
>
Re: HowTo configure purge for coordinator materializations?
Posted by Serega Sheypak <se...@gmail.com>.
Hi Robert, thanks for reply.
Cloudera distirbution doesn't have property oozie.services
1. I've tried to set all props for purge service with purge period = 100
(seconds???) Nothing happened for half an hour
2. I've tired to explicitly declare PurgeService and got NPE.
I don't understand how it works at all... I don't see expected behaviour.
2013/10/2 Robert Kanter <rk...@cloudera.com>
> Hi Serega,
>
> oozie-default.xml specifies a number of oozie services to load via the
> "oozie.services" property; but specifying
> <property> <name>oozie.services</name> <value>
> org.apache.oozie.service.PurgeService,
> org.apache.oozie.service.SchedulerService </value> </property>
> you've now removed all but 2 of them. You should remove that property from
> your oozie-site.xml.
>
> If you want to add additional services, you should use the
> "oozie.services.ext" property instead. That said, the PurgeService should
> already be listed under "oozie.services" so it should be running by
> default.
>
> - Robert
>
>
> On Wed, Oct 2, 2013 at 2:06 AM, Serega Sheypak <serega.sheypak@gmail.com
> >wrote:
>
> > Hi, we are using Oozie vervsion Oozie BUILD_VERSION [3.3.2-cdh4.3.0]
> > compiled by [jenkins] on [2013.05.28-04:29:38GMT]
> >
> > I want to PurgeService work.
> > I did set these props:
> > <!-- Purging old materializations--> <!-- Keep them for three months -->
> > <property> <name>oozie.service.PurgeService.older.than</name>
> > <value>90</value> </property> <property>
> > <name>oozie.service.PurgeService.coord.older.than</name>
> <value>90</value>
> > </property> <property>
> <name>oozie.service.PurgeService.purge.limit</name>
> > <value>100</value> </property> <property>
> > <name>oozie.service.PurgeService.purge.interval</name> <value>100</value>
> > </property>
> >
> >
> > The idea is: keep actions data for 90 dayes or not more than last 100
> > items.
> > oozie.service.PurgeService.purge.interval is set to 100 seconds.
> > Nothing happened after 100 seconds.
> >
> > I've tried to add this section:
> > <property> <name>oozie.services</name> <value>
> > org.apache.oozie.service.PurgeService,
> > org.apache.oozie.service.SchedulerService </value> </property>
> >
> > Now Ooize is not starting:
> >
> >
> > org.apache.oozie.service.ServiceException: E0103: Could not load
> > service classes, null
> > at
> > org.apache.oozie.service.Services.loadServices(Services.java:291)
> > at org.apache.oozie.service.Services.init(Services.java:208)
> >
> > at
> >
> org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:39)
> > at
> >
> org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4206)
> > at
> > org.apache.catalina.core.StandardContext.start(StandardContext.java:4705)
> >
> > at
> >
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
> > at
> > org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
> > at
> > org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
> >
> > at
> > org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:943)
> > at
> > org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:778)
> > at
> > org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:504)
> >
> > at
> > org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317)
> > at
> >
> org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
> > at
> >
> org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
> >
> > at
> > org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065)
> > at
> > org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
> > at
> > org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057)
> >
> > at
> > org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
> > at
> > org.apache.catalina.core.StandardService.start(StandardService.java:525)
> > at
> > org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
> >
> > at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
> > at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
> >
> > Caused by: java.lang.NullPointerException
> > at
> org.apache.oozie.service.PurgeService.init(PurgeService.java:76)
> > at
> > org.apache.oozie.service.Services.setServiceInternal(Services.java:368)
> > at
> org.apache.oozie.service.Services.setService(Services.java:354)
> >
> > at
> > org.apache.oozie.service.Services.loadServices(Services.java:287)
> > ... 26 more
> >
> >
> > What do I do wrong?
> >
>
Re: HowTo configure purge for coordinator materializations?
Posted by Robert Kanter <rk...@cloudera.com>.
Hi Serega,
oozie-default.xml specifies a number of oozie services to load via the
"oozie.services" property; but specifying
<property> <name>oozie.services</name> <value>
org.apache.oozie.service.PurgeService,
org.apache.oozie.service.SchedulerService </value> </property>
you've now removed all but 2 of them. You should remove that property from
your oozie-site.xml.
If you want to add additional services, you should use the
"oozie.services.ext" property instead. That said, the PurgeService should
already be listed under "oozie.services" so it should be running by
default.
- Robert
On Wed, Oct 2, 2013 at 2:06 AM, Serega Sheypak <se...@gmail.com>wrote:
> Hi, we are using Oozie vervsion Oozie BUILD_VERSION [3.3.2-cdh4.3.0]
> compiled by [jenkins] on [2013.05.28-04:29:38GMT]
>
> I want to PurgeService work.
> I did set these props:
> <!-- Purging old materializations--> <!-- Keep them for three months -->
> <property> <name>oozie.service.PurgeService.older.than</name>
> <value>90</value> </property> <property>
> <name>oozie.service.PurgeService.coord.older.than</name> <value>90</value>
> </property> <property> <name>oozie.service.PurgeService.purge.limit</name>
> <value>100</value> </property> <property>
> <name>oozie.service.PurgeService.purge.interval</name> <value>100</value>
> </property>
>
>
> The idea is: keep actions data for 90 dayes or not more than last 100
> items.
> oozie.service.PurgeService.purge.interval is set to 100 seconds.
> Nothing happened after 100 seconds.
>
> I've tried to add this section:
> <property> <name>oozie.services</name> <value>
> org.apache.oozie.service.PurgeService,
> org.apache.oozie.service.SchedulerService </value> </property>
>
> Now Ooize is not starting:
>
>
> org.apache.oozie.service.ServiceException: E0103: Could not load
> service classes, null
> at
> org.apache.oozie.service.Services.loadServices(Services.java:291)
> at org.apache.oozie.service.Services.init(Services.java:208)
>
> at
> org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:39)
> at
> org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4206)
> at
> org.apache.catalina.core.StandardContext.start(StandardContext.java:4705)
>
> at
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
> at
> org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
> at
> org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
>
> at
> org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:943)
> at
> org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:778)
> at
> org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:504)
>
> at
> org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317)
> at
> org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
> at
> org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
>
> at
> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065)
> at
> org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
> at
> org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057)
>
> at
> org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
> at
> org.apache.catalina.core.StandardService.start(StandardService.java:525)
> at
> org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
>
> at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
> at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
>
> Caused by: java.lang.NullPointerException
> at org.apache.oozie.service.PurgeService.init(PurgeService.java:76)
> at
> org.apache.oozie.service.Services.setServiceInternal(Services.java:368)
> at org.apache.oozie.service.Services.setService(Services.java:354)
>
> at
> org.apache.oozie.service.Services.loadServices(Services.java:287)
> ... 26 more
>
>
> What do I do wrong?
>