You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomee.apache.org by Matej <gm...@gmail.com> on 2015/02/11 15:48:13 UTC
EJB timers stop working.
Hi all.
In production under load, we experienced a case were a particular timer
stopped working.
I thought maybe a runstime exception stops a ScheduleExpression, but could
not simulate this. Glassfish I know was very sensible to this.
Could a low EJB timer thread seeting cause a timer to stop, working.
We use tomee 1.7
The timer is initialized like that.
@Singleton
@Lock(LockType.READ)
@Startup
public class CommEtaUpdateCacheTimer {
@Inject
private TimerScheduler scheduler;
@PostConstruct
public void doit() {
final ScheduleExpression schedule = new ScheduleExpression()
.hour("*")
.minute("*")
.second("*/5");
scheduler.scheduleEvent(schedule, new
CommEtaUpdateCacheTimerEvent());
}
public void observe(@Observes CommEtaUpdateCacheTimerEvent event) {
/*
processing code.
*/
}
public static class CommEtaUpdateCacheTimerEvent {
}
}
BR
Matej
Re: EJB timers stop working.
Posted by Romain Manni-Bucau <rm...@gmail.com>.
Here the log I'd check: "Timer" (note: not sure it is on by default I
didnt check its default config in tomee)
Some additional checks: with jstack check threads "EjbTimerPool*"
Also check your config of the timer pool (here some basic
conf/system.properties properties for default setup):
EjbTimerPool.CorePoolSize=10
EjbTimerPool.RejectedExecutionHandlerClass=no default for it so it
uses next property
EjbTimerPool.OfferTimeout = 30 seconds
My guess is either these threads are stucked somewhere and we need to
know (can just be a standard queueing issue due to overload) or they
are rejecting tasks but this should be logged AFAIK
Romain Manni-Bucau
@rmannibucau
http://www.tomitribe.com
http://rmannibucau.wordpress.com
https://github.com/rmannibucau
2015-02-11 17:29 GMT+01:00 jieryn <ji...@gmail.com>:
> I've seen this in production for quite a while, too, on Apache TomEE 1.7.1.
>
> I've never been able to catch a log message which showed it, we end up
> having to restart periodically in order to ensure the timers are
> reactivated. I suspect this is a user error, but .. no log messages.
> It's not easy to detect when something isn't running..
>
> Is there some class name I can grep my logs for that would help?
> Whatever the class that runs the @Scheduled @Singleton @Stateless
> @Startup beans.
>
> On Wed, Feb 11, 2015 at 9:48 AM, Matej <gm...@gmail.com> wrote:
>> Hi all.
>>
>> In production under load, we experienced a case were a particular timer
>> stopped working.
>>
>> I thought maybe a runstime exception stops a ScheduleExpression, but could
>> not simulate this. Glassfish I know was very sensible to this.
>>
>> Could a low EJB timer thread seeting cause a timer to stop, working.
>>
>> We use tomee 1.7
>>
>> The timer is initialized like that.
>>
>> @Singleton
>> @Lock(LockType.READ)
>> @Startup
>> public class CommEtaUpdateCacheTimer {
>>
>> @Inject
>> private TimerScheduler scheduler;
>>
>> @PostConstruct
>> public void doit() {
>> final ScheduleExpression schedule = new ScheduleExpression()
>> .hour("*")
>> .minute("*")
>> .second("*/5");
>>
>> scheduler.scheduleEvent(schedule, new
>> CommEtaUpdateCacheTimerEvent());
>> }
>>
>> public void observe(@Observes CommEtaUpdateCacheTimerEvent event) {
>> /*
>> processing code.
>> */
>>
>> }
>>
>> public static class CommEtaUpdateCacheTimerEvent {
>> }
>> }
>>
>> BR
>>
>> Matej
Re: EJB timers stop working.
Posted by jieryn <ji...@gmail.com>.
I've seen this in production for quite a while, too, on Apache TomEE 1.7.1.
I've never been able to catch a log message which showed it, we end up
having to restart periodically in order to ensure the timers are
reactivated. I suspect this is a user error, but .. no log messages.
It's not easy to detect when something isn't running..
Is there some class name I can grep my logs for that would help?
Whatever the class that runs the @Scheduled @Singleton @Stateless
@Startup beans.
On Wed, Feb 11, 2015 at 9:48 AM, Matej <gm...@gmail.com> wrote:
> Hi all.
>
> In production under load, we experienced a case were a particular timer
> stopped working.
>
> I thought maybe a runstime exception stops a ScheduleExpression, but could
> not simulate this. Glassfish I know was very sensible to this.
>
> Could a low EJB timer thread seeting cause a timer to stop, working.
>
> We use tomee 1.7
>
> The timer is initialized like that.
>
> @Singleton
> @Lock(LockType.READ)
> @Startup
> public class CommEtaUpdateCacheTimer {
>
> @Inject
> private TimerScheduler scheduler;
>
> @PostConstruct
> public void doit() {
> final ScheduleExpression schedule = new ScheduleExpression()
> .hour("*")
> .minute("*")
> .second("*/5");
>
> scheduler.scheduleEvent(schedule, new
> CommEtaUpdateCacheTimerEvent());
> }
>
> public void observe(@Observes CommEtaUpdateCacheTimerEvent event) {
> /*
> processing code.
> */
>
> }
>
> public static class CommEtaUpdateCacheTimerEvent {
> }
> }
>
> BR
>
> Matej
Re: EJB timers stop working.
Posted by Romain Manni-Bucau <rm...@gmail.com>.
Hmm, not having any log is weird if you dont have a JVM crash. If you
manage to reproduce it having at least a thread stack dump (using
jstack for instance) would help
Romain Manni-Bucau
@rmannibucau
http://www.tomitribe.com
http://rmannibucau.wordpress.com
https://github.com/rmannibucau
2015-02-11 16:04 GMT+01:00 Matej <gm...@gmail.com>:
> No log was noticed. But maybe we missed it. We will look further. But
> definitely no periodic log.
>
> Like that:
>
> @Singleton
> @Lock(LockType.READ)
> public class TimerScheduler {
>
> @Resource
> private TimerService timerService;
> @Inject
> Logger logger;
> @Resource
> private BeanManager beanManager;
>
> public void scheduleEvent(ScheduleExpression schedule, Object event,
> Annotation... qualifiers) {
>
> timerService.createCalendarTimer(schedule, new TimerConfig(new
> EventConfig(event, qualifiers), false));
> }
>
> @Timeout
> private void timeout(Timer timer) {
> final EventConfig config = (EventConfig) timer.getInfo();
>
> beanManager.fireEvent(config.getEvent(), config.getQualifiers());
> }
>
> private final class EventConfig implements Serializable {
>
> private final Object event;
> private final Annotation[] qualifiers;
>
> private EventConfig(Object event, Annotation[] qualifiers) {
> this.event = event;
> this.qualifiers = qualifiers;
> }
>
> public Object getEvent() {
> return event;
> }
>
> public Annotation[] getQualifiers() {
> return qualifiers;
> }
> }
> }
>
> 2015-02-11 15:54 GMT+01:00 Romain Manni-Bucau <rm...@gmail.com>:
>
>> Hi
>>
>> how do you use TimerService? No log at all saying there is an issue?
>> What's @Timeout method?
>>
>>
>> Romain Manni-Bucau
>> @rmannibucau
>> http://www.tomitribe.com
>> http://rmannibucau.wordpress.com
>> https://github.com/rmannibucau
>>
>>
>> 2015-02-11 15:48 GMT+01:00 Matej <gm...@gmail.com>:
>> > Hi all.
>> >
>> > In production under load, we experienced a case were a particular timer
>> > stopped working.
>> >
>> > I thought maybe a runstime exception stops a ScheduleExpression, but
>> could
>> > not simulate this. Glassfish I know was very sensible to this.
>> >
>> > Could a low EJB timer thread seeting cause a timer to stop, working.
>> >
>> > We use tomee 1.7
>> >
>> > The timer is initialized like that.
>> >
>> > @Singleton
>> > @Lock(LockType.READ)
>> > @Startup
>> > public class CommEtaUpdateCacheTimer {
>> >
>> > @Inject
>> > private TimerScheduler scheduler;
>> >
>> > @PostConstruct
>> > public void doit() {
>> > final ScheduleExpression schedule = new ScheduleExpression()
>> > .hour("*")
>> > .minute("*")
>> > .second("*/5");
>> >
>> > scheduler.scheduleEvent(schedule, new
>> > CommEtaUpdateCacheTimerEvent());
>> > }
>> >
>> > public void observe(@Observes CommEtaUpdateCacheTimerEvent event) {
>> > /*
>> > processing code.
>> > */
>> >
>> > }
>> >
>> > public static class CommEtaUpdateCacheTimerEvent {
>> > }
>> > }
>> >
>> > BR
>> >
>> > Matej
>>
Re: EJB timers stop working.
Posted by Romain Manni-Bucau <rm...@gmail.com>.
cause of @Lock(LockType.READ) it shouldn't be the case
Romain Manni-Bucau
@rmannibucau
http://www.tomitribe.com
http://rmannibucau.wordpress.com
https://github.com/rmannibucau
2015-02-11 16:32 GMT+01:00 romych <ro...@mail.ru>:
> @Timeout private void timeout(Timer timer) { final EventConfig
> config = (EventConfig) timer.getInfo();
> beanManager.fireEvent(config.getEvent(), config.getQualifiers()); }
> There may be problem in concurency management. May be previous work hangs
> inside timeout method. Then timer can not call timeout method, it works like
> synchronized.You may try to annotate singleton class with
> @ConcurrencyManagement(ConcurrencyManagementType.BEAN).And use something
> about on timeout method:
> private static final Lock inTimer = new ReentrantLock();@Timeoutprivate void
> timeout(Timer timer) { if (inTimer.tryLock(5000, TimeUnit.MILLISECONDS))
> { try { doWork(); } finally { inTimer.unlock(); } } else {
> logger.warn("Can not get lock!"); }}
> So you can see where problem is.
>
>
>
> --
> View this message in context: http://tomee-openejb.979440.n4.nabble.com/EJB-timers-stop-working-tp4673750p4673755.html
> Sent from the TomEE Users mailing list archive at Nabble.com.
Re: EJB timers stop working.
Posted by romych <ro...@mail.ru>.
@Timeout private void timeout(Timer timer) { final EventConfig
config = (EventConfig) timer.getInfo();
beanManager.fireEvent(config.getEvent(), config.getQualifiers()); }
There may be problem in concurency management. May be previous work hangs
inside timeout method. Then timer can not call timeout method, it works like
synchronized.You may try to annotate singleton class with
@ConcurrencyManagement(ConcurrencyManagementType.BEAN).And use something
about on timeout method:
private static final Lock inTimer = new ReentrantLock();@Timeoutprivate void
timeout(Timer timer) { if (inTimer.tryLock(5000, TimeUnit.MILLISECONDS))
{ try { doWork(); } finally { inTimer.unlock(); } } else {
logger.warn("Can not get lock!"); }}
So you can see where problem is.
--
View this message in context: http://tomee-openejb.979440.n4.nabble.com/EJB-timers-stop-working-tp4673750p4673755.html
Sent from the TomEE Users mailing list archive at Nabble.com.
Re: EJB timers stop working.
Posted by Matej <gm...@gmail.com>.
No log was noticed. But maybe we missed it. We will look further. But
definitely no periodic log.
Like that:
@Singleton
@Lock(LockType.READ)
public class TimerScheduler {
@Resource
private TimerService timerService;
@Inject
Logger logger;
@Resource
private BeanManager beanManager;
public void scheduleEvent(ScheduleExpression schedule, Object event,
Annotation... qualifiers) {
timerService.createCalendarTimer(schedule, new TimerConfig(new
EventConfig(event, qualifiers), false));
}
@Timeout
private void timeout(Timer timer) {
final EventConfig config = (EventConfig) timer.getInfo();
beanManager.fireEvent(config.getEvent(), config.getQualifiers());
}
private final class EventConfig implements Serializable {
private final Object event;
private final Annotation[] qualifiers;
private EventConfig(Object event, Annotation[] qualifiers) {
this.event = event;
this.qualifiers = qualifiers;
}
public Object getEvent() {
return event;
}
public Annotation[] getQualifiers() {
return qualifiers;
}
}
}
2015-02-11 15:54 GMT+01:00 Romain Manni-Bucau <rm...@gmail.com>:
> Hi
>
> how do you use TimerService? No log at all saying there is an issue?
> What's @Timeout method?
>
>
> Romain Manni-Bucau
> @rmannibucau
> http://www.tomitribe.com
> http://rmannibucau.wordpress.com
> https://github.com/rmannibucau
>
>
> 2015-02-11 15:48 GMT+01:00 Matej <gm...@gmail.com>:
> > Hi all.
> >
> > In production under load, we experienced a case were a particular timer
> > stopped working.
> >
> > I thought maybe a runstime exception stops a ScheduleExpression, but
> could
> > not simulate this. Glassfish I know was very sensible to this.
> >
> > Could a low EJB timer thread seeting cause a timer to stop, working.
> >
> > We use tomee 1.7
> >
> > The timer is initialized like that.
> >
> > @Singleton
> > @Lock(LockType.READ)
> > @Startup
> > public class CommEtaUpdateCacheTimer {
> >
> > @Inject
> > private TimerScheduler scheduler;
> >
> > @PostConstruct
> > public void doit() {
> > final ScheduleExpression schedule = new ScheduleExpression()
> > .hour("*")
> > .minute("*")
> > .second("*/5");
> >
> > scheduler.scheduleEvent(schedule, new
> > CommEtaUpdateCacheTimerEvent());
> > }
> >
> > public void observe(@Observes CommEtaUpdateCacheTimerEvent event) {
> > /*
> > processing code.
> > */
> >
> > }
> >
> > public static class CommEtaUpdateCacheTimerEvent {
> > }
> > }
> >
> > BR
> >
> > Matej
>
Re: EJB timers stop working.
Posted by Romain Manni-Bucau <rm...@gmail.com>.
Hi
how do you use TimerService? No log at all saying there is an issue?
What's @Timeout method?
Romain Manni-Bucau
@rmannibucau
http://www.tomitribe.com
http://rmannibucau.wordpress.com
https://github.com/rmannibucau
2015-02-11 15:48 GMT+01:00 Matej <gm...@gmail.com>:
> Hi all.
>
> In production under load, we experienced a case were a particular timer
> stopped working.
>
> I thought maybe a runstime exception stops a ScheduleExpression, but could
> not simulate this. Glassfish I know was very sensible to this.
>
> Could a low EJB timer thread seeting cause a timer to stop, working.
>
> We use tomee 1.7
>
> The timer is initialized like that.
>
> @Singleton
> @Lock(LockType.READ)
> @Startup
> public class CommEtaUpdateCacheTimer {
>
> @Inject
> private TimerScheduler scheduler;
>
> @PostConstruct
> public void doit() {
> final ScheduleExpression schedule = new ScheduleExpression()
> .hour("*")
> .minute("*")
> .second("*/5");
>
> scheduler.scheduleEvent(schedule, new
> CommEtaUpdateCacheTimerEvent());
> }
>
> public void observe(@Observes CommEtaUpdateCacheTimerEvent event) {
> /*
> processing code.
> */
>
> }
>
> public static class CommEtaUpdateCacheTimerEvent {
> }
> }
>
> BR
>
> Matej