You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@servicemix.apache.org by Ryan Moquin <fr...@gmail.com> on 2009/09/01 01:38:33 UTC
Servicemix 4.0 endpoints inconsistently shutting down.
I wanted to reask about SAs in Servicemix 4.0. When I start Servicemix 4.0
fresh (no data directory), certain SAs that it deploys, will end up in the
stopped state. There are no errors, and it seems to be random. If I stop
Servicemix, delete the data directory and start it back up, then different
SAs will be stopped, or sometimes they'll all start up and be fine. It
appears that if they do all start up, then they seem to start everytime if I
restart the server. It just seems like on a fresh start, sometimes an SA
won't start up properly. Is there anyway to find out the reason why an SA
doesn't start? I don't see anything in the log, is there something I can
turn on to see what might be causing the intermittent problem?
If I try to access an external endpoint on one of the SAs that don't show up
in jbi/list (such as a CXF endpoint), I'll see this error (it's the only
error I see).
Interceptor has thrown exception, unwinding now
org.apache.cxf.interceptor.Fault: Endpoint is stopped
at
org.apache.servicemix.cxfbc.CxfBcConsumer$1.handleMessage(CxfBcConsumer.java:440)
at
org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:236)
at
org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:89)
at
org.apache.servicemix.cxfbc.CxfBcConsumer$JbiChainInitiationObserver.onMessage(CxfBcConsumer.java:646)
at
org.apache.cxf.transport.http_jetty.JettyHTTPDestination.serviceRequest(JettyHTTPDestination.java:302)
at
org.apache.cxf.transport.http_jetty.JettyHTTPDestination.doService(JettyHTTPDestination.java:266)
at
org.apache.cxf.transport.http_jetty.JettyHTTPHandler.handle(JettyHTTPHandler.java:70)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerList.handle(HandlerList.java:49)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
at
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
Caused by: java.lang.Exception: Endpoint is stopped
... 19 more
Re: Servicemix 4.0 endpoints inconsistently shutting down.
Posted by Ryan Moquin <fr...@gmail.com>.
I believe this issue of mine is related to this issue I just filed:
https://issues.apache.org/activemq/browse/SMX4-361
On Tue, Sep 1, 2009 at 12:42 AM, Gert Vanthienen
<ge...@gmail.com>wrote:
> Ryan,
>
> I guess you already enabled DEBUG logging for this? What does
> osgi/list say about the bundles/SA/XML files you are using? Does it
> mark the bundles as Active -- if not, something might have gone wrong
> at file install time?
>
> Regards,
>
> Gert Vanthienen
> ------------------------
> Open Source SOA: http://fusesource.com
> Blog: http://gertvanthienen.blogspot.com/
>
>
>
> 2009/9/1 Ryan Moquin <fr...@gmail.com>:
> > I wanted to reask about SAs in Servicemix 4.0. When I start Servicemix
> 4.0
> > fresh (no data directory), certain SAs that it deploys, will end up in
> the
> > stopped state. There are no errors, and it seems to be random. If I
> stop
> > Servicemix, delete the data directory and start it back up, then
> different
> > SAs will be stopped, or sometimes they'll all start up and be fine. It
> > appears that if they do all start up, then they seem to start everytime
> if I
> > restart the server. It just seems like on a fresh start, sometimes an SA
> > won't start up properly. Is there anyway to find out the reason why an
> SA
> > doesn't start? I don't see anything in the log, is there something I can
> > turn on to see what might be causing the intermittent problem?
> >
> > If I try to access an external endpoint on one of the SAs that don't show
> up
> > in jbi/list (such as a CXF endpoint), I'll see this error (it's the only
> > error I see).
> >
> > Interceptor has thrown exception, unwinding now
> > org.apache.cxf.interceptor.Fault: Endpoint is stopped
> > at
> >
> org.apache.servicemix.cxfbc.CxfBcConsumer$1.handleMessage(CxfBcConsumer.java:440)
> > at
> >
> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:236)
> > at
> >
> org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:89)
> > at
> >
> org.apache.servicemix.cxfbc.CxfBcConsumer$JbiChainInitiationObserver.onMessage(CxfBcConsumer.java:646)
> > at
> >
> org.apache.cxf.transport.http_jetty.JettyHTTPDestination.serviceRequest(JettyHTTPDestination.java:302)
> > at
> >
> org.apache.cxf.transport.http_jetty.JettyHTTPDestination.doService(JettyHTTPDestination.java:266)
> > at
> >
> org.apache.cxf.transport.http_jetty.JettyHTTPHandler.handle(JettyHTTPHandler.java:70)
> > at
> > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> > at
> >
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> > at org.mortbay.jetty.handler.HandlerList.handle(HandlerList.java:49)
> > at
> > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> > at org.mortbay.jetty.Server.handle(Server.java:324)
> > at
> > org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
> > at
> >
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
> > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
> > at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
> > at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
> > at
> >
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
> > at
> >
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
> > Caused by: java.lang.Exception: Endpoint is stopped
> > ... 19 more
> >
>
Re: Servicemix 4.0 endpoints inconsistently shutting down.
Posted by Ryan Moquin <fr...@gmail.com>.
One more strange thing I noticed. I started Servicemix up, everything
deployed fine. I then had to shut it down and restart it. when I did it
started undeploying some of my components and SAs. I noticed some debug
messages like this:
21:25:33,671 | DEBUG | xtenderThread-28 | XBeanNamespaceHandler |
ontext.v2c.XBeanNamespaceHandler 839 | Could not find resource:
META-INF/services/org/apache/xbean/spring/http/
service.com/definitions/taskDescription
I see the reference that it's talking about in one of the jars in the SE
component that is deployed. Though, it's part defined in a different jar
than the jar that is created for the SE. Is that now a limitation in
Servicemix 4.0? I didn't have any issue like this in Servicemix 3.3.1.
I'm thoroughly confused. That looks to be the only thing I can find that
remotely looks like an error.
On Tue, Sep 1, 2009 at 8:52 PM, Ryan Moquin <fr...@gmail.com> wrote:
> Just for some extra info, on that last run where only some stuff started,
> for one of the SAs that only made it to the resolved state, this is the last
> bit of the log that mentions that SA:
>
> 19:33:26,937 | DEBUG | pool-1-thread-1 | Activator
> | org.apache.camel.osgi.Activator 73 | Bundle resolved:
> tasking-service-sa
> 19:33:26,937 | DEBUG | lixDispatchQueue | tasking-service-sa |
> ? ? | BundleEvent RESOLVED
> 19:33:26,937 | DEBUG | lixDispatchQueue | tasking-service-sa |
> ? ? | BundleEvent INSTALLED
>
> It doesn't mention that SA anywhere else in the log after that.
>
>
> On Tue, Sep 1, 2009 at 7:41 PM, Ryan Moquin <fr...@gmail.com>wrote:
>
>> Also, I would say the most consistent situation I seem to hit is where all
>> of the Servicemix components install, my components install, and only 2 or 3
>> of my SAs install and the other 4 or 5 don't install.
>>
>> I didn't notice you said osgi/list, I was doing jbi/list. I just deleted
>> the data directory and restarted, I see all my stuff as listed Active in
>> osgi/list and the Spring column is blank. jbi/list shows all my stuff as
>> started. I'll go ahead and try it again since it decided to deploy
>> everything this time.
>>
>> This time, I deleted the data directory and started servicemix back up.
>> This time when it stops deploying things, all 4 of my components show as
>> Active, 5 of my SAs show as Active and 3 of my SAs show as resolved. If I
>> look at jbi/list, I see all 4 of my components and only 2 SAs out of the 8
>> total show as started. The other 6 aren't even listed. This time when I
>> was doing it, I was emptying my recycle bin. It seems to me like the
>> majority of the time when not everything finishes, some other process was
>> working on my machine as well. Could some of the deployments be timing
>> out? It's been sitting for a while and none of the statuses have changed.
>>
>> I'm not really sure what's going on but it seems like some of the
>> components just get abandoned ...
>>
>> Ryan
>>
>>
>> On Tue, Sep 1, 2009 at 7:13 PM, Ryan Moquin <fr...@gmail.com>wrote:
>>
>>> I didn't set it to debug, since I figured if there was an error it would
>>> tell me but like I said, it non-deterministically will sometimes only start
>>> 4 total components and no SA's or it will start all of them... or it will
>>> start any number in between. If I delete the data directory and restart,
>>> then a different number of components/SAs will start without me modifying
>>> anything other than deleting the data directory.
>>>
>>> The worst case I've seen so far, was 4 of my components all started and
>>> non of the servicemix components started. I was getting all kinds of weird
>>> activemq errors about not being able to connect to 0.0.0. I should have
>>> saved some of the log. I was getting weird JAXB errors as well.. basically
>>> everything went nuts. I then deleted the data directory and everything
>>> started up without any issues.
>>>
>>> Anyhow, I turned on debug, deleted the data directory and started up. 3
>>> of my SA's didn't start, but all of my components and servicemix components
>>> started. Now, I don't see any errors, this is a snippet from the log
>>> regarding one of my SAs which rely on servicemix-cxf-se (which hadn't been
>>> installed as of this point):
>>>
>>> 18:38:21,828 | DEBUG | pool-1-thread-1 |
>>> BundleWatcher | .swissbox.extender.BundleWatcher 176 |
>>> Scanning bundle [process-service-sa]
>>> 18:38:21,828 | DEBUG | pool-1-thread-1 |
>>> Deployer | cemix.jbi.deployer.impl.Deployer 301 |
>>> Checking bundle: 'null (process-service-sa)'
>>> 18:38:21,859 | INFO | pool-1-thread-1 |
>>> Deployer | cemix.jbi.deployer.impl.Deployer 343 |
>>> Deploying bundle 'null (process-service-sa)' as a JBI service assembly
>>> 18:38:21,906 | WARN | pool-1-thread-1 |
>>> Deployer | cemix.jbi.deployer.impl.Deployer 359 |
>>> Requirements not met for JBI artifact in bundle null (process-service-sa).
>>> Installation pending.
>>> org.apache.servicemix.jbi.deployer.impl.PendingException: Component not
>>> installed: servicemix-cxf-se
>>> 18:38:21,921 | DEBUG | pool-1-thread-1 |
>>> ContextLoaderListener | .activator.ContextLoaderListener 709 |
>>> Scanning bundle [null (process-service-sa)] for configurations...
>>> 18:38:21,921 | DEBUG | pool-1-thread-1 |
>>> ContextLoaderListener | .activator.ContextLoaderListener 715 |
>>> Creating an application context for bundle [null (process-service-sa)]
>>> 18:38:21,921 | DEBUG | pool-1-thread-1 |
>>> ContextLoaderListener | .activator.ContextLoaderListener 726 |
>>> No application context created for bundle [null (process-service-sa)]
>>> 18:38:21,921 | INFO | pool-1-thread-1 |
>>> FileMonitor | x.kernel.filemonitor.FileMonitor 550 |
>>> Started: process-service-sa [119]
>>> 18:38:21,921 | DEBUG | lixDispatchQueue |
>>> process-service-sa | ? ?
>>> | BundleEvent STARTED
>>>
>>> After this point in the log, there isn't a single mention of that SA
>>> again anywhere in the rest of the log.... About 4000 lines down from the
>>> above segment in the log file, the servicemix-cxf-se component bundle gets
>>> deployed, but it never finishes installing process-service-sa.
>>>
>>> 18:38:29,343 | DEBUG | Thread-5 |
>>> BundleWatcher | .swissbox.extender.BundleWatcher 176 |
>>> Scanning bundle [servicemix-cxf-se]
>>> 18:38:29,359 | DEBUG | Thread-5 |
>>> Deployer | cemix.jbi.deployer.impl.Deployer 301 |
>>> Checking bundle: 'ServiceMix :: CXF Service Engine (servicemix-cxf-se)'
>>> 18:38:29,359 | DEBUG | Thread-5 |
>>> Deployer | cemix.jbi.deployer.impl.Deployer 304 |
>>> Bundle 'ServiceMix :: CXF Service Engine (servicemix-cxf-se)' does not
>>> contain any JBI descriptor.
>>> 18:38:29,359 | DEBUG | Thread-5 |
>>> ContextLoaderListener | .activator.ContextLoaderListener 709 |
>>> Scanning bundle [ServiceMix :: CXF Service Engine (servicemix-cxf-se)] for
>>> configurations...
>>> 18:38:29,359 | DEBUG | Thread-5 |
>>> ContextLoaderListener | .activator.ContextLoaderListener 715 |
>>> Creating an application context for bundle [ServiceMix :: CXF Service Engine
>>> (servicemix-cxf-se)]
>>> 18:38:29,359 | INFO | Thread-5 |
>>> ultOsgiApplicationContextCreator | ultOsgiApplicationContextCreator 67 |
>>> Discovered configurations {osgibundle:/META-INF/spring/*.xml} in bundle
>>> [ServiceMix :: CXF Service Engine (servicemix-cxf-se)]
>>> 18:38:29,359 | DEBUG | Thread-5 |
>>> ContextLoaderListener | .activator.ContextLoaderListener 735 |
>>> Bundle [ServiceMix :: CXF Service Engine (servicemix-cxf-se)] is Spring type
>>> compatible with Spring-DM
>>> 18:38:29,359 | DEBUG | Thread-5 |
>>> ContextLoaderListener | .activator.ContextLoaderListener 784 |
>>> Asynchronous context creation for bundle [ServiceMix :: CXF Service Engine
>>> (servicemix-cxf-se)]
>>> 18:38:29,359 | DEBUG | lixDispatchQueue |
>>> servicemix-cxf-se | ? ? |
>>> BundleEvent STARTED
>>> 18:38:29,359 | DEBUG | xtenderThread-49 |
>>> WaiterApplicationContextExecutor | WaiterApplicationContextExecutor 162 |
>>> Starting first stage of refresh for
>>> OsgiBundleXmlApplicationContext(bundle=servicemix-cxf-se,
>>> config=osgibundle:/META-INF/spring/*.xml)
>>> 18:38:29,359 | DEBUG | xtenderThread-49 |
>>> WaiterApplicationContextExecutor | WaiterApplicationContextExecutor 202 |
>>> Calling preRefresh on
>>> ....
>>>
>>> Sometimes rather than just not being resumed once the dependencies are
>>> installed, sometimes the SA is installed correctly but later is says it's
>>> undeploying it.
>>>
>>> About 9000 more lines down in the log, it finishes up installing another
>>> one of my SAs. This one starts deploying after the cxf-se, which may be the
>>> reason it actually goes through with installing it. I don't see any errors
>>> installing any of the SUs in the SA, and it appears that it finishes
>>> installing it but I don't see it listed in the console with jbi/list. Other
>>> SAs of mine install but don't show up in the console either.
>>>
>>> This is the last bit of the log where it creates the service for my SA
>>> after processing all of the SU's. After that I don't see anything about
>>> that SA anymore, or anything saying it's started...
>>>
>>> 18:39:00,343 | INFO | pool-1-thread-1 |
>>> ReflectionServiceFactoryBean | .support.JaxWsServiceFactoryBean 163 |
>>> Creating Service {http://service.com/salina/workflow}workflow-service<http://service.com/salina/workflow%7Dworkflow-service>from class com.service.WorkflowServicePort
>>> 18:39:00,343 | DEBUG | xtenderThread-64 |
>>> SaxonComponent | icemix.common.AsyncBaseLifeCycle 296 |
>>> Starting component
>>> 18:39:00,375 | DEBUG | xtenderThread-64 |
>>> SaxonComponent | icemix.common.AsyncBaseLifeCycle 304 |
>>> Component started
>>>
>>> It just seems like SAs just randomly die or never get installed. If I
>>> stop servicemix, delete the data directory, restart, then sometimes
>>> everything will fully install... I don't see any errors or any reason that
>>> those SAs don't install and like I said it's not consistent on what it
>>> installs or doesn't install. Sometimes it doesn't install all the
>>> Servicemix components either.
>>>
>>>
>>> On Tue, Sep 1, 2009 at 12:42 AM, Gert Vanthienen <
>>> gert.vanthienen@gmail.com> wrote:
>>>
>>>> Ryan,
>>>>
>>>> I guess you already enabled DEBUG logging for this? What does
>>>> osgi/list say about the bundles/SA/XML files you are using? Does it
>>>> mark the bundles as Active -- if not, something might have gone wrong
>>>> at file install time?
>>>>
>>>> Regards,
>>>>
>>>> Gert Vanthienen
>>>> ------------------------
>>>> Open Source SOA: http://fusesource.com
>>>> Blog: http://gertvanthienen.blogspot.com/
>>>>
>>>>
>>>>
>>>> 2009/9/1 Ryan Moquin <fr...@gmail.com>:
>>>> > I wanted to reask about SAs in Servicemix 4.0. When I start
>>>> Servicemix 4.0
>>>> > fresh (no data directory), certain SAs that it deploys, will end up in
>>>> the
>>>> > stopped state. There are no errors, and it seems to be random. If I
>>>> stop
>>>> > Servicemix, delete the data directory and start it back up, then
>>>> different
>>>> > SAs will be stopped, or sometimes they'll all start up and be fine.
>>>> It
>>>> > appears that if they do all start up, then they seem to start
>>>> everytime if I
>>>> > restart the server. It just seems like on a fresh start, sometimes an
>>>> SA
>>>> > won't start up properly. Is there anyway to find out the reason why
>>>> an SA
>>>> > doesn't start? I don't see anything in the log, is there something I
>>>> can
>>>> > turn on to see what might be causing the intermittent problem?
>>>> >
>>>> > If I try to access an external endpoint on one of the SAs that don't
>>>> show up
>>>> > in jbi/list (such as a CXF endpoint), I'll see this error (it's the
>>>> only
>>>> > error I see).
>>>> >
>>>> > Interceptor has thrown exception, unwinding now
>>>> > org.apache.cxf.interceptor.Fault: Endpoint is stopped
>>>> > at
>>>> >
>>>> org.apache.servicemix.cxfbc.CxfBcConsumer$1.handleMessage(CxfBcConsumer.java:440)
>>>> > at
>>>> >
>>>> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:236)
>>>> > at
>>>> >
>>>> org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:89)
>>>> > at
>>>> >
>>>> org.apache.servicemix.cxfbc.CxfBcConsumer$JbiChainInitiationObserver.onMessage(CxfBcConsumer.java:646)
>>>> > at
>>>> >
>>>> org.apache.cxf.transport.http_jetty.JettyHTTPDestination.serviceRequest(JettyHTTPDestination.java:302)
>>>> > at
>>>> >
>>>> org.apache.cxf.transport.http_jetty.JettyHTTPDestination.doService(JettyHTTPDestination.java:266)
>>>> > at
>>>> >
>>>> org.apache.cxf.transport.http_jetty.JettyHTTPHandler.handle(JettyHTTPHandler.java:70)
>>>> > at
>>>> >
>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>> > at
>>>> >
>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>> > at
>>>> org.mortbay.jetty.handler.HandlerList.handle(HandlerList.java:49)
>>>> > at
>>>> >
>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>> > at org.mortbay.jetty.Server.handle(Server.java:324)
>>>> > at
>>>> >
>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
>>>> > at
>>>> >
>>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
>>>> > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
>>>> > at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
>>>> > at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
>>>> > at
>>>> >
>>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
>>>> > at
>>>> >
>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
>>>> > Caused by: java.lang.Exception: Endpoint is stopped
>>>> > ... 19 more
>>>> >
>>>>
>>>
>>>
>>
>
Re: Servicemix 4.0 endpoints inconsistently shutting down.
Posted by Ryan Moquin <fr...@gmail.com>.
Just for some extra info, on that last run where only some stuff started,
for one of the SAs that only made it to the resolved state, this is the last
bit of the log that mentions that SA:
19:33:26,937 | DEBUG | pool-1-thread-1 | Activator |
org.apache.camel.osgi.Activator 73 | Bundle resolved: tasking-service-sa
19:33:26,937 | DEBUG | lixDispatchQueue | tasking-service-sa |
? ? | BundleEvent RESOLVED
19:33:26,937 | DEBUG | lixDispatchQueue | tasking-service-sa |
? ? | BundleEvent INSTALLED
It doesn't mention that SA anywhere else in the log after that.
On Tue, Sep 1, 2009 at 7:41 PM, Ryan Moquin <fr...@gmail.com> wrote:
> Also, I would say the most consistent situation I seem to hit is where all
> of the Servicemix components install, my components install, and only 2 or 3
> of my SAs install and the other 4 or 5 don't install.
>
> I didn't notice you said osgi/list, I was doing jbi/list. I just deleted
> the data directory and restarted, I see all my stuff as listed Active in
> osgi/list and the Spring column is blank. jbi/list shows all my stuff as
> started. I'll go ahead and try it again since it decided to deploy
> everything this time.
>
> This time, I deleted the data directory and started servicemix back up.
> This time when it stops deploying things, all 4 of my components show as
> Active, 5 of my SAs show as Active and 3 of my SAs show as resolved. If I
> look at jbi/list, I see all 4 of my components and only 2 SAs out of the 8
> total show as started. The other 6 aren't even listed. This time when I
> was doing it, I was emptying my recycle bin. It seems to me like the
> majority of the time when not everything finishes, some other process was
> working on my machine as well. Could some of the deployments be timing
> out? It's been sitting for a while and none of the statuses have changed.
>
> I'm not really sure what's going on but it seems like some of the
> components just get abandoned ...
>
> Ryan
>
>
> On Tue, Sep 1, 2009 at 7:13 PM, Ryan Moquin <fr...@gmail.com>wrote:
>
>> I didn't set it to debug, since I figured if there was an error it would
>> tell me but like I said, it non-deterministically will sometimes only start
>> 4 total components and no SA's or it will start all of them... or it will
>> start any number in between. If I delete the data directory and restart,
>> then a different number of components/SAs will start without me modifying
>> anything other than deleting the data directory.
>>
>> The worst case I've seen so far, was 4 of my components all started and
>> non of the servicemix components started. I was getting all kinds of weird
>> activemq errors about not being able to connect to 0.0.0. I should have
>> saved some of the log. I was getting weird JAXB errors as well.. basically
>> everything went nuts. I then deleted the data directory and everything
>> started up without any issues.
>>
>> Anyhow, I turned on debug, deleted the data directory and started up. 3
>> of my SA's didn't start, but all of my components and servicemix components
>> started. Now, I don't see any errors, this is a snippet from the log
>> regarding one of my SAs which rely on servicemix-cxf-se (which hadn't been
>> installed as of this point):
>>
>> 18:38:21,828 | DEBUG | pool-1-thread-1 | BundleWatcher
>> | .swissbox.extender.BundleWatcher 176 | Scanning bundle
>> [process-service-sa]
>> 18:38:21,828 | DEBUG | pool-1-thread-1 | Deployer
>> | cemix.jbi.deployer.impl.Deployer 301 | Checking bundle: 'null
>> (process-service-sa)'
>> 18:38:21,859 | INFO | pool-1-thread-1 | Deployer
>> | cemix.jbi.deployer.impl.Deployer 343 | Deploying bundle 'null
>> (process-service-sa)' as a JBI service assembly
>> 18:38:21,906 | WARN | pool-1-thread-1 | Deployer
>> | cemix.jbi.deployer.impl.Deployer 359 | Requirements not met for JBI
>> artifact in bundle null (process-service-sa). Installation pending.
>> org.apache.servicemix.jbi.deployer.impl.PendingException: Component not
>> installed: servicemix-cxf-se
>> 18:38:21,921 | DEBUG | pool-1-thread-1 | ContextLoaderListener
>> | .activator.ContextLoaderListener 709 | Scanning bundle [null
>> (process-service-sa)] for configurations...
>> 18:38:21,921 | DEBUG | pool-1-thread-1 | ContextLoaderListener
>> | .activator.ContextLoaderListener 715 | Creating an application context
>> for bundle [null (process-service-sa)]
>> 18:38:21,921 | DEBUG | pool-1-thread-1 | ContextLoaderListener
>> | .activator.ContextLoaderListener 726 | No application context created for
>> bundle [null (process-service-sa)]
>> 18:38:21,921 | INFO | pool-1-thread-1 | FileMonitor
>> | x.kernel.filemonitor.FileMonitor 550 | Started: process-service-sa [119]
>> 18:38:21,921 | DEBUG | lixDispatchQueue |
>> process-service-sa | ? ?
>> | BundleEvent STARTED
>>
>> After this point in the log, there isn't a single mention of that SA again
>> anywhere in the rest of the log.... About 4000 lines down from the above
>> segment in the log file, the servicemix-cxf-se component bundle gets
>> deployed, but it never finishes installing process-service-sa.
>>
>> 18:38:29,343 | DEBUG | Thread-5 | BundleWatcher
>> | .swissbox.extender.BundleWatcher 176 | Scanning bundle
>> [servicemix-cxf-se]
>> 18:38:29,359 | DEBUG | Thread-5 | Deployer
>> | cemix.jbi.deployer.impl.Deployer 301 | Checking bundle: 'ServiceMix ::
>> CXF Service Engine (servicemix-cxf-se)'
>> 18:38:29,359 | DEBUG | Thread-5 | Deployer
>> | cemix.jbi.deployer.impl.Deployer 304 | Bundle 'ServiceMix :: CXF Service
>> Engine (servicemix-cxf-se)' does not contain any JBI descriptor.
>> 18:38:29,359 | DEBUG | Thread-5 | ContextLoaderListener
>> | .activator.ContextLoaderListener 709 | Scanning bundle [ServiceMix :: CXF
>> Service Engine (servicemix-cxf-se)] for configurations...
>> 18:38:29,359 | DEBUG | Thread-5 | ContextLoaderListener
>> | .activator.ContextLoaderListener 715 | Creating an application context
>> for bundle [ServiceMix :: CXF Service Engine (servicemix-cxf-se)]
>> 18:38:29,359 | INFO | Thread-5 | ultOsgiApplicationContextCreator
>> | ultOsgiApplicationContextCreator 67 | Discovered configurations
>> {osgibundle:/META-INF/spring/*.xml} in bundle [ServiceMix :: CXF Service
>> Engine (servicemix-cxf-se)]
>> 18:38:29,359 | DEBUG | Thread-5 | ContextLoaderListener
>> | .activator.ContextLoaderListener 735 | Bundle [ServiceMix :: CXF Service
>> Engine (servicemix-cxf-se)] is Spring type compatible with Spring-DM
>> 18:38:29,359 | DEBUG | Thread-5 | ContextLoaderListener
>> | .activator.ContextLoaderListener 784 | Asynchronous context creation for
>> bundle [ServiceMix :: CXF Service Engine (servicemix-cxf-se)]
>> 18:38:29,359 | DEBUG | lixDispatchQueue | servicemix-cxf-se
>> | ? ? | BundleEvent STARTED
>> 18:38:29,359 | DEBUG | xtenderThread-49 | WaiterApplicationContextExecutor
>> | WaiterApplicationContextExecutor 162 | Starting first stage of refresh
>> for OsgiBundleXmlApplicationContext(bundle=servicemix-cxf-se,
>> config=osgibundle:/META-INF/spring/*.xml)
>> 18:38:29,359 | DEBUG | xtenderThread-49 | WaiterApplicationContextExecutor
>> | WaiterApplicationContextExecutor 202 | Calling preRefresh on
>> ....
>>
>> Sometimes rather than just not being resumed once the dependencies are
>> installed, sometimes the SA is installed correctly but later is says it's
>> undeploying it.
>>
>> About 9000 more lines down in the log, it finishes up installing another
>> one of my SAs. This one starts deploying after the cxf-se, which may be the
>> reason it actually goes through with installing it. I don't see any errors
>> installing any of the SUs in the SA, and it appears that it finishes
>> installing it but I don't see it listed in the console with jbi/list. Other
>> SAs of mine install but don't show up in the console either.
>>
>> This is the last bit of the log where it creates the service for my SA
>> after processing all of the SU's. After that I don't see anything about
>> that SA anymore, or anything saying it's started...
>>
>> 18:39:00,343 | INFO | pool-1-thread-1 | ReflectionServiceFactoryBean
>> | .support.JaxWsServiceFactoryBean 163 | Creating Service {
>> http://service.com/salina/workflow}workflow-service<http://service.com/salina/workflow%7Dworkflow-service>from class com.service.WorkflowServicePort
>> 18:39:00,343 | DEBUG | xtenderThread-64 | SaxonComponent
>> | icemix.common.AsyncBaseLifeCycle 296 | Starting component
>> 18:39:00,375 | DEBUG | xtenderThread-64 | SaxonComponent
>> | icemix.common.AsyncBaseLifeCycle 304 | Component started
>>
>> It just seems like SAs just randomly die or never get installed. If I
>> stop servicemix, delete the data directory, restart, then sometimes
>> everything will fully install... I don't see any errors or any reason that
>> those SAs don't install and like I said it's not consistent on what it
>> installs or doesn't install. Sometimes it doesn't install all the
>> Servicemix components either.
>>
>>
>> On Tue, Sep 1, 2009 at 12:42 AM, Gert Vanthienen <
>> gert.vanthienen@gmail.com> wrote:
>>
>>> Ryan,
>>>
>>> I guess you already enabled DEBUG logging for this? What does
>>> osgi/list say about the bundles/SA/XML files you are using? Does it
>>> mark the bundles as Active -- if not, something might have gone wrong
>>> at file install time?
>>>
>>> Regards,
>>>
>>> Gert Vanthienen
>>> ------------------------
>>> Open Source SOA: http://fusesource.com
>>> Blog: http://gertvanthienen.blogspot.com/
>>>
>>>
>>>
>>> 2009/9/1 Ryan Moquin <fr...@gmail.com>:
>>> > I wanted to reask about SAs in Servicemix 4.0. When I start Servicemix
>>> 4.0
>>> > fresh (no data directory), certain SAs that it deploys, will end up in
>>> the
>>> > stopped state. There are no errors, and it seems to be random. If I
>>> stop
>>> > Servicemix, delete the data directory and start it back up, then
>>> different
>>> > SAs will be stopped, or sometimes they'll all start up and be fine. It
>>> > appears that if they do all start up, then they seem to start everytime
>>> if I
>>> > restart the server. It just seems like on a fresh start, sometimes an
>>> SA
>>> > won't start up properly. Is there anyway to find out the reason why an
>>> SA
>>> > doesn't start? I don't see anything in the log, is there something I
>>> can
>>> > turn on to see what might be causing the intermittent problem?
>>> >
>>> > If I try to access an external endpoint on one of the SAs that don't
>>> show up
>>> > in jbi/list (such as a CXF endpoint), I'll see this error (it's the
>>> only
>>> > error I see).
>>> >
>>> > Interceptor has thrown exception, unwinding now
>>> > org.apache.cxf.interceptor.Fault: Endpoint is stopped
>>> > at
>>> >
>>> org.apache.servicemix.cxfbc.CxfBcConsumer$1.handleMessage(CxfBcConsumer.java:440)
>>> > at
>>> >
>>> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:236)
>>> > at
>>> >
>>> org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:89)
>>> > at
>>> >
>>> org.apache.servicemix.cxfbc.CxfBcConsumer$JbiChainInitiationObserver.onMessage(CxfBcConsumer.java:646)
>>> > at
>>> >
>>> org.apache.cxf.transport.http_jetty.JettyHTTPDestination.serviceRequest(JettyHTTPDestination.java:302)
>>> > at
>>> >
>>> org.apache.cxf.transport.http_jetty.JettyHTTPDestination.doService(JettyHTTPDestination.java:266)
>>> > at
>>> >
>>> org.apache.cxf.transport.http_jetty.JettyHTTPHandler.handle(JettyHTTPHandler.java:70)
>>> > at
>>> >
>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>> > at
>>> >
>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>> > at org.mortbay.jetty.handler.HandlerList.handle(HandlerList.java:49)
>>> > at
>>> >
>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>> > at org.mortbay.jetty.Server.handle(Server.java:324)
>>> > at
>>> > org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
>>> > at
>>> >
>>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
>>> > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
>>> > at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
>>> > at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
>>> > at
>>> >
>>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
>>> > at
>>> >
>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
>>> > Caused by: java.lang.Exception: Endpoint is stopped
>>> > ... 19 more
>>> >
>>>
>>
>>
>
Re: Servicemix 4.0 endpoints inconsistently shutting down.
Posted by Ryan Moquin <fr...@gmail.com>.
Also, I would say the most consistent situation I seem to hit is where all
of the Servicemix components install, my components install, and only 2 or 3
of my SAs install and the other 4 or 5 don't install.
I didn't notice you said osgi/list, I was doing jbi/list. I just deleted
the data directory and restarted, I see all my stuff as listed Active in
osgi/list and the Spring column is blank. jbi/list shows all my stuff as
started. I'll go ahead and try it again since it decided to deploy
everything this time.
This time, I deleted the data directory and started servicemix back up.
This time when it stops deploying things, all 4 of my components show as
Active, 5 of my SAs show as Active and 3 of my SAs show as resolved. If I
look at jbi/list, I see all 4 of my components and only 2 SAs out of the 8
total show as started. The other 6 aren't even listed. This time when I
was doing it, I was emptying my recycle bin. It seems to me like the
majority of the time when not everything finishes, some other process was
working on my machine as well. Could some of the deployments be timing
out? It's been sitting for a while and none of the statuses have changed.
I'm not really sure what's going on but it seems like some of the components
just get abandoned ...
Ryan
On Tue, Sep 1, 2009 at 7:13 PM, Ryan Moquin <fr...@gmail.com> wrote:
> I didn't set it to debug, since I figured if there was an error it would
> tell me but like I said, it non-deterministically will sometimes only start
> 4 total components and no SA's or it will start all of them... or it will
> start any number in between. If I delete the data directory and restart,
> then a different number of components/SAs will start without me modifying
> anything other than deleting the data directory.
>
> The worst case I've seen so far, was 4 of my components all started and non
> of the servicemix components started. I was getting all kinds of weird
> activemq errors about not being able to connect to 0.0.0. I should have
> saved some of the log. I was getting weird JAXB errors as well.. basically
> everything went nuts. I then deleted the data directory and everything
> started up without any issues.
>
> Anyhow, I turned on debug, deleted the data directory and started up. 3 of
> my SA's didn't start, but all of my components and servicemix components
> started. Now, I don't see any errors, this is a snippet from the log
> regarding one of my SAs which rely on servicemix-cxf-se (which hadn't been
> installed as of this point):
>
> 18:38:21,828 | DEBUG | pool-1-thread-1 | BundleWatcher
> | .swissbox.extender.BundleWatcher 176 | Scanning bundle
> [process-service-sa]
> 18:38:21,828 | DEBUG | pool-1-thread-1 | Deployer
> | cemix.jbi.deployer.impl.Deployer 301 | Checking bundle: 'null
> (process-service-sa)'
> 18:38:21,859 | INFO | pool-1-thread-1 | Deployer
> | cemix.jbi.deployer.impl.Deployer 343 | Deploying bundle 'null
> (process-service-sa)' as a JBI service assembly
> 18:38:21,906 | WARN | pool-1-thread-1 | Deployer
> | cemix.jbi.deployer.impl.Deployer 359 | Requirements not met for JBI
> artifact in bundle null (process-service-sa). Installation pending.
> org.apache.servicemix.jbi.deployer.impl.PendingException: Component not
> installed: servicemix-cxf-se
> 18:38:21,921 | DEBUG | pool-1-thread-1 | ContextLoaderListener
> | .activator.ContextLoaderListener 709 | Scanning bundle [null
> (process-service-sa)] for configurations...
> 18:38:21,921 | DEBUG | pool-1-thread-1 | ContextLoaderListener
> | .activator.ContextLoaderListener 715 | Creating an application context
> for bundle [null (process-service-sa)]
> 18:38:21,921 | DEBUG | pool-1-thread-1 | ContextLoaderListener
> | .activator.ContextLoaderListener 726 | No application context created for
> bundle [null (process-service-sa)]
> 18:38:21,921 | INFO | pool-1-thread-1 | FileMonitor
> | x.kernel.filemonitor.FileMonitor 550 | Started: process-service-sa [119]
> 18:38:21,921 | DEBUG | lixDispatchQueue |
> process-service-sa | ? ?
> | BundleEvent STARTED
>
> After this point in the log, there isn't a single mention of that SA again
> anywhere in the rest of the log.... About 4000 lines down from the above
> segment in the log file, the servicemix-cxf-se component bundle gets
> deployed, but it never finishes installing process-service-sa.
>
> 18:38:29,343 | DEBUG | Thread-5 | BundleWatcher
> | .swissbox.extender.BundleWatcher 176 | Scanning bundle
> [servicemix-cxf-se]
> 18:38:29,359 | DEBUG | Thread-5 | Deployer
> | cemix.jbi.deployer.impl.Deployer 301 | Checking bundle: 'ServiceMix ::
> CXF Service Engine (servicemix-cxf-se)'
> 18:38:29,359 | DEBUG | Thread-5 | Deployer
> | cemix.jbi.deployer.impl.Deployer 304 | Bundle 'ServiceMix :: CXF Service
> Engine (servicemix-cxf-se)' does not contain any JBI descriptor.
> 18:38:29,359 | DEBUG | Thread-5 | ContextLoaderListener
> | .activator.ContextLoaderListener 709 | Scanning bundle [ServiceMix :: CXF
> Service Engine (servicemix-cxf-se)] for configurations...
> 18:38:29,359 | DEBUG | Thread-5 | ContextLoaderListener
> | .activator.ContextLoaderListener 715 | Creating an application context
> for bundle [ServiceMix :: CXF Service Engine (servicemix-cxf-se)]
> 18:38:29,359 | INFO | Thread-5 | ultOsgiApplicationContextCreator
> | ultOsgiApplicationContextCreator 67 | Discovered configurations
> {osgibundle:/META-INF/spring/*.xml} in bundle [ServiceMix :: CXF Service
> Engine (servicemix-cxf-se)]
> 18:38:29,359 | DEBUG | Thread-5 | ContextLoaderListener
> | .activator.ContextLoaderListener 735 | Bundle [ServiceMix :: CXF Service
> Engine (servicemix-cxf-se)] is Spring type compatible with Spring-DM
> 18:38:29,359 | DEBUG | Thread-5 | ContextLoaderListener
> | .activator.ContextLoaderListener 784 | Asynchronous context creation for
> bundle [ServiceMix :: CXF Service Engine (servicemix-cxf-se)]
> 18:38:29,359 | DEBUG | lixDispatchQueue | servicemix-cxf-se
> | ? ? | BundleEvent STARTED
> 18:38:29,359 | DEBUG | xtenderThread-49 | WaiterApplicationContextExecutor
> | WaiterApplicationContextExecutor 162 | Starting first stage of refresh
> for OsgiBundleXmlApplicationContext(bundle=servicemix-cxf-se,
> config=osgibundle:/META-INF/spring/*.xml)
> 18:38:29,359 | DEBUG | xtenderThread-49 | WaiterApplicationContextExecutor
> | WaiterApplicationContextExecutor 202 | Calling preRefresh on
> ....
>
> Sometimes rather than just not being resumed once the dependencies are
> installed, sometimes the SA is installed correctly but later is says it's
> undeploying it.
>
> About 9000 more lines down in the log, it finishes up installing another
> one of my SAs. This one starts deploying after the cxf-se, which may be the
> reason it actually goes through with installing it. I don't see any errors
> installing any of the SUs in the SA, and it appears that it finishes
> installing it but I don't see it listed in the console with jbi/list. Other
> SAs of mine install but don't show up in the console either.
>
> This is the last bit of the log where it creates the service for my SA
> after processing all of the SU's. After that I don't see anything about
> that SA anymore, or anything saying it's started...
>
> 18:39:00,343 | INFO | pool-1-thread-1 | ReflectionServiceFactoryBean
> | .support.JaxWsServiceFactoryBean 163 | Creating Service {
> http://service.com/salina/workflow}workflow-service<http://service.com/salina/workflow%7Dworkflow-service>from class com.service.WorkflowServicePort
> 18:39:00,343 | DEBUG | xtenderThread-64 | SaxonComponent
> | icemix.common.AsyncBaseLifeCycle 296 | Starting component
> 18:39:00,375 | DEBUG | xtenderThread-64 | SaxonComponent
> | icemix.common.AsyncBaseLifeCycle 304 | Component started
>
> It just seems like SAs just randomly die or never get installed. If I stop
> servicemix, delete the data directory, restart, then sometimes everything
> will fully install... I don't see any errors or any reason that those SAs
> don't install and like I said it's not consistent on what it installs or
> doesn't install. Sometimes it doesn't install all the Servicemix components
> either.
>
>
> On Tue, Sep 1, 2009 at 12:42 AM, Gert Vanthienen <
> gert.vanthienen@gmail.com> wrote:
>
>> Ryan,
>>
>> I guess you already enabled DEBUG logging for this? What does
>> osgi/list say about the bundles/SA/XML files you are using? Does it
>> mark the bundles as Active -- if not, something might have gone wrong
>> at file install time?
>>
>> Regards,
>>
>> Gert Vanthienen
>> ------------------------
>> Open Source SOA: http://fusesource.com
>> Blog: http://gertvanthienen.blogspot.com/
>>
>>
>>
>> 2009/9/1 Ryan Moquin <fr...@gmail.com>:
>> > I wanted to reask about SAs in Servicemix 4.0. When I start Servicemix
>> 4.0
>> > fresh (no data directory), certain SAs that it deploys, will end up in
>> the
>> > stopped state. There are no errors, and it seems to be random. If I
>> stop
>> > Servicemix, delete the data directory and start it back up, then
>> different
>> > SAs will be stopped, or sometimes they'll all start up and be fine. It
>> > appears that if they do all start up, then they seem to start everytime
>> if I
>> > restart the server. It just seems like on a fresh start, sometimes an
>> SA
>> > won't start up properly. Is there anyway to find out the reason why an
>> SA
>> > doesn't start? I don't see anything in the log, is there something I
>> can
>> > turn on to see what might be causing the intermittent problem?
>> >
>> > If I try to access an external endpoint on one of the SAs that don't
>> show up
>> > in jbi/list (such as a CXF endpoint), I'll see this error (it's the only
>> > error I see).
>> >
>> > Interceptor has thrown exception, unwinding now
>> > org.apache.cxf.interceptor.Fault: Endpoint is stopped
>> > at
>> >
>> org.apache.servicemix.cxfbc.CxfBcConsumer$1.handleMessage(CxfBcConsumer.java:440)
>> > at
>> >
>> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:236)
>> > at
>> >
>> org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:89)
>> > at
>> >
>> org.apache.servicemix.cxfbc.CxfBcConsumer$JbiChainInitiationObserver.onMessage(CxfBcConsumer.java:646)
>> > at
>> >
>> org.apache.cxf.transport.http_jetty.JettyHTTPDestination.serviceRequest(JettyHTTPDestination.java:302)
>> > at
>> >
>> org.apache.cxf.transport.http_jetty.JettyHTTPDestination.doService(JettyHTTPDestination.java:266)
>> > at
>> >
>> org.apache.cxf.transport.http_jetty.JettyHTTPHandler.handle(JettyHTTPHandler.java:70)
>> > at
>> > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>> > at
>> >
>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>> > at org.mortbay.jetty.handler.HandlerList.handle(HandlerList.java:49)
>> > at
>> > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>> > at org.mortbay.jetty.Server.handle(Server.java:324)
>> > at
>> > org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
>> > at
>> >
>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
>> > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
>> > at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
>> > at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
>> > at
>> >
>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
>> > at
>> >
>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
>> > Caused by: java.lang.Exception: Endpoint is stopped
>> > ... 19 more
>> >
>>
>
>
Re: Servicemix 4.0 endpoints inconsistently shutting down.
Posted by Ryan Moquin <fr...@gmail.com>.
I didn't set it to debug, since I figured if there was an error it would
tell me but like I said, it non-deterministically will sometimes only start
4 total components and no SA's or it will start all of them... or it will
start any number in between. If I delete the data directory and restart,
then a different number of components/SAs will start without me modifying
anything other than deleting the data directory.
The worst case I've seen so far, was 4 of my components all started and non
of the servicemix components started. I was getting all kinds of weird
activemq errors about not being able to connect to 0.0.0. I should have
saved some of the log. I was getting weird JAXB errors as well.. basically
everything went nuts. I then deleted the data directory and everything
started up without any issues.
Anyhow, I turned on debug, deleted the data directory and started up. 3 of
my SA's didn't start, but all of my components and servicemix components
started. Now, I don't see any errors, this is a snippet from the log
regarding one of my SAs which rely on servicemix-cxf-se (which hadn't been
installed as of this point):
18:38:21,828 | DEBUG | pool-1-thread-1 | BundleWatcher |
.swissbox.extender.BundleWatcher 176 | Scanning bundle [process-service-sa]
18:38:21,828 | DEBUG | pool-1-thread-1 | Deployer |
cemix.jbi.deployer.impl.Deployer 301 | Checking bundle: 'null
(process-service-sa)'
18:38:21,859 | INFO | pool-1-thread-1 | Deployer |
cemix.jbi.deployer.impl.Deployer 343 | Deploying bundle 'null
(process-service-sa)' as a JBI service assembly
18:38:21,906 | WARN | pool-1-thread-1 | Deployer |
cemix.jbi.deployer.impl.Deployer 359 | Requirements not met for JBI
artifact in bundle null (process-service-sa). Installation pending.
org.apache.servicemix.jbi.deployer.impl.PendingException: Component not
installed: servicemix-cxf-se
18:38:21,921 | DEBUG | pool-1-thread-1 | ContextLoaderListener |
.activator.ContextLoaderListener 709 | Scanning bundle [null
(process-service-sa)] for configurations...
18:38:21,921 | DEBUG | pool-1-thread-1 | ContextLoaderListener |
.activator.ContextLoaderListener 715 | Creating an application context for
bundle [null (process-service-sa)]
18:38:21,921 | DEBUG | pool-1-thread-1 | ContextLoaderListener |
.activator.ContextLoaderListener 726 | No application context created for
bundle [null (process-service-sa)]
18:38:21,921 | INFO | pool-1-thread-1 | FileMonitor |
x.kernel.filemonitor.FileMonitor 550 | Started: process-service-sa [119]
18:38:21,921 | DEBUG | lixDispatchQueue |
process-service-sa | ? ?
| BundleEvent STARTED
After this point in the log, there isn't a single mention of that SA again
anywhere in the rest of the log.... About 4000 lines down from the above
segment in the log file, the servicemix-cxf-se component bundle gets
deployed, but it never finishes installing process-service-sa.
18:38:29,343 | DEBUG | Thread-5 | BundleWatcher |
.swissbox.extender.BundleWatcher 176 | Scanning bundle [servicemix-cxf-se]
18:38:29,359 | DEBUG | Thread-5 | Deployer |
cemix.jbi.deployer.impl.Deployer 301 | Checking bundle: 'ServiceMix :: CXF
Service Engine (servicemix-cxf-se)'
18:38:29,359 | DEBUG | Thread-5 | Deployer |
cemix.jbi.deployer.impl.Deployer 304 | Bundle 'ServiceMix :: CXF Service
Engine (servicemix-cxf-se)' does not contain any JBI descriptor.
18:38:29,359 | DEBUG | Thread-5 | ContextLoaderListener |
.activator.ContextLoaderListener 709 | Scanning bundle [ServiceMix :: CXF
Service Engine (servicemix-cxf-se)] for configurations...
18:38:29,359 | DEBUG | Thread-5 | ContextLoaderListener |
.activator.ContextLoaderListener 715 | Creating an application context for
bundle [ServiceMix :: CXF Service Engine (servicemix-cxf-se)]
18:38:29,359 | INFO | Thread-5 | ultOsgiApplicationContextCreator |
ultOsgiApplicationContextCreator 67 | Discovered configurations
{osgibundle:/META-INF/spring/*.xml} in bundle [ServiceMix :: CXF Service
Engine (servicemix-cxf-se)]
18:38:29,359 | DEBUG | Thread-5 | ContextLoaderListener |
.activator.ContextLoaderListener 735 | Bundle [ServiceMix :: CXF Service
Engine (servicemix-cxf-se)] is Spring type compatible with Spring-DM
18:38:29,359 | DEBUG | Thread-5 | ContextLoaderListener |
.activator.ContextLoaderListener 784 | Asynchronous context creation for
bundle [ServiceMix :: CXF Service Engine (servicemix-cxf-se)]
18:38:29,359 | DEBUG | lixDispatchQueue | servicemix-cxf-se |
? ? | BundleEvent STARTED
18:38:29,359 | DEBUG | xtenderThread-49 | WaiterApplicationContextExecutor |
WaiterApplicationContextExecutor 162 | Starting first stage of refresh for
OsgiBundleXmlApplicationContext(bundle=servicemix-cxf-se,
config=osgibundle:/META-INF/spring/*.xml)
18:38:29,359 | DEBUG | xtenderThread-49 | WaiterApplicationContextExecutor |
WaiterApplicationContextExecutor 202 | Calling preRefresh on
....
Sometimes rather than just not being resumed once the dependencies are
installed, sometimes the SA is installed correctly but later is says it's
undeploying it.
About 9000 more lines down in the log, it finishes up installing another one
of my SAs. This one starts deploying after the cxf-se, which may be the
reason it actually goes through with installing it. I don't see any errors
installing any of the SUs in the SA, and it appears that it finishes
installing it but I don't see it listed in the console with jbi/list. Other
SAs of mine install but don't show up in the console either.
This is the last bit of the log where it creates the service for my SA after
processing all of the SU's. After that I don't see anything about that SA
anymore, or anything saying it's started...
18:39:00,343 | INFO | pool-1-thread-1 | ReflectionServiceFactoryBean |
.support.JaxWsServiceFactoryBean 163 | Creating Service {
http://service.com/salina/workflow}workflow-service from class
com.service.WorkflowServicePort
18:39:00,343 | DEBUG | xtenderThread-64 | SaxonComponent |
icemix.common.AsyncBaseLifeCycle 296 | Starting component
18:39:00,375 | DEBUG | xtenderThread-64 | SaxonComponent |
icemix.common.AsyncBaseLifeCycle 304 | Component started
It just seems like SAs just randomly die or never get installed. If I stop
servicemix, delete the data directory, restart, then sometimes everything
will fully install... I don't see any errors or any reason that those SAs
don't install and like I said it's not consistent on what it installs or
doesn't install. Sometimes it doesn't install all the Servicemix components
either.
On Tue, Sep 1, 2009 at 12:42 AM, Gert Vanthienen
<ge...@gmail.com>wrote:
> Ryan,
>
> I guess you already enabled DEBUG logging for this? What does
> osgi/list say about the bundles/SA/XML files you are using? Does it
> mark the bundles as Active -- if not, something might have gone wrong
> at file install time?
>
> Regards,
>
> Gert Vanthienen
> ------------------------
> Open Source SOA: http://fusesource.com
> Blog: http://gertvanthienen.blogspot.com/
>
>
>
> 2009/9/1 Ryan Moquin <fr...@gmail.com>:
> > I wanted to reask about SAs in Servicemix 4.0. When I start Servicemix
> 4.0
> > fresh (no data directory), certain SAs that it deploys, will end up in
> the
> > stopped state. There are no errors, and it seems to be random. If I
> stop
> > Servicemix, delete the data directory and start it back up, then
> different
> > SAs will be stopped, or sometimes they'll all start up and be fine. It
> > appears that if they do all start up, then they seem to start everytime
> if I
> > restart the server. It just seems like on a fresh start, sometimes an SA
> > won't start up properly. Is there anyway to find out the reason why an
> SA
> > doesn't start? I don't see anything in the log, is there something I can
> > turn on to see what might be causing the intermittent problem?
> >
> > If I try to access an external endpoint on one of the SAs that don't show
> up
> > in jbi/list (such as a CXF endpoint), I'll see this error (it's the only
> > error I see).
> >
> > Interceptor has thrown exception, unwinding now
> > org.apache.cxf.interceptor.Fault: Endpoint is stopped
> > at
> >
> org.apache.servicemix.cxfbc.CxfBcConsumer$1.handleMessage(CxfBcConsumer.java:440)
> > at
> >
> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:236)
> > at
> >
> org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:89)
> > at
> >
> org.apache.servicemix.cxfbc.CxfBcConsumer$JbiChainInitiationObserver.onMessage(CxfBcConsumer.java:646)
> > at
> >
> org.apache.cxf.transport.http_jetty.JettyHTTPDestination.serviceRequest(JettyHTTPDestination.java:302)
> > at
> >
> org.apache.cxf.transport.http_jetty.JettyHTTPDestination.doService(JettyHTTPDestination.java:266)
> > at
> >
> org.apache.cxf.transport.http_jetty.JettyHTTPHandler.handle(JettyHTTPHandler.java:70)
> > at
> > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> > at
> >
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> > at org.mortbay.jetty.handler.HandlerList.handle(HandlerList.java:49)
> > at
> > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> > at org.mortbay.jetty.Server.handle(Server.java:324)
> > at
> > org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
> > at
> >
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
> > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
> > at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
> > at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
> > at
> >
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
> > at
> >
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
> > Caused by: java.lang.Exception: Endpoint is stopped
> > ... 19 more
> >
>
Re: Servicemix 4.0 endpoints inconsistently shutting down.
Posted by Gert Vanthienen <ge...@gmail.com>.
Ryan,
I guess you already enabled DEBUG logging for this? What does
osgi/list say about the bundles/SA/XML files you are using? Does it
mark the bundles as Active -- if not, something might have gone wrong
at file install time?
Regards,
Gert Vanthienen
------------------------
Open Source SOA: http://fusesource.com
Blog: http://gertvanthienen.blogspot.com/
2009/9/1 Ryan Moquin <fr...@gmail.com>:
> I wanted to reask about SAs in Servicemix 4.0. When I start Servicemix 4.0
> fresh (no data directory), certain SAs that it deploys, will end up in the
> stopped state. There are no errors, and it seems to be random. If I stop
> Servicemix, delete the data directory and start it back up, then different
> SAs will be stopped, or sometimes they'll all start up and be fine. It
> appears that if they do all start up, then they seem to start everytime if I
> restart the server. It just seems like on a fresh start, sometimes an SA
> won't start up properly. Is there anyway to find out the reason why an SA
> doesn't start? I don't see anything in the log, is there something I can
> turn on to see what might be causing the intermittent problem?
>
> If I try to access an external endpoint on one of the SAs that don't show up
> in jbi/list (such as a CXF endpoint), I'll see this error (it's the only
> error I see).
>
> Interceptor has thrown exception, unwinding now
> org.apache.cxf.interceptor.Fault: Endpoint is stopped
> at
> org.apache.servicemix.cxfbc.CxfBcConsumer$1.handleMessage(CxfBcConsumer.java:440)
> at
> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:236)
> at
> org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:89)
> at
> org.apache.servicemix.cxfbc.CxfBcConsumer$JbiChainInitiationObserver.onMessage(CxfBcConsumer.java:646)
> at
> org.apache.cxf.transport.http_jetty.JettyHTTPDestination.serviceRequest(JettyHTTPDestination.java:302)
> at
> org.apache.cxf.transport.http_jetty.JettyHTTPDestination.doService(JettyHTTPDestination.java:266)
> at
> org.apache.cxf.transport.http_jetty.JettyHTTPHandler.handle(JettyHTTPHandler.java:70)
> at
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at org.mortbay.jetty.handler.HandlerList.handle(HandlerList.java:49)
> at
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:324)
> at
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
> at
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
> at
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
> at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
> Caused by: java.lang.Exception: Endpoint is stopped
> ... 19 more
>