You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@aurora.apache.org by Mauricio Garavaglia <ma...@gmail.com> on 2015/09/24 06:32:57 UTC
TransientStorageException: Storage is not READY
Hi guys, I have this problem when suddenly one scheduler stops working,
every request ended up in the following stack trace (since this was not the
leader it also fails to redirect the request to the proper host).
I have 5 schedulers configured with 5zk; this error goes away if I restart
the scheduler; but I couldn't find yet how to reproduce the issue
consistently. Please let me know if I could give you more info to
troubleshoot. Thanks
I0924 04:01:46.763665 244 leveldb.cpp:343] Persisting action (86 bytes)
to leveldb took 3.839569ms
I0924 04:01:46.763694 244 replica.cpp:679] Persisted action at 3413
I0924 04:01:46.765290 217 replica.cpp:658] Replica received learned
notice for position 3413
I0924 04:01:46.768878 217 leveldb.cpp:343] Persisting action (88 bytes)
to leveldb took 3.558083ms
I0924 04:01:46.768910 217 replica.cpp:679] Persisted action at 3413
I0924 04:01:46.768925 217 replica.cpp:664] Replica learned APPEND action
at position 3413
I0924 04:01:46.853 THREAD133
com.twitter.common.zookeeper.ServerSetImpl$ServerSetWatcher.logChange:
server set /aurora/scheduler change: from 0 members to 1
joined:
ServiceInstance(serviceEndpoint:Endpoint(host:10.192.255.21, port:8081),
additionalEndpoints:{http=Endpoint(host:10.192.255.21, port:8081)},
status:ALIVE)
I0924 04:01:46.853 THREAD133
org.apache.aurora.scheduler.http.LeaderRedirect$SchedulerMonitor.onChange:
Found leader scheduler at
[ServiceInstance(serviceEndpoint:Endpoint(host:10.192.255.21, port:8081),
additionalEndpoints:{http=Endpoint(host:10.192.255.21, port:8081)},
status:ALIVE)]
I0924 04:01:57.685 THREAD133
com.twitter.common.zookeeper.CandidateImpl$4.onGroupChange: Candidate
/aurora/scheduler/singleton_candidate_0000000196 waiting for the next
leader election, current voting: [singleton_candidate_0000000198,
singleton_candidate_0000000200, singleton_candidate_0000000193,
singleton_candidate_0000000194, singleton_candidate_0000000196]
I0924 04:01:57.736865 261 network.hpp:424] ZooKeeper group memberships
changed
I0924 04:01:57.737118 234 group.cpp:659] Trying to get
'/aurora/replicated-log/0000000193' in ZooKeeper
I0924 04:01:57.737802 234 group.cpp:659] Trying to get
'/aurora/replicated-log/0000000194' in ZooKeeper
I0924 04:01:57.738327 234 group.cpp:659] Trying to get
'/aurora/replicated-log/0000000195' in ZooKeeper
I0924 04:01:57.738834 234 group.cpp:659] Trying to get
'/aurora/replicated-log/0000000196' in ZooKeeper
I0924 04:01:57.739236 234 group.cpp:659] Trying to get
'/aurora/replicated-log/0000000197' in ZooKeeper
I0924 04:01:57.739910 241 network.hpp:466] ZooKeeper group PIDs: {
log-replica(1)@10.192.255.21:8083, log-replica(1)@10.192.255.22:8083,
log-replica(1)@10.192.255.23:8083, log-replica(1)@10.192.255.24:8083,
log-replica(1)@10.192.255.25:8083 }
I0924 04:02:59.327913 219 replica.cpp:511] Replica received write request
for position 3414
I0924 04:02:59.333317 219 leveldb.cpp:343] Persisting action (217 bytes)
to leveldb took 5.351646ms
I0924 04:02:59.333377 219 replica.cpp:679] Persisted action at 3414
I0924 04:02:59.334916 241 replica.cpp:658] Replica received learned
notice for position 3414
I0924 04:02:59.338354 241 leveldb.cpp:343] Persisting action (219 bytes)
to leveldb took 3.403633ms
I0924 04:02:59.338392 241 replica.cpp:679] Persisted action at 3414
I0924 04:02:59.338404 241 replica.cpp:664] Replica learned APPEND action
at position 3414
I0924 04:03:24.492 THREAD171
org.apache.aurora.scheduler.http.RequestLogger$1.log: 10.192.255.23
10.207.4.10 [24/Sep/2015:04:03:24 +0000] "POST /api HTTP/1.1" 200 0 "
http://10.192.255.23:8081/scheduler" "Mozilla/5.0 (Macintosh; Intel Mac OS
X 10.10; rv:40.0) Gecko/20100101 Firefox/40.0" 1
I0924 04:03:24.505 THREAD171
org.apache.aurora.scheduler.thrift.aop.LoggingInterceptor.invoke:
getRoleSummary()
W0924 04:03:24.512 THREAD171
org.apache.aurora.scheduler.thrift.aop.LoggingInterceptor.invoke: Uncaught
transient exception while handling getRoleSummary()
org.apache.aurora.scheduler.storage.Storage$TransientStorageException:
Storage is not READY
at
org.apache.aurora.scheduler.storage.CallOrderEnforcingStorage.checkInState(CallOrderEnforcingStorage.java:78)
at
org.apache.aurora.scheduler.storage.CallOrderEnforcingStorage.read(CallOrderEnforcingStorage.java:114)
at
org.apache.aurora.scheduler.thrift.ReadOnlySchedulerImpl.getRoleSummary(ReadOnlySchedulerImpl.java:230)
at
org.apache.aurora.scheduler.thrift.SchedulerThriftInterface.getRoleSummary(SchedulerThriftInterface.java:463)
at
org.apache.aurora.scheduler.thrift.aop.ThriftStatsExporterInterceptor.invoke(ThriftStatsExporterInterceptor.java:47)
at
org.apache.aurora.scheduler.thrift.aop.FeatureToggleInterceptor.invoke(FeatureToggleInterceptor.java:38)
at
org.apache.aurora.scheduler.thrift.aop.LoggingInterceptor.invoke(LoggingInterceptor.java:102)
at
org.apache.aurora.scheduler.thrift.aop.ServerInfoInterceptor.invoke(ServerInfoInterceptor.java:30)
at
org.apache.aurora.gen.ReadOnlyScheduler$Processor$getRoleSummary.getResult(ReadOnlyScheduler.java:886)
at
org.apache.aurora.gen.ReadOnlyScheduler$Processor$getRoleSummary.getResult(ReadOnlyScheduler.java:871)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.thrift.server.TServlet.doPost(TServlet.java:83)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
at
com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
at
com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at
org.apache.aurora.scheduler.http.HttpStatsFilter.doFilter(HttpStatsFilter.java:69)
at
org.apache.aurora.scheduler.http.AbstractFilter.doFilter(AbstractFilter.java:44)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at
org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82)
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:294)
at
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
at
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at
com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1288)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:443)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1044)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:372)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:978)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:317)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:369)
at
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:486)
at
org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:944)
at
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1005)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
at
org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
at
org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667)
at
org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)
Re: TransientStorageException: Storage is not READY
Posted by Mauricio Garavaglia <ma...@gmail.com>.
Great, thanks for the answer!
On Wed, Sep 30, 2015 at 10:05 PM, Kevin Sweeney <ke...@apache.org> wrote:
> Hi Mauricio,
>
> Sorry for the delay in reply.
>
> This can sometimes happen during startup or shutdown. It's a race between
> the leading scheuler (de)registering itself in ZooKeeper (and the client or
> UI noticing) and starting up (tearing down) its storage. It's usually
> transient, but definitely a wart. I've filed
> https://issues.apache.org/jira/browse/AURORA-1509 to track improving this.
>
> On Wed, Sep 23, 2015 at 9:33 PM Mauricio Garavaglia <
> mauriciogaravaglia@gmail.com> wrote:
>
> > Hi guys, I have this problem when suddenly one scheduler stops working,
> > every request ended up in the following stack trace (since this was not
> the
> > leader it also fails to redirect the request to the proper host).
> >
> > I have 5 schedulers configured with 5zk; this error goes away if I
> restart
> > the scheduler; but I couldn't find yet how to reproduce the issue
> > consistently. Please let me know if I could give you more info to
> > troubleshoot. Thanks
> >
> >
> > I0924 04:01:46.763665 244 leveldb.cpp:343] Persisting action (86 bytes)
> > to leveldb took 3.839569ms
> > I0924 04:01:46.763694 244 replica.cpp:679] Persisted action at 3413
> > I0924 04:01:46.765290 217 replica.cpp:658] Replica received learned
> > notice for position 3413
> > I0924 04:01:46.768878 217 leveldb.cpp:343] Persisting action (88 bytes)
> > to leveldb took 3.558083ms
> > I0924 04:01:46.768910 217 replica.cpp:679] Persisted action at 3413
> > I0924 04:01:46.768925 217 replica.cpp:664] Replica learned APPEND
> action
> > at position 3413
> > I0924 04:01:46.853 THREAD133
> > com.twitter.common.zookeeper.ServerSetImpl$ServerSetWatcher.logChange:
> > server set /aurora/scheduler change: from 0 members to 1
> > joined:
> > ServiceInstance(serviceEndpoint:Endpoint(host:10.192.255.21, port:8081),
> > additionalEndpoints:{http=Endpoint(host:10.192.255.21, port:8081)},
> > status:ALIVE)
> > I0924 04:01:46.853 THREAD133
> >
> org.apache.aurora.scheduler.http.LeaderRedirect$SchedulerMonitor.onChange:
> > Found leader scheduler at
> > [ServiceInstance(serviceEndpoint:Endpoint(host:10.192.255.21, port:8081),
> > additionalEndpoints:{http=Endpoint(host:10.192.255.21, port:8081)},
> > status:ALIVE)]
> > I0924 04:01:57.685 THREAD133
> > com.twitter.common.zookeeper.CandidateImpl$4.onGroupChange: Candidate
> > /aurora/scheduler/singleton_candidate_0000000196 waiting for the next
> > leader election, current voting: [singleton_candidate_0000000198,
> > singleton_candidate_0000000200, singleton_candidate_0000000193,
> > singleton_candidate_0000000194, singleton_candidate_0000000196]
> > I0924 04:01:57.736865 261 network.hpp:424] ZooKeeper group memberships
> > changed
> > I0924 04:01:57.737118 234 group.cpp:659] Trying to get
> > '/aurora/replicated-log/0000000193' in ZooKeeper
> > I0924 04:01:57.737802 234 group.cpp:659] Trying to get
> > '/aurora/replicated-log/0000000194' in ZooKeeper
> > I0924 04:01:57.738327 234 group.cpp:659] Trying to get
> > '/aurora/replicated-log/0000000195' in ZooKeeper
> > I0924 04:01:57.738834 234 group.cpp:659] Trying to get
> > '/aurora/replicated-log/0000000196' in ZooKeeper
> > I0924 04:01:57.739236 234 group.cpp:659] Trying to get
> > '/aurora/replicated-log/0000000197' in ZooKeeper
> > I0924 04:01:57.739910 241 network.hpp:466] ZooKeeper group PIDs: {
> > log-replica(1)@10.192.255.21:8083, log-replica(1)@10.192.255.22:8083,
> > log-replica(1)@10.192.255.23:8083, log-replica(1)@10.192.255.24:8083,
> > log-replica(1)@10.192.255.25:8083 }
> > I0924 04:02:59.327913 219 replica.cpp:511] Replica received write
> request
> > for position 3414
> > I0924 04:02:59.333317 219 leveldb.cpp:343] Persisting action (217
> bytes)
> > to leveldb took 5.351646ms
> > I0924 04:02:59.333377 219 replica.cpp:679] Persisted action at 3414
> > I0924 04:02:59.334916 241 replica.cpp:658] Replica received learned
> > notice for position 3414
> > I0924 04:02:59.338354 241 leveldb.cpp:343] Persisting action (219
> bytes)
> > to leveldb took 3.403633ms
> > I0924 04:02:59.338392 241 replica.cpp:679] Persisted action at 3414
> > I0924 04:02:59.338404 241 replica.cpp:664] Replica learned APPEND
> action
> > at position 3414
> > I0924 04:03:24.492 THREAD171
> > org.apache.aurora.scheduler.http.RequestLogger$1.log: 10.192.255.23
> > 10.207.4.10 [24/Sep/2015:04:03:24 +0000] "POST /api HTTP/1.1" 200 0 "
> > http://10.192.255.23:8081/scheduler" "Mozilla/5.0 (Macintosh; Intel Mac
> OS
> > X 10.10; rv:40.0) Gecko/20100101 Firefox/40.0" 1
> > I0924 04:03:24.505 THREAD171
> > org.apache.aurora.scheduler.thrift.aop.LoggingInterceptor.invoke:
> > getRoleSummary()
> > W0924 04:03:24.512 THREAD171
> > org.apache.aurora.scheduler.thrift.aop.LoggingInterceptor.invoke:
> Uncaught
> > transient exception while handling getRoleSummary()
> > org.apache.aurora.scheduler.storage.Storage$TransientStorageException:
> > Storage is not READY
> > at
> >
> >
> org.apache.aurora.scheduler.storage.CallOrderEnforcingStorage.checkInState(CallOrderEnforcingStorage.java:78)
> > at
> >
> >
> org.apache.aurora.scheduler.storage.CallOrderEnforcingStorage.read(CallOrderEnforcingStorage.java:114)
> > at
> >
> >
> org.apache.aurora.scheduler.thrift.ReadOnlySchedulerImpl.getRoleSummary(ReadOnlySchedulerImpl.java:230)
> > at
> >
> >
> org.apache.aurora.scheduler.thrift.SchedulerThriftInterface.getRoleSummary(SchedulerThriftInterface.java:463)
> > at
> >
> >
> org.apache.aurora.scheduler.thrift.aop.ThriftStatsExporterInterceptor.invoke(ThriftStatsExporterInterceptor.java:47)
> > at
> >
> >
> org.apache.aurora.scheduler.thrift.aop.FeatureToggleInterceptor.invoke(FeatureToggleInterceptor.java:38)
> > at
> >
> >
> org.apache.aurora.scheduler.thrift.aop.LoggingInterceptor.invoke(LoggingInterceptor.java:102)
> > at
> >
> >
> org.apache.aurora.scheduler.thrift.aop.ServerInfoInterceptor.invoke(ServerInfoInterceptor.java:30)
> > at
> >
> >
> org.apache.aurora.gen.ReadOnlyScheduler$Processor$getRoleSummary.getResult(ReadOnlyScheduler.java:886)
> > at
> >
> >
> org.apache.aurora.gen.ReadOnlyScheduler$Processor$getRoleSummary.getResult(ReadOnlyScheduler.java:871)
> > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> > at org.apache.thrift.server.TServlet.doPost(TServlet.java:83)
> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> > at
> >
> >
> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
> > at
> >
> >
> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
> > at
> >
> >
> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
> > at
> >
> >
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
> > at
> >
> >
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> > at
> >
> >
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> > at
> >
> >
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> > at
> >
> >
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> > at
> >
> >
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> > at
> >
> >
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> > at
> >
> >
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> > at
> >
> >
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> > at
> >
> >
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> > at
> >
> >
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> > at
> >
> >
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> > at
> >
> >
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> > at
> >
> >
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> > at
> >
> >
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> > at
> >
> >
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> > at
> >
> >
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> > at
> >
> >
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> > at
> >
> >
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> > at
> >
> >
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> > at
> >
> >
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> > at
> >
> >
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> > at
> >
> >
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> > at
> >
> >
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> > at
> >
> >
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> > at
> >
> >
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> > at
> >
> >
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> > at
> >
> >
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> > at
> >
> >
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> > at
> >
> >
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> > at
> >
> >
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> > at
> >
> >
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> > at
> >
> >
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> > at
> >
> >
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> > at
> >
> >
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> > at
> >
> >
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> > at
> >
> >
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> > at
> >
> >
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> > at
> >
> >
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> > at
> >
> >
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> > at
> >
> >
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> > at
> >
> >
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> > at
> >
> >
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> > at
> >
> >
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> > at
> >
> >
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> > at
> >
> >
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> > at
> >
> >
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> > at
> >
> >
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> > at
> >
> >
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> > at
> >
> >
> org.apache.aurora.scheduler.http.HttpStatsFilter.doFilter(HttpStatsFilter.java:69)
> > at
> >
> >
> org.apache.aurora.scheduler.http.AbstractFilter.doFilter(AbstractFilter.java:44)
> > at
> >
> >
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
> > at
> >
> >
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> > at
> >
> >
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> > at
> >
> >
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> > at
> >
> >
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> > at
> >
> >
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> > at
> >
> >
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> > at
> >
> >
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> > at
> >
> >
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> > at
> >
> >
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> > at
> >
> >
> org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82)
> > at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:294)
> > at
> >
> >
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
> > at
> >
> >
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> > at
> >
> >
> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
> > at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
> > at
> >
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1288)
> > at
> >
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:443)
> > at
> >
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1044)
> > at
> > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:372)
> > at
> >
> >
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:978)
> > at
> >
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
> > at
> >
> >
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
> > at
> >
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
> > at
> >
> >
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:317)
> > at
> >
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
> > at org.eclipse.jetty.server.Server.handle(Server.java:369)
> > at
> >
> >
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:486)
> > at
> >
> >
> org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:944)
> > at
> >
> >
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1005)
> > at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)
> > at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
> > at
> >
> >
> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
> > at
> >
> >
> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667)
> > at
> >
> >
> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
> > at
> >
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> > at
> >
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> > at java.lang.Thread.run(Thread.java:745)
> >
>
Re: TransientStorageException: Storage is not READY
Posted by Kevin Sweeney <ke...@apache.org>.
Hi Mauricio,
Sorry for the delay in reply.
This can sometimes happen during startup or shutdown. It's a race between
the leading scheuler (de)registering itself in ZooKeeper (and the client or
UI noticing) and starting up (tearing down) its storage. It's usually
transient, but definitely a wart. I've filed
https://issues.apache.org/jira/browse/AURORA-1509 to track improving this.
On Wed, Sep 23, 2015 at 9:33 PM Mauricio Garavaglia <
mauriciogaravaglia@gmail.com> wrote:
> Hi guys, I have this problem when suddenly one scheduler stops working,
> every request ended up in the following stack trace (since this was not the
> leader it also fails to redirect the request to the proper host).
>
> I have 5 schedulers configured with 5zk; this error goes away if I restart
> the scheduler; but I couldn't find yet how to reproduce the issue
> consistently. Please let me know if I could give you more info to
> troubleshoot. Thanks
>
>
> I0924 04:01:46.763665 244 leveldb.cpp:343] Persisting action (86 bytes)
> to leveldb took 3.839569ms
> I0924 04:01:46.763694 244 replica.cpp:679] Persisted action at 3413
> I0924 04:01:46.765290 217 replica.cpp:658] Replica received learned
> notice for position 3413
> I0924 04:01:46.768878 217 leveldb.cpp:343] Persisting action (88 bytes)
> to leveldb took 3.558083ms
> I0924 04:01:46.768910 217 replica.cpp:679] Persisted action at 3413
> I0924 04:01:46.768925 217 replica.cpp:664] Replica learned APPEND action
> at position 3413
> I0924 04:01:46.853 THREAD133
> com.twitter.common.zookeeper.ServerSetImpl$ServerSetWatcher.logChange:
> server set /aurora/scheduler change: from 0 members to 1
> joined:
> ServiceInstance(serviceEndpoint:Endpoint(host:10.192.255.21, port:8081),
> additionalEndpoints:{http=Endpoint(host:10.192.255.21, port:8081)},
> status:ALIVE)
> I0924 04:01:46.853 THREAD133
> org.apache.aurora.scheduler.http.LeaderRedirect$SchedulerMonitor.onChange:
> Found leader scheduler at
> [ServiceInstance(serviceEndpoint:Endpoint(host:10.192.255.21, port:8081),
> additionalEndpoints:{http=Endpoint(host:10.192.255.21, port:8081)},
> status:ALIVE)]
> I0924 04:01:57.685 THREAD133
> com.twitter.common.zookeeper.CandidateImpl$4.onGroupChange: Candidate
> /aurora/scheduler/singleton_candidate_0000000196 waiting for the next
> leader election, current voting: [singleton_candidate_0000000198,
> singleton_candidate_0000000200, singleton_candidate_0000000193,
> singleton_candidate_0000000194, singleton_candidate_0000000196]
> I0924 04:01:57.736865 261 network.hpp:424] ZooKeeper group memberships
> changed
> I0924 04:01:57.737118 234 group.cpp:659] Trying to get
> '/aurora/replicated-log/0000000193' in ZooKeeper
> I0924 04:01:57.737802 234 group.cpp:659] Trying to get
> '/aurora/replicated-log/0000000194' in ZooKeeper
> I0924 04:01:57.738327 234 group.cpp:659] Trying to get
> '/aurora/replicated-log/0000000195' in ZooKeeper
> I0924 04:01:57.738834 234 group.cpp:659] Trying to get
> '/aurora/replicated-log/0000000196' in ZooKeeper
> I0924 04:01:57.739236 234 group.cpp:659] Trying to get
> '/aurora/replicated-log/0000000197' in ZooKeeper
> I0924 04:01:57.739910 241 network.hpp:466] ZooKeeper group PIDs: {
> log-replica(1)@10.192.255.21:8083, log-replica(1)@10.192.255.22:8083,
> log-replica(1)@10.192.255.23:8083, log-replica(1)@10.192.255.24:8083,
> log-replica(1)@10.192.255.25:8083 }
> I0924 04:02:59.327913 219 replica.cpp:511] Replica received write request
> for position 3414
> I0924 04:02:59.333317 219 leveldb.cpp:343] Persisting action (217 bytes)
> to leveldb took 5.351646ms
> I0924 04:02:59.333377 219 replica.cpp:679] Persisted action at 3414
> I0924 04:02:59.334916 241 replica.cpp:658] Replica received learned
> notice for position 3414
> I0924 04:02:59.338354 241 leveldb.cpp:343] Persisting action (219 bytes)
> to leveldb took 3.403633ms
> I0924 04:02:59.338392 241 replica.cpp:679] Persisted action at 3414
> I0924 04:02:59.338404 241 replica.cpp:664] Replica learned APPEND action
> at position 3414
> I0924 04:03:24.492 THREAD171
> org.apache.aurora.scheduler.http.RequestLogger$1.log: 10.192.255.23
> 10.207.4.10 [24/Sep/2015:04:03:24 +0000] "POST /api HTTP/1.1" 200 0 "
> http://10.192.255.23:8081/scheduler" "Mozilla/5.0 (Macintosh; Intel Mac OS
> X 10.10; rv:40.0) Gecko/20100101 Firefox/40.0" 1
> I0924 04:03:24.505 THREAD171
> org.apache.aurora.scheduler.thrift.aop.LoggingInterceptor.invoke:
> getRoleSummary()
> W0924 04:03:24.512 THREAD171
> org.apache.aurora.scheduler.thrift.aop.LoggingInterceptor.invoke: Uncaught
> transient exception while handling getRoleSummary()
> org.apache.aurora.scheduler.storage.Storage$TransientStorageException:
> Storage is not READY
> at
>
> org.apache.aurora.scheduler.storage.CallOrderEnforcingStorage.checkInState(CallOrderEnforcingStorage.java:78)
> at
>
> org.apache.aurora.scheduler.storage.CallOrderEnforcingStorage.read(CallOrderEnforcingStorage.java:114)
> at
>
> org.apache.aurora.scheduler.thrift.ReadOnlySchedulerImpl.getRoleSummary(ReadOnlySchedulerImpl.java:230)
> at
>
> org.apache.aurora.scheduler.thrift.SchedulerThriftInterface.getRoleSummary(SchedulerThriftInterface.java:463)
> at
>
> org.apache.aurora.scheduler.thrift.aop.ThriftStatsExporterInterceptor.invoke(ThriftStatsExporterInterceptor.java:47)
> at
>
> org.apache.aurora.scheduler.thrift.aop.FeatureToggleInterceptor.invoke(FeatureToggleInterceptor.java:38)
> at
>
> org.apache.aurora.scheduler.thrift.aop.LoggingInterceptor.invoke(LoggingInterceptor.java:102)
> at
>
> org.apache.aurora.scheduler.thrift.aop.ServerInfoInterceptor.invoke(ServerInfoInterceptor.java:30)
> at
>
> org.apache.aurora.gen.ReadOnlyScheduler$Processor$getRoleSummary.getResult(ReadOnlyScheduler.java:886)
> at
>
> org.apache.aurora.gen.ReadOnlyScheduler$Processor$getRoleSummary.getResult(ReadOnlyScheduler.java:871)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at org.apache.thrift.server.TServlet.doPost(TServlet.java:83)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at
>
> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
> at
>
> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
> at
>
> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
> at
>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
> at
>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> at
>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> at
>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> at
>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> at
>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> at
>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> at
>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> at
>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> at
>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> at
>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> at
>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> at
>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> at
>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> at
>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> at
>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> at
>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> at
>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> at
>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> at
>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> at
>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> at
>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> at
>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> at
>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> at
>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> at
>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> at
>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> at
>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> at
>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> at
>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> at
>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> at
>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> at
>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> at
>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> at
>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> at
>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> at
>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> at
>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> at
>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> at
>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> at
>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> at
>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> at
>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> at
>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> at
>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> at
>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> at
>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> at
>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> at
>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> at
>
> org.apache.aurora.scheduler.http.HttpStatsFilter.doFilter(HttpStatsFilter.java:69)
> at
>
> org.apache.aurora.scheduler.http.AbstractFilter.doFilter(AbstractFilter.java:44)
> at
>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
> at
>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> at
>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> at
>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> at
>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> at
>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> at
>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> at
>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> at
>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:168)
> at
>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> at
>
> org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82)
> at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:294)
> at
>
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
> at
>
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> at
>
> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
> at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
> at
>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1288)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:443)
> at
>
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1044)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:372)
> at
>
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:978)
> at
>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
> at
>
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
> at
>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
> at
>
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:317)
> at
>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
> at org.eclipse.jetty.server.Server.handle(Server.java:369)
> at
>
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:486)
> at
>
> org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:944)
> at
>
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1005)
> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)
> at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
> at
>
> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
> at
>
> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667)
> at
>
> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
> at
>
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at
>
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:745)
>