You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@usergrid.apache.org by "Michael Russo (JIRA)" <ji...@apache.org> on 2016/05/11 06:08:12 UTC
[jira] [Comment Edited] (USERGRID-507) Many Histrix timeouts though
Cassndra is not under load
[ https://issues.apache.org/jira/browse/USERGRID-507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15279616#comment-15279616 ]
Michael Russo edited comment on USERGRID-507 at 5/11/16 6:07 AM:
-----------------------------------------------------------------
This is a non issues in 2.1.0 as there is no hystrix code in the classes mentioned in this stack strace:
at org.apache.usergrid.persistence.core.astyanax.MultiRowColumnIterator.advance(MultiRowColumnIterator.java:187)
at org.apache.usergrid.persistence.core.astyanax.MultiRowColumnIterator.hasNext(MultiRowColumnIterator.java:124)
at org.apache.usergrid.persistence.graph.serialization.impl.shard.impl.ShardsColumnIterator.hasNext(ShardsColumnIterator.java:65)
at org.apache.usergrid.persistence.graph.serialization.impl.shard.impl.NodeShardAllocationImpl.auditShard(NodeShardAllocationImpl.java:208)
....
apache.usergrid.persistence.graph.serialization.impl.shard.impl.ShardGroupCompactionImpl$ShardAuditTask.call(ShardGroupCompactionImpl.java:368)
at
However, there is Hystrix code in WriteUniqueVerify and using a default timeout in Hystrix of 1 sec. Bumping that up to 5 seconds.
was (Author: mrusso):
This is a non issues in 2.1.1 as there is no hystrix code in the classes mentioned in this stack strace:
at org.apache.usergrid.persistence.core.astyanax.MultiRowColumnIterator.advance(MultiRowColumnIterator.java:187)
at org.apache.usergrid.persistence.core.astyanax.MultiRowColumnIterator.hasNext(MultiRowColumnIterator.java:124)
at org.apache.usergrid.persistence.graph.serialization.impl.shard.impl.ShardsColumnIterator.hasNext(ShardsColumnIterator.java:65)
at org.apache.usergrid.persistence.graph.serialization.impl.shard.impl.NodeShardAllocationImpl.auditShard(NodeShardAllocationImpl.java:208)
....
apache.usergrid.persistence.graph.serialization.impl.shard.impl.ShardGroupCompactionImpl$ShardAuditTask.call(ShardGroupCompactionImpl.java:368)
at
However, there is Hystrix code in WriteUniqueVerify and using a default timeout in Hystrix of 1 sec. Bumping that up to 5 seconds.
> Many Histrix timeouts though Cassndra is not under load
> -------------------------------------------------------
>
> Key: USERGRID-507
> URL: https://issues.apache.org/jira/browse/USERGRID-507
> Project: Usergrid
> Issue Type: Bug
> Components: Stack
> Affects Versions: 2.1.0
> Reporter: Jeffrey
> Assignee: Michael Russo
> Priority: Minor
> Fix For: 2.1.1
>
>
> Cassandra is not under heavy load but I am seeing a high number of Hystrix timeouts. Should we increase connection pool size? Is there an operational setting that can help this?
> 2015-03-24 18:34:17,420 [graphTaskExecutor-3] ERROR org.apache.usergrid.persistence.core.task.NamedTaskExecutorImpl- Unable to execute task. Exception is
> com.netflix.hystrix.exception.HystrixRuntimeException: HystrixCassandra$1 timed-out and no fallback available.
> at com.netflix.hystrix.HystrixCommand.getFallbackOrThrowException(HystrixCommand.java:1646)
> at com.netflix.hystrix.HystrixCommand.access$1900(HystrixCommand.java:103)
> at com.netflix.hystrix.HystrixCommand$TimeoutObservable$1$1.run(HystrixCommand.java:1023)
> at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable$1.call(HystrixContextRunnable.java:41)
> at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable$1.call(HystrixContextRunnable.java:37)
> at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable.run(HystrixContextRunnable.java:57)
> at com.netflix.hystrix.HystrixCommand$TimeoutObservable$1$2.tick(HystrixCommand.java:1047)
> at com.netflix.hystrix.HystrixCommand$1.performBlockingGetWithTimeout(HystrixCommand.java:627)
> at com.netflix.hystrix.HystrixCommand$1.get(HystrixCommand.java:522)
> at com.netflix.hystrix.HystrixCommand.execute(HystrixCommand.java:431)
> at org.apache.usergrid.persistence.core.hystrix.HystrixCassandra.user(HystrixCassandra.java:69)
> at org.apache.usergrid.persistence.core.astyanax.MultiRowColumnIterator.advance(MultiRowColumnIterator.java:187)
> at org.apache.usergrid.persistence.core.astyanax.MultiRowColumnIterator.hasNext(MultiRowColumnIterator.java:124)
> at org.apache.usergrid.persistence.graph.serialization.impl.shard.impl.ShardsColumnIterator.hasNext(ShardsColumnIterator.java:65)
> at org.apache.usergrid.persistence.graph.serialization.impl.shard.impl.NodeShardAllocationImpl.auditShard(NodeShardAllocationImpl.java:208)
> at sun.reflect.GeneratedMethodAccessor193.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37)
> at com.sun.proxy.$Proxy92.auditShard(Unknown Source)
> at org.apache.usergrid.persistence.graph.serialization.impl.shard.impl.ShardGroupCompactionImpl$ShardAuditTask.call(ShardGroupCompactionImpl.java:368)
> at org.apache.usergrid.persistence.graph.serialization.impl.shard.impl.ShardGroupCompactionImpl$ShardAuditTask.call(ShardGroupCompactionImpl.java:319)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.util.concurrent.TimeoutException
> ... 25 more
> 2015-03-24 18:34:17,420 [graphTaskExecutor-3] ERROR org.apache.usergrid.persistence.graph.serialization.impl.shard.impl.ShardGroupCompactionImpl- Unable to execute audit for shard of {}
> com.netflix.hystrix.exception.HystrixRuntimeException: HystrixCassandra$1 timed-out and no fallback available.
> at com.netflix.hystrix.HystrixCommand.getFallbackOrThrowException(HystrixCommand.java:1646)
> at com.netflix.hystrix.HystrixCommand.access$1900(HystrixCommand.java:103)
> at com.netflix.hystrix.HystrixCommand$TimeoutObservable$1$1.run(HystrixCommand.java:1023)
> at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable$1.call(HystrixContextRunnable.java:41)
> at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable$1.call(HystrixContextRunnable.java:37)
> at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable.run(HystrixContextRunnable.java:57)
> at com.netflix.hystrix.HystrixCommand$TimeoutObservable$1$2.tick(HystrixCommand.java:1047)
> at com.netflix.hystrix.HystrixCommand$1.performBlockingGetWithTimeout(HystrixCommand.java:627)
> at com.netflix.hystrix.HystrixCommand$1.get(HystrixCommand.java:522)
> at com.netflix.hystrix.HystrixCommand.execute(HystrixCommand.java:431)
> at org.apache.usergrid.persistence.core.hystrix.HystrixCassandra.user(HystrixCassandra.java:69)
> at org.apache.usergrid.persistence.core.astyanax.MultiRowColumnIterator.advance(MultiRowColumnIterator.java:187)
> at org.apache.usergrid.persistence.core.astyanax.MultiRowColumnIterator.hasNext(MultiRowColumnIterator.java:124)
> at org.apache.usergrid.persistence.graph.serialization.impl.shard.impl.ShardsColumnIterator.hasNext(ShardsColumnIterator.java:65)
> at org.apache.usergrid.persistence.graph.serialization.impl.shard.impl.NodeShardAllocationImpl.auditShard(NodeShardAllocationImpl.java:208)
> at sun.reflect.GeneratedMethodAccessor193.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37)
> at com.sun.proxy.$Proxy92.auditShard(Unknown Source)
> at org.apache.usergrid.persistence.graph.serialization.impl.shard.impl.ShardGroupCompactionImpl$ShardAuditTask.call(ShardGroupCompactionImpl.java:368)
> at org.apache.usergrid.persistence.graph.serialization.impl.shard.impl.ShardGroupCompactionImpl$ShardAuditTask.call(ShardGroupCompactionImpl.java:319)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.util.concurrent.TimeoutException
> ... 25 more
> 2015-03-24 18:34:17,420 [graphTaskExecutor-3] ERROR org.apache.usergrid.persistence.graph.serialization.impl.shard.impl.ShardGroupCompactionImpl- Unable to perform audit. Exception is
> com.netflix.hystrix.exception.HystrixRuntimeException: HystrixCassandra$1 timed-out and no fallback available.
> at com.netflix.hystrix.HystrixCommand.getFallbackOrThrowException(HystrixCommand.java:1646)
> at com.netflix.hystrix.HystrixCommand.access$1900(HystrixCommand.java:103)
> at com.netflix.hystrix.HystrixCommand$TimeoutObservable$1$1.run(HystrixCommand.java:1023)
> at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable$1.call(HystrixContextRunnable.java:41)
> at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable$1.call(HystrixContextRunnable.java:37)
> at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable.run(HystrixContextRunnable.java:57)
> at com.netflix.hystrix.HystrixCommand$TimeoutObservable$1$2.tick(HystrixCommand.java:1047)
> at com.netflix.hystrix.HystrixCommand$1.performBlockingGetWithTimeout(HystrixCommand.java:627)
> at com.netflix.hystrix.HystrixCommand$1.get(HystrixCommand.java:522)
> at com.netflix.hystrix.HystrixCommand.execute(HystrixCommand.java:431)
> at org.apache.usergrid.persistence.core.hystrix.HystrixCassandra.user(HystrixCassandra.java:69)
> at org.apache.usergrid.persistence.core.astyanax.MultiRowColumnIterator.advance(MultiRowColumnIterator.java:187)
> at org.apache.usergrid.persistence.core.astyanax.MultiRowColumnIterator.hasNext(MultiRowColumnIterator.java:124)
> at org.apache.usergrid.persistence.graph.serialization.impl.shard.impl.ShardsColumnIterator.hasNext(ShardsColumnIterator.java:65)
> at org.apache.usergrid.persistence.graph.serialization.impl.shard.impl.NodeShardAllocationImpl.auditShard(NodeShardAllocationImpl.java:208)
> at sun.reflect.GeneratedMethodAccessor193.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37)
> at com.sun.proxy.$Proxy92.auditShard(Unknown Source)
> at org.apache.usergrid.persistence.graph.serialization.impl.shard.impl.ShardGroupCompactionImpl$ShardAuditTask.call(ShardGroupCompactionImpl.java:368)
> at org.apache.usergrid.persistence.graph.serialization.impl.shard.impl.ShardGroupCompactionImpl$ShardAuditTask.call(ShardGroupCompactionImpl.java:319)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.util.concurrent.TimeoutException
> ... 25 more
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)