You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2021/01/13 20:53:15 UTC
[GitHub] [druid] Karthikeyantkr opened a new issue #10753: Redis connection error
Karthikeyantkr opened a new issue #10753:
URL: https://github.com/apache/druid/issues/10753
`WARN [ForkJoinPool-1-worker-22] org.apache.druid.client.cache.AbstractRedisCache - Exception pushing item to cache
redis.clients.jedis.exceptions.JedisNoReachableClusterNodeException: No reachable node in cluster
at redis.clients.jedis.JedisSlotBasedConnectionHandler.getConnection(JedisSlotBasedConnectionHandler.java:57) ~[jedis-2.9.0.jar:?]
at redis.clients.jedis.JedisSlotBasedConnectionHandler.getConnectionFromSlot(JedisSlotBasedConnectionHandler.java:74) ~[jedis-2.9.0.jar:?]
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:116) ~[jedis-2.9.0.jar:?]
at redis.clients.jedis.JedisClusterCommand.runBinary(JedisClusterCommand.java:60) ~[jedis-2.9.0.jar:?]
at redis.clients.jedis.BinaryJedisCluster.setex(BinaryJedisCluster.java:268) ~[jedis-2.9.0.jar:?]
at org.apache.druid.client.cache.RedisClusterCache.putToRedis(RedisClusterCache.java:46) ~[druid-redis-cache-0.20.0.jar:0.20.0]
at org.apache.druid.client.cache.AbstractRedisCache.put(AbstractRedisCache.java:84) [druid-redis-cache-0.20.0.jar:0.20.0]
at org.apache.druid.client.cache.ForegroundCachePopulator$1.after(ForegroundCachePopulator.java:114) [druid-server-0.20.0.jar:0.20.0]
at org.apache.druid.java.util.common.guava.WrappingYielder.close(WrappingYielder.java:98) [druid-core-0.20.0.jar:0.20.0]
at org.apache.druid.java.util.common.guava.MergeSequence.makeYielder(MergeSequence.java:146) [druid-core-0.20.0.jar:0.20.0]
at org.apache.druid.java.util.common.guava.MergeSequence.toYielder(MergeSequence.java:105) [druid-core-0.20.0.jar:0.20.0]
at org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence$ResultBatch.fromSequence(ParallelMergeCombiningSequence.java:869) [druid-core-0.20.0.jar:0.20.0]
at org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence$SequenceBatcher.block(ParallelMergeCombiningSequence.java:920) [druid-core-0.20.0.jar:0.20.0]
at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3313) [?:1.8.0_121]
at org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence$SequenceBatcher.getBatchYielder(ParallelMergeCombiningSequence.java:909) [druid-core-0.20.0.jar:0.20.0]
at org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence$YielderBatchedResultsCursor.initialize(ParallelMergeCombiningSequence.java:1017) [druid-core-0.20.0.jar:0.20.0]
at org.apache.druid.java.util.common.guava.ParallelMergeCombiningSequence$PrepareMergeCombineInputsAction.compute(ParallelMergeCombiningSequence.java:721) [druid-core-0.20.0.jar:0.20.0]
at java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189) [?:1.8.0_121]
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) [?:1.8.0_121]
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) [?:1.8.0_121]
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) [?:1.8.0_121]
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157) [?:1.8.0_121]`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 commented on issue #10753: Redis cluster connection error while trying to use as cache
Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on issue #10753:
URL: https://github.com/apache/druid/issues/10753#issuecomment-765067654
> @FrankChen021 I tried a large timeout and was able to solve segment cache in historical.
> But still getting the mget exception in broker.
Getting multiple values from different nodes by MGET are not supported by JEDIS now. I didn't notice this problem when I implemented redis cluster extension. We have to update the extension, either by calculating slots before such an operation or replacing jedis with lettuce.
I don't have a large chunk of time to do it these days, so I'm sorry that patch may not be shipped with 0.21 which is already on the way of release.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] Karthikeyantkr edited a comment on issue #10753: Redis cluster connection error while trying to use as cache
Posted by GitBox <gi...@apache.org>.
Karthikeyantkr edited a comment on issue #10753:
URL: https://github.com/apache/druid/issues/10753#issuecomment-761224017
@FrankChen021 I tried a large timeout and was able to solve segment cache in historical.
But still getting the mget exception in broker.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 commented on issue #10753: Redis cluster connection error while trying to use as cache
Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on issue #10753:
URL: https://github.com/apache/druid/issues/10753#issuecomment-759874052
`cluster info` commands should outputs something like this
> 172.16.1.1:48003> cluster info
> cluster_state:ok
> cluster_slots_assigned:16384
> cluster_slots_ok:16384
> cluster_slots_pfail:0
> cluster_slots_fail:0
> cluster_known_nodes:6
> cluster_size:3
> cluster_current_epoch:6
> cluster_my_epoch:3
> cluster_stats_messages_ping_sent:72
> cluster_stats_messages_pong_sent:75
> cluster_stats_messages_sent:147
> cluster_stats_messages_ping_received:75
> cluster_stats_messages_pong_received:72
> cluster_stats_messages_received:147
And could you tell me:
1. does this problem come out every time a query is being executed ?
2. what's the version of your redis server ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 commented on issue #10753: Redis cluster connection error while trying to use as cache
Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on issue #10753:
URL: https://github.com/apache/druid/issues/10753#issuecomment-759866529
The exception stack reports an exception message saying `No reachable node in cluster`
Please use the redis command `cluster info` to check if the cluster is running OK
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis closed issue #10753: Redis cluster connection error while trying to use as cache
Posted by GitBox <gi...@apache.org>.
clintropolis closed issue #10753:
URL: https://github.com/apache/druid/issues/10753
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 commented on issue #10753: Redis cluster connection error while trying to use as cache
Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on issue #10753:
URL: https://github.com/apache/druid/issues/10753#issuecomment-760587384
> @FrankChen021 Any thoughts on this? Thanks for your time.
Based on my knowledge, I think this problem is more related to redis cluster itself and I have no idea what happened. It will take some time for me to investigate it. Thanks for your patience.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] Karthikeyantkr commented on issue #10753: Redis cluster connection error while trying to use as cache
Posted by GitBox <gi...@apache.org>.
Karthikeyantkr commented on issue #10753:
URL: https://github.com/apache/druid/issues/10753#issuecomment-760477607
This error in historical node : ```
2021-01-14T20:56:13,647 WARN [segmentMetadata_dsp_media_and_bids_[2020-08-09T09:00:00.000Z/2020-08-09T10:00:00.000Z]] org.apache.druid.client.cache.AbstractRedisCache - Exception pulling item from cache
redis.clients.jedis.exceptions.JedisNoReachableClusterNodeException: No reachable node in cluster
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 commented on issue #10753: Redis cluster connection error while trying to use as cache
Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on issue #10753:
URL: https://github.com/apache/druid/issues/10753#issuecomment-765067654
> @FrankChen021 I tried a large timeout and was able to solve segment cache in historical.
> But still getting the mget exception in broker.
Getting multiple values from different nodes by MGET are not supported by JEDIS now. I didn't notice this problem when I implemented redis cluster extension. We have to update the extension, either by calculating slots before such an operation or replacing jedis with lettuce.
I don't have a large chunk of time to do it these days, so I'm sorry that patch may not be shipped with 0.21 which is already on the way of release.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] Karthikeyantkr commented on issue #10753: Redis cluster connection error while trying to use as cache
Posted by GitBox <gi...@apache.org>.
Karthikeyantkr commented on issue #10753:
URL: https://github.com/apache/druid/issues/10753#issuecomment-760473363
I am seeing this message now when I restart the broker.
```
2021-01-14T20:56:13,304 WARN [DruidSchema-Cache-0] org.apache.druid.client.cache.AbstractRedisCache - Exception pulling items from cache
redis.clients.jedis.exceptions.JedisClusterException: No way to dispatch this command to Redis Cluster because keys have different slots.
at redis.clients.jedis.JedisClusterCommand.runBinary(JedisClusterCommand.java:75) ~[?:?]
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] Karthikeyantkr commented on issue #10753: Redis cluster connection error while trying to use as cache
Posted by GitBox <gi...@apache.org>.
Karthikeyantkr commented on issue #10753:
URL: https://github.com/apache/druid/issues/10753#issuecomment-760472690
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] Karthikeyantkr commented on issue #10753: Redis cluster connection error while trying to use as cache
Posted by GitBox <gi...@apache.org>.
Karthikeyantkr commented on issue #10753:
URL: https://github.com/apache/druid/issues/10753#issuecomment-760474818
@FrankChen021 Any thoughts on this? Thanks for your time.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] Karthikeyantkr edited a comment on issue #10753: Redis cluster connection error while trying to use as cache
Posted by GitBox <gi...@apache.org>.
Karthikeyantkr edited a comment on issue #10753:
URL: https://github.com/apache/druid/issues/10753#issuecomment-760473363
I am seeing this message now when I restart the broker. Broker log:
```
2021-01-14T20:56:13,304 WARN [DruidSchema-Cache-0] org.apache.druid.client.cache.AbstractRedisCache - Exception pulling items from cache
redis.clients.jedis.exceptions.JedisClusterException: No way to dispatch this command to Redis Cluster because keys have different slots.
at redis.clients.jedis.JedisClusterCommand.runBinary(JedisClusterCommand.java:75) ~[?:?]
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] Karthikeyantkr commented on issue #10753: Redis cluster connection error while trying to use as cache
Posted by GitBox <gi...@apache.org>.
Karthikeyantkr commented on issue #10753:
URL: https://github.com/apache/druid/issues/10753#issuecomment-760474355
1) I don't see an error message every time when I run a query. (Since the Redis was not connected at startup itself)
2) Redis version : redis-cli 6.0.5
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] Karthikeyantkr commented on issue #10753: Redis cluster connection error while trying to use as cache
Posted by GitBox <gi...@apache.org>.
Karthikeyantkr commented on issue #10753:
URL: https://github.com/apache/druid/issues/10753#issuecomment-761224017
I tried a large timeout and also bind IP. still the same exception.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] Karthikeyantkr commented on issue #10753: Redis cluster connection error while trying to use as cache
Posted by GitBox <gi...@apache.org>.
Karthikeyantkr commented on issue #10753:
URL: https://github.com/apache/druid/issues/10753#issuecomment-761223669
@FrankChen021
```
2021-01-15T22:06:13,611 WARN [DruidSchema-Cache-0] org.apache.druid.client.cache.AbstractRedisCache - Exception pulling items from cache
redis.clients.jedis.exceptions.JedisClusterException: No way to dispatch this command to Redis Cluster because keys have different slots.
at redis.clients.jedis.JedisClusterCommand.runBinary(JedisClusterCommand.java:75) ~[?:?]
at redis.clients.jedis.BinaryJedisCluster.mget(BinaryJedisCluster.java:1362) ~[?:?]
at org.apache.druid.client.cache.RedisClusterCache.mgetFromRedis(RedisClusterCache.java:52) ~[?:?]
at org.apache.druid.client.cache.AbstractRedisCache.getBulk(AbstractRedisCache.java:102) ~[?:?]
at org.apache.druid.client.CachingClusteredClient$SpecificQueryRunnable.computeCachedValues(CachingClusteredClient.java:586) ~[druid-server-0.20.0.jar:0.20.0]
at org.apache.druid.client.CachingClusteredClient$SpecificQueryRunnable.pruneSegmentsWithCachedResults(CachingClusteredClient.java:546) ~[druid-server-0.20.0.jar:0.20.0]
at org.apache.druid.client.CachingClusteredClient$SpecificQueryRunnable.run(CachingClusteredClient.java:359) ~[druid-server-0.20.0.jar:0.20.0]
at org.apache.druid.client.CachingClusteredClient.run(CachingClusteredClient.java:196) ~[druid-server-0.20.0.jar:0.20.0]
at org.apache.druid.client.CachingClusteredClient.access$100(CachingClusteredClient.java:111) ~[druid-server-0.20.0.jar:0.20.0]
at org.apache.druid.client.CachingClusteredClient$3.run(CachingClusteredClient.java:221) ~[druid-server-0.20.0.jar:0.20.0]
at org.apache.druid.query.RetryQueryRunner.run(RetryQueryRunner.java:108) ~[druid-server-0.20.0.jar:0.20.0]
at org.apache.druid.server.SetAndVerifyContextQueryRunner.run(SetAndVerifyContextQueryRunner.java:51) ~[druid-server-0.20.0.jar:0.20.0]
at org.apache.druid.query.UnionQueryRunner.run(UnionQueryRunner.java:92) ~[druid-processing-0.20.0.jar:0.20.0]
at org.apache.druid.query.metadata.SegmentMetadataQueryQueryToolChest$3.doRun(SegmentMetadataQueryQueryToolChest.java:124) ~[druid-processing-0.20.0.jar:0.20.0]
at org.apache.druid.query.BySegmentSkippingQueryRunner.run(BySegmentSkippingQueryRunner.java:46) ~[druid-processing-0.20.0.jar:0.20.0]
at org.apache.druid.query.FinalizeResultsQueryRunner.run(FinalizeResultsQueryRunner.java:110) ~[druid-processing-0.20.0.jar:0.20.0]
at org.apache.druid.query.CPUTimeMetricQueryRunner.run(CPUTimeMetricQueryRunner.java:65) ~[druid-processing-0.20.0.jar:0.20.0]
at org.apache.druid.query.ResultLevelCachingQueryRunner.run(ResultLevelCachingQueryRunner.java:96) ~[druid-server-0.20.0.jar:0.20.0]
at org.apache.druid.query.FluentQueryRunnerBuilder$FluentQueryRunner.run(FluentQueryRunnerBuilder.java:55) ~[druid-processing-0.20.0.jar:0.20.0]
at org.apache.druid.server.ClientQuerySegmentWalker$QuerySwappingQueryRunner.run(ClientQuerySegmentWalker.java:508) ~[druid-server-0.20.0.jar:0.20.0]
at org.apache.druid.query.QueryPlus.run(QueryPlus.java:149) ~[druid-processing-0.20.0.jar:0.20.0]
at org.apache.druid.server.QueryLifecycle.execute(QueryLifecycle.java:268) ~[druid-server-0.20.0.jar:0.20.0]
at org.apache.druid.server.QueryLifecycle.runSimple(QueryLifecycle.java:142) ~[druid-server-0.20.0.jar:0.20.0]
at org.apache.druid.sql.calcite.schema.DruidSchema.runSegmentMetadataQuery(DruidSchema.java:681) ~[druid-sql-0.20.0.jar:0.20.0]
at org.apache.druid.sql.calcite.schema.DruidSchema.refreshSegmentsForDataSource(DruidSchema.java:532) ~[druid-sql-0.20.0.jar:0.20.0]
at org.apache.druid.sql.calcite.schema.DruidSchema.refreshSegments(DruidSchema.java:494) ~[druid-sql-0.20.0.jar:0.20.0]
at org.apache.druid.sql.calcite.schema.DruidSchema.lambda$start$2(DruidSchema.java:267) ~[druid-sql-0.20.0.jar:0.20.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_121]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] Karthikeyantkr edited a comment on issue #10753: Redis cluster connection error while trying to use as cache
Posted by GitBox <gi...@apache.org>.
Karthikeyantkr edited a comment on issue #10753:
URL: https://github.com/apache/druid/issues/10753#issuecomment-760472690
> 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] Karthikeyantkr commented on issue #10753: Redis cluster connection error while trying to use as cache
Posted by GitBox <gi...@apache.org>.
Karthikeyantkr commented on issue #10753:
URL: https://github.com/apache/druid/issues/10753#issuecomment-759870459
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] Karthikeyantkr edited a comment on issue #10753: Redis cluster connection error while trying to use as cache
Posted by GitBox <gi...@apache.org>.
Karthikeyantkr edited a comment on issue #10753:
URL: https://github.com/apache/druid/issues/10753#issuecomment-761224017
@FrankChen021 I tried a large timeout and was able to solve segment cache in historical.
But still getting the mget exception in broker.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis closed issue #10753: Redis cluster connection error while trying to use as cache
Posted by GitBox <gi...@apache.org>.
clintropolis closed issue #10753:
URL: https://github.com/apache/druid/issues/10753
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] Karthikeyantkr edited a comment on issue #10753: Redis cluster connection error while trying to use as cache
Posted by GitBox <gi...@apache.org>.
Karthikeyantkr edited a comment on issue #10753:
URL: https://github.com/apache/druid/issues/10753#issuecomment-760477607
This error in historical node :
```
2021-01-14T20:56:13,647 WARN [segmentMetadata_dsp_media_and_bids_[2020-08-09T09:00:00.000Z/2020-08-09T10:00:00.000Z]] org.apache.druid.client.cache.AbstractRedisCache - Exception pulling item from cache
redis.clients.jedis.exceptions.JedisNoReachableClusterNodeException: No reachable node in cluster
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] Karthikeyantkr commented on issue #10753: Redis cluster connection error while trying to use as cache
Posted by GitBox <gi...@apache.org>.
Karthikeyantkr commented on issue #10753:
URL: https://github.com/apache/druid/issues/10753#issuecomment-759869238
yes I did that and I see the clusters are running ok
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 commented on issue #10753: Redis cluster connection error while trying to use as cache
Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on issue #10753:
URL: https://github.com/apache/druid/issues/10753#issuecomment-760939773
> I am seeing this message now when I restart the broker. Broker log:
>
> ```
> 2021-01-14T20:56:13,304 WARN [DruidSchema-Cache-0] org.apache.druid.client.cache.AbstractRedisCache - Exception pulling items from cache
> redis.clients.jedis.exceptions.JedisClusterException: No way to dispatch this command to Redis Cluster because keys have different slots.
> at redis.clients.jedis.JedisClusterCommand.runBinary(JedisClusterCommand.java:75) ~[?:?]
> ```
This is because there's a mget operation trying to get values with different keys distributed on different redis slots. Jedis does not support this kind of operation. Maybe in the future, we need to replace **jedis** with **lettuce** to eliminate this error.
And could you paste the full exception stack so that I could try to reproduce it when I have time to repair it ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 commented on issue #10753: Redis cluster connection error while trying to use as cache
Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on issue #10753:
URL: https://github.com/apache/druid/issues/10753#issuecomment-760941017
> This error in historical node :
>
> ```
> 2021-01-14T20:56:13,647 WARN [segmentMetadata_dsp_media_and_bids_[2020-08-09T09:00:00.000Z/2020-08-09T10:00:00.000Z]] org.apache.druid.client.cache.AbstractRedisCache - Exception pulling item from cache
> redis.clients.jedis.exceptions.JedisNoReachableClusterNodeException: No reachable node in cluster
> ```
For this problem, please check https://github.com/redis/jedis/issues/995
It provides two solutions for two problems:
1. enlarge the timeout configuration
2. or change bind-ip
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org