Cassandra Caused by: UnavailableException()

["Alaa Zubaidi (PDF)" <al...@pdf.com>]

Hi,
We have an 8 Node 1.0.10 Cassandra cluster with Replication of 3, running
on customer location, they all should be in the same DC, but we are NOT
100% sure.
We are having problem with the our own Cassandra client, the log file of
the client shows many "socket timeout" and "socket closed" errors, and many
delays, in the Reads.

while debugging the issue we found the following error in the our Cassandra
client:
Caused by: UnavailableException()

Cassandra log file does not show errors..

What does that mean? any idea?
 =========================================
[com.pdf.common.COMN_Log4jLogger] Failed to write Context Flow row to
column 20140317210716767_a3ccda262dbdd287a5e538ec95483a74_40:
UnavailableException()
at
org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:19487)
at
org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:1035)
at
org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:1009)
at
com.pdf.datastore.cassandra.DSTO_CassandraClient.putLTStoreContextFlowRow(DSTO_CassandraClient.java:2673)
at
com.pdf.rtapi.RAPI_DataStore.putLTStoreContextFlowRow(RAPI_DataStore.java:1254)
at
com.pdf.rtapi.RAPI_InternalInterface.putLTStoreContextFlowRow(RAPI_InternalInterface.java:756)
at
com.pdf.common.rtapi.impl.RtApiFacadeImpl.putLTStoreContextFlowRow(RtApiFacadeImpl.java:340)
at
com.pdf.online.service.impl.OnlineServiceImpl.notifyDCContextFlow(OnlineServiceImpl.java:1499)
at
com.pdf.online.service.impl.OnlineServiceImpl.process(OnlineServiceImpl.java:602)

============================================
Different error..

2014-Mar-17 21:51:20,531 - INFO  TID=Thread-6
(HornetQ-client-global-threads-4002470)
[com.pdf.online.service.impl.OnlineServiceImpl] Recipe name is:
P13_01_28NM_M5_002AS3_NOVA
2014-Mar-17 21:51:20,531 - INFO  TID=Thread-6
(HornetQ-client-global-threads-4002470)
[com.pdf.online.service.impl.OnlineServiceImpl] Reading framed data (trace
+ context) for 'dcId' = 44140
2014-Mar-17 21:51:20,531 - INFO  TID=Thread-6
(HornetQ-client-global-threads-4002470) [com.pdf.common.COMN_Log4jLogger]
RAPI_DataStore::readFramedData() -- Creating key base:
equipmentId_moduleId: POL1702_CleanerBrush1
2014-Mar-17 21:51:20,531 - INFO  TID=Thread-6
(HornetQ-client-global-threads-4002470) [com.pdf.common.COMN_Log4jLogger]
RAPI_DataStore::readFramedData() -- Hashed key:
12169de1f273c1d38c893c0dfcbf10ff
2014-Mar-17 21:51:20,531 - INFO  TID=Thread-6
(HornetQ-client-global-threads-4002470) [com.pdf.common.COMN_Log4jLogger]
RAPI_DataStore::readFramedData() -- DSTO_ContextKey =
12169de1f273c1d38c893c0dfcbf10ff:20131115070726207, dcId = 44140
2014-Mar-17 21:51:20,531 - INFO  TID=Thread-6
(HornetQ-client-global-threads-4002470) [com.pdf.common.COMN_Log4jLogger]
Call checkConnection().
2014-Mar-17 21:51:20,531 - INFO  TID=Thread-6
(HornetQ-client-global-threads-4002470) [com.pdf.common.COMN_Log4jLogger]
Call checkConnection().
2014-Mar-17 21:51:20,531 - INFO  TID=Thread-6
(HornetQ-client-global-threads-4002470) [com.pdf.common.COMN_Log4jLogger]
UnavailableException : null: UnavailableException()
at
org.apache.cassandra.thrift.Cassandra$get_slice_result.read(Cassandra.java:7764)
at
org.apache.cassandra.thrift.Cassandra$Client.recv_get_slice(Cassandra.java:570)
at
org.apache.cassandra.thrift.Cassandra$Client.get_slice(Cassandra.java:542)
at
com.pdf.datastore.cassandra.DSTO_CassandraClient.readContextData(DSTO_CassandraClient.java:1894)
at
com.pdf.datastore.cassandra.DSTO_CassandraClient.readFramedData(DSTO_CassandraClient.java:1848)
at com.pdf.rtapi.RAPI_DataStore.readFramedData(RAPI_DataStore.java:720)
at
com.pdf.rtapi.RAPI_InternalInterface.readFramedData(RAPI_InternalInterface.java:741)
at
com.pdf.common.rtapi.impl.RtApiFacadeImpl.readFramedData(RtApiFacadeImpl.java:261)
at
com.pdf.online.service.impl.OnlineServiceImpl.readCassandraData(OnlineServiceImpl.java:2189)
at
com.pdf.online.service.impl.OnlineServiceImpl.process(OnlineServiceImpl.java:328)
at
com.pdf.online.mdb.OnlineEventDrivenBean.process(OnlineEventDrivenBean.java:302)
at
com.pdf.online.mdb.OnlineEventDrivenBean.onMessage(OnlineEventDrivenBean.java:211)
at sun.reflect.GeneratedMethodAccessor83.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jboss.invocation.Invocation.performCall(Invocation.java:359)
at
org.jboss.ejb.MessageDrivenContainer$ContainerInterceptor.invoke(MessageDrivenContainer.java:495)
at
org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:158)
at
org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)
at
org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)
at
org.jboss.ejb.plugins.AbstractTxInterceptorBMT.invokeNext(AbstractTxInterceptorBMT.java:173)
at
org.jboss.ejb.plugins.MessageDrivenTxInterceptorBMT.invoke(MessageDrivenTxInterceptorBMT.java:47)
at
org.jboss.ejb.plugins.MessageDrivenInstanceInterceptor.invoke(MessageDrivenInstanceInterceptor.java:116)
at
org.jboss.ejb.plugins.RunAsSecurityInterceptor.invoke(RunAsSecurityInterceptor.java:109)
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)
at
org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)
at
org.jboss.ejb.MessageDrivenContainer.internalInvoke(MessageDrivenContainer.java:402)
at org.jboss.ejb.Container.invoke(Container.java:960)
at sun.reflect.GeneratedMethodAccessor82.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
at
org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
at
org.jboss.invocation.local.LocalInvoker$MBeanServerAction.invoke(LocalInvoker.java:169)
at org.jboss.invocation.local.LocalInvoker.invoke(LocalInvoker.java:118)
at
org.jboss.invocation.InvokerInterceptor.invokeLocal(InvokerInterceptor.java:209)
at
org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:195)
at
org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:61)
at
org.jboss.ejb.plugins.inflow.MessageEndpointInterceptor.delivery(MessageEndpointInterceptor.java:249)
at
org.jboss.ejb.plugins.inflow.MessageEndpointInterceptor.invoke(MessageEndpointInterceptor.java:128)
at
org.jboss.proxy.ClientMethodInterceptor.invoke(ClientMethodInterceptor.java:74)
at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:100)
at $Proxy51.onMessage(Unknown Source)
at
org.hornetq.ra.inflow.HornetQMessageHandler.onMessage(HornetQMessageHandler.java:278)
at
org.hornetq.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:994)
at
org.hornetq.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:48)
at
org.hornetq.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1120)
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

2014-Mar-17 21:51:20,562 - ERROR TID=Thread-6
(HornetQ-client-global-threads-4002470)
[com.pdf.online.service.impl.OnlineServiceImpl] Couldn't read data from
cassandra database using the parameters: key '2013-11-15T07:07:26.207',
equipment name: 'POL1702', module name 'CleanerBrush1'.
com.pdf.common.rtapi.exceptions.RtApiException: Could not read Framed data.
at
com.pdf.common.rtapi.impl.RtApiFacadeImpl.readFramedData(RtApiFacadeImpl.java:285)
at
com.pdf.online.service.impl.OnlineServiceImpl.readCassandraData(OnlineServiceImpl.java:2189)
at
com.pdf.online.service.impl.OnlineServiceImpl.process(OnlineServiceImpl.java:328)
at
com.pdf.online.mdb.OnlineEventDrivenBean.process(OnlineEventDrivenBean.java:302)
at
com.pdf.online.mdb.OnlineEventDrivenBean.onMessage(OnlineEventDrivenBean.java:211)
at sun.reflect.GeneratedMethodAccessor83.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jboss.invocation.Invocation.performCall(Invocation.java:359)
at
org.jboss.ejb.MessageDrivenContainer$ContainerInterceptor.invoke(MessageDrivenContainer.java:495)
at
org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:158)
at
org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)
at
org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)
at
org.jboss.ejb.plugins.AbstractTxInterceptorBMT.invokeNext(AbstractTxInterceptorBMT.java:173)
at
org.jboss.ejb.plugins.MessageDrivenTxInterceptorBMT.invoke(MessageDrivenTxInterceptorBMT.java:47)
at
org.jboss.ejb.plugins.MessageDrivenInstanceInterceptor.invoke(MessageDrivenInstanceInterceptor.java:116)
at
org.jboss.ejb.plugins.RunAsSecurityInterceptor.invoke(RunAsSecurityInterceptor.java:109)
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)
at
org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)
at
org.jboss.ejb.MessageDrivenContainer.internalInvoke(MessageDrivenContainer.java:402)
at org.jboss.ejb.Container.invoke(Container.java:960)
at sun.reflect.GeneratedMethodAccessor82.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
at
org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
at
org.jboss.invocation.local.LocalInvoker$MBeanServerAction.invoke(LocalInvoker.java:169)
at org.jboss.invocation.local.LocalInvoker.invoke(LocalInvoker.java:118)
at
org.jboss.invocation.InvokerInterceptor.invokeLocal(InvokerInterceptor.java:209)
at
org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:195)
at
org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:61)
at
org.jboss.ejb.plugins.inflow.MessageEndpointInterceptor.delivery(MessageEndpointInterceptor.java:249)
at
org.jboss.ejb.plugins.inflow.MessageEndpointInterceptor.invoke(MessageEndpointInterceptor.java:128)
at
org.jboss.proxy.ClientMethodInterceptor.invoke(ClientMethodInterceptor.java:74)
at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:100)
at $Proxy51.onMessage(Unknown Source)
at
org.hornetq.ra.inflow.HornetQMessageHandler.onMessage(HornetQMessageHandler.java:278)
at
org.hornetq.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:994)
at
org.hornetq.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:48)
at
org.hornetq.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1120)
at
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: com.pdf.datastore.exception.DSTO_Exception: Failed to read
ContextData using key = 12169de1f273c1d38c893c0dfcbf10ff:20131115070726207,
DCID = 44140
at
com.pdf.datastore.cassandra.DSTO_CassandraClient.readContextData(DSTO_CassandraClient.java:1949)
at
com.pdf.datastore.cassandra.DSTO_CassandraClient.readFramedData(DSTO_CassandraClient.java:1848)
at com.pdf.rtapi.RAPI_DataStore.readFramedData(RAPI_DataStore.java:720)
at
com.pdf.rtapi.RAPI_InternalInterface.readFramedData(RAPI_InternalInterface.java:741)
at
com.pdf.common.rtapi.impl.RtApiFacadeImpl.readFramedData(RtApiFacadeImpl.java:261)
... 46 more

-- 

Alaa

-- 
*This message may contain confidential and privileged information. If it 
has been sent to you in error, please reply to advise the sender of the 
error and then immediately permanently delete it and all attachments to it 
from your systems. If you are not the intended recipient, do not read, 
copy, disclose or otherwise use this message or any attachments to it. The 
sender disclaims any liability for such unauthorized use. PLEASE NOTE that 
all incoming e-mails sent to PDF e-mail accounts will be archived and may 
be scanned by us and/or by external service providers to detect and prevent 
threats to our systems, investigate illegal or inappropriate behavior, 
and/or eliminate unsolicited promotional e-mails ("spam"). If you have any 
concerns about this process, please contact us at *
*legal.department@pdf.com* <le...@pdf.com>*.*