You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@accumulo.apache.org by "mohit.kaushik" <mo...@orkash.com> on 2016/03/11 10:41:50 UTC
Re: IOException in internalRead! & transient exception communicating
with ZooKeeper
I have upgraded to Accumulo-1.7.1 but the problem doesn't goes
completely. Now strangely I am getting the same error on a single server
not all. Is it because of the lookup that the application always does to
check the existence of a document before inserting one?
recent logs
Keith if you say I will create a jira issue for this if required.
Thanks
On 02/27/2016 04:03 AM, Keith Turner wrote:
>
>
> On Fri, Feb 26, 2016 at 7:33 AM, mohit.kaushik
> <mohit.kaushik@orkash.com <ma...@orkash.com>> wrote:
>
> Thanks Keith, But My Accumulo clients are using same connection
> object. And the count for these WARN increase every second . Can
> Monitor cause these exceptions?
>
>
> I don't think so, but not 100% sure. I think the MAster process
> usually talks to the tservers to gather info and then the monitor
> talks to the tserver.
>
> I am wondering if there is any reason that this message should be
> logged at WARN. Seems like a routine event, should we open an issue
> to look into logging this at a lower level?
>
>
>
> On 02/24/2016 08:18 PM, Keith Turner wrote:
>> You can probably ignore those. I think its caused by an Accumulo
>> client closing its connection.
>>
>> On Wed, Feb 24, 2016 at 6:35 AM, mohit.kaushik
>> <mohit.kaushik@orkash.com <ma...@orkash.com>> wrote:
>>
>> here is screenshot, should I ignore these warnings?
>>
>>
>> internal read exception
>>
>>
>> On 02/22/2016 12:23 PM, mohit.kaushik wrote:
>>> Sent so early...
>>>
>>> Another exception I am getting frequently with zookeeper
>>> which is a bigger problem.
>>> ACCUMULO-3336
>>> <https://issues.apache.org/jira/browse/ACCUMULO-3336> says
>>> it is unresolved yet
>>> Saw (possibly) transient exception communicating with ZooKeeper
>>> org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /accumulo/f8708e0d-9238-41f5-b948-8f435fd01207/gc/lock
>>> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
>>> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>>> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
>>> at org.apache.accumulo.fate.zookeeper.ZooReader.getStatus(ZooReader.java:132)
>>> at org.apache.accumulo.fate.zookeeper.ZooLock.process(ZooLock.java:383)
>>> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
>>> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
>>> And the worst case is whenever a zookeeper goes down cluster
>>> becomes unreacheble for the time being, untill it restarts
>>> ingest process halts.
>>>
>>> What do you suggest, I need to resolve these problems. I do
>>> not want to be the ingest process to stop ever.
>>>
>>> Thanks
>>> Mohit kaushik
>>>
>>>
>>> On 02/22/2016 12:06 PM, mohit.kaushik wrote:
>>>> I am facing the below given exception continuously, the count keeps on increasing every sec(current value around 3000 on a server) I can see the exception for all 3 tablet servers.
>>>>
>>>> ACCUMULO-2420 <https://issues.apache.org/jira/browse/ACCUMULO-2420> says that this exception comes when a client closes a connection before scan completes. But the connection is not closed every thread uses a common connection object to ingest and query, then what could cause this exception?
>>>>
>>>> java.io.IOException: Connection reset by peer
>>>> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
>>>> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
>>>> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
>>>> at sun.nio.ch.IOUtil.read(IOUtil.java:197)
>>>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
>>>> at org.apache.thrift.transport.TNonblockingSocket.read(TNonblockingSocket.java:141)
>>>> at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.internalRead(AbstractNonblockingServer.java:537)
>>>> at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.read(AbstractNonblockingServer.java:338)
>>>> at org.apache.thrift.server.AbstractNonblockingServer$AbstractSelectThread.handleRead(AbstractNonblockingServer.java:203)
>>>> at org.apache.accumulo.server.rpc.CustomNonBlockingServer$SelectAcceptThread.select(CustomNonBlockingServer.java:228)
>>>> at org.apache.accumulo.server.rpc.CustomNonBlockingServer$SelectAcceptThread.run(CustomNonBlockingServer.java:184)
>>>>
>>>> Regards
>>>> Mohit kaushik
>>>>
>
Re: IOException in internalRead! & transient exception communicating
with ZooKeeper
Posted by Josh Elser <jo...@gmail.com>.
Are you using a Scanner and then a BatchWriter for that existence check?
mohit.kaushik wrote:
> I have upgraded to Accumulo-1.7.1 but the problem doesn't goes
> completely. Now strangely I am getting the same error on a single server
> not all. Is it because of the lookup that the application always does to
> check the existence of a document before inserting one?
>
> recent logs
> Keith if you say I will create a jira issue for this if required.
>
> Thanks
>
>
> On 02/27/2016 04:03 AM, Keith Turner wrote:
>>
>>
>> On Fri, Feb 26, 2016 at 7:33 AM, mohit.kaushik
>> <mohit.kaushik@orkash.com <ma...@orkash.com>> wrote:
>>
>> Thanks Keith, But My Accumulo clients are using same connection
>> object. And the count for these WARN increase every second . Can
>> Monitor cause these exceptions?
>>
>>
>> I don't think so, but not 100% sure. I think the MAster process
>> usually talks to the tservers to gather info and then the monitor
>> talks to the tserver.
>>
>> I am wondering if there is any reason that this message should be
>> logged at WARN. Seems like a routine event, should we open an issue
>> to look into logging this at a lower level?
>>
>>
>>
>> On 02/24/2016 08:18 PM, Keith Turner wrote:
>>> You can probably ignore those. I think its caused by an Accumulo
>>> client closing its connection.
>>>
>>> On Wed, Feb 24, 2016 at 6:35 AM, mohit.kaushik
>>> <mohit.kaushik@orkash.com <ma...@orkash.com>> wrote:
>>>
>>> here is screenshot, should I ignore these warnings?
>>>
>>>
>>> internal read exception
>>>
>>>
>>> On 02/22/2016 12:23 PM, mohit.kaushik wrote:
>>>> Sent so early...
>>>>
>>>> Another exception I am getting frequently with zookeeper
>>>> which is a bigger problem.
>>>> ACCUMULO-3336
>>>> <https://issues.apache.org/jira/browse/ACCUMULO-3336> says
>>>> it is unresolved yet
>>>> Saw (possibly) transient exception communicating with ZooKeeper
>>>> org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /accumulo/f8708e0d-9238-41f5-b948-8f435fd01207/gc/lock
>>>> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
>>>> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>>>> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
>>>> at org.apache.accumulo.fate.zookeeper.ZooReader.getStatus(ZooReader.java:132)
>>>> at org.apache.accumulo.fate.zookeeper.ZooLock.process(ZooLock.java:383)
>>>> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
>>>> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
>>>> And the worst case is whenever a zookeeper goes down cluster
>>>> becomes unreacheble for the time being, untill it restarts
>>>> ingest process halts.
>>>>
>>>> What do you suggest, I need to resolve these problems. I do
>>>> not want to be the ingest process to stop ever.
>>>>
>>>> Thanks
>>>> Mohit kaushik
>>>>
>>>>
>>>> On 02/22/2016 12:06 PM, mohit.kaushik wrote:
>>>>> I am facing the below given exception continuously, the count keeps on increasing every sec(current value around 3000 on a server) I can see the exception for all 3 tablet servers.
>>>>>
>>>>> ACCUMULO-2420 <https://issues.apache.org/jira/browse/ACCUMULO-2420> says that this exception comes when a client closes a connection before scan completes. But the connection is not closed every thread uses a common connection object to ingest and query, then what could cause this exception?
>>>>>
>>>>> java.io.IOException: Connection reset by peer
>>>>> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
>>>>> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
>>>>> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
>>>>> at sun.nio.ch.IOUtil.read(IOUtil.java:197)
>>>>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
>>>>> at org.apache.thrift.transport.TNonblockingSocket.read(TNonblockingSocket.java:141)
>>>>> at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.internalRead(AbstractNonblockingServer.java:537)
>>>>> at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.read(AbstractNonblockingServer.java:338)
>>>>> at org.apache.thrift.server.AbstractNonblockingServer$AbstractSelectThread.handleRead(AbstractNonblockingServer.java:203)
>>>>> at org.apache.accumulo.server.rpc.CustomNonBlockingServer$SelectAcceptThread.select(CustomNonBlockingServer.java:228)
>>>>> at org.apache.accumulo.server.rpc.CustomNonBlockingServer$SelectAcceptThread.run(CustomNonBlockingServer.java:184)
>>>>>
>>>>> Regards
>>>>> Mohit kaushik
>>>>>
>>