You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@accumulo.apache.org by "mohit.kaushik" <mo...@orkash.com> on 2015/11/09 13:12:25 UTC

Slow ReadProcessor (Degrade in performance)

I have a application which query/inserts data in to Accumulo 
continuously. But the ingestion performance degrades after a day and 
continuously degrades by a considerable factor. I scanned all the hadoop 
and Accumulo logs but only found a warning in my two tservers logs after 
some inserts regularly.

[hdfs.DFSClient] WARN : Slow ReadProcessor read fields took 30783ms 
(threshold=30000ms); ack: seqno: 427381 reply: SUCCESS reply: SUCCESS 
reply: SUCCESS downstreamAckTimeNanos: 66816204 flag: 0 flag: 0 flag: 0, 
targets: 
[DatanodeInfoWithStorage[192.168.10.122:50010,DS-188489f9-89d3-40bd-9d20-9db358d644c9,DISK], 
DatanodeInfoWithStorage[192.168.10.123:50010,DS-804290f9-edd0-45a4-ad2d-bb1bd336a348,DISK], 
DatanodeInfoWithStorage[192.168.10.121:50010,DS-dd6d6a25-122f-4958-a20b-4ccb82f49f11,DISK]] 


After monitoring the query time, It seems to be increased(for ex from 30 
ms to 3 sec)after a day and continues. I also get the connection reset 
exception regularly in Tserver logs in both read and write operations. 
Which I read that it comes when a connection is closed before scan or 
write completes. But how does it affect?

015-11-08 11:25:23,272 [util.CustomNonBlockingServer$CustomFrameBuffer] 
WARN : Got an IOException in internalRead!
java.io.IOException: Connection reset by peer
         at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
         at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
         at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
         at sun.nio.ch.IOUtil.read(IOUtil.java:197)
         at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
         at 
org.apache.thrift.transport.TNonblockingSocket.read(TNonblockingSocket.java:141)
         at 
org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.internalRead(AbstractNonblockingServer.java:537)
         at 
org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.read(AbstractNonblockingServer.java:338)
         at 
org.apache.thrift.server.AbstractNonblockingServer$AbstractSelectThread.handleRead(AbstractNonblockingServer.java:203)
         at 
org.apache.accumulo.server.util.CustomNonBlockingServer$SelectAcceptThread.select(CustomNonBlockingServer.java:227)
         at 
org.apache.accumulo.server.util.CustomNonBlockingServer$SelectAcceptThread.run(CustomNonBlockingServer.java:183) 


  Please provide your inputs.

-Mohit Kaushik

Re: Slow ReadProcessor (Degrade in performance)

Posted by Keith Turner <ke...@deenlo.com>.
Mohit,

Were you able to figure this problem out?   Do you have enough info to know
if its a hdfs, Accumulo, network, or OS problem?

Keith

On Mon, Nov 9, 2015 at 7:12 AM, mohit.kaushik <mo...@orkash.com>
wrote:

>
> I have a application which query/inserts data in to Accumulo continuously.
> But the ingestion performance degrades after a day and continuously
> degrades by a considerable factor. I scanned all the hadoop and Accumulo
> logs but only found a warning in my two tservers logs after some inserts
> regularly.
>
> [hdfs.DFSClient] WARN : Slow ReadProcessor read fields took 30783ms
> (threshold=30000ms); ack: seqno: 427381 reply: SUCCESS reply: SUCCESS
> reply: SUCCESS downstreamAckTimeNanos: 66816204 flag: 0 flag: 0 flag: 0,
> targets: [DatanodeInfoWithStorage[192.168.10.122:50010,DS-188489f9-89d3-40bd-9d20-9db358d644c9,DISK],
> DatanodeInfoWithStorage[192.168.10.123:50010,DS-804290f9-edd0-45a4-ad2d-bb1bd336a348,DISK],
> DatanodeInfoWithStorage[192.168.10.121:50010,DS-dd6d6a25-122f-4958-a20b-4ccb82f49f11,DISK]]
>
>
> After monitoring the query time, It seems to be increased(for ex from 30
> ms to 3 sec)after a day and continues. I also get the connection reset
> exception regularly in Tserver logs in both read and write operations.
> Which I read that it comes when a connection is closed before scan or write
> completes. But how does it affect?
>
> 015-11-08 11:25:23,272 [util.CustomNonBlockingServer$CustomFrameBuffer]
> WARN : Got an IOException in internalRead!
> java.io.IOException: Connection reset by peer
>         at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
>         at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
>         at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
>         at sun.nio.ch.IOUtil.read(IOUtil.java:197)
>         at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
>         at
> org.apache.thrift.transport.TNonblockingSocket.read(TNonblockingSocket.java:141)
>         at
> org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.internalRead(AbstractNonblockingServer.java:537)
>         at
> org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.read(AbstractNonblockingServer.java:338)
>         at
> org.apache.thrift.server.AbstractNonblockingServer$AbstractSelectThread.handleRead(AbstractNonblockingServer.java:203)
>         at
> org.apache.accumulo.server.util.CustomNonBlockingServer$SelectAcceptThread.select(CustomNonBlockingServer.java:227)
>         at
> org.apache.accumulo.server.util.CustomNonBlockingServer$SelectAcceptThread.run(CustomNonBlockingServer.java:183)
>
>
>  Please provide your inputs.
>
> -Mohit Kaushik
>
>
>