You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by ch huang <ju...@gmail.com> on 2013/08/06 09:39:26 UTC
scan very slow in hbase
my workmate tell me hbase is very slow for scan something ,i check the
region server find the following information,anyone can help?
13/08/06 15:30:34 WARN ipc.HBaseServer: IPC Server listener on 60020:
readAndProcess threw exception java.io.IOException: Connection reset by
peer. Count of bytes read: 0
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198)
at sun.nio.ch.IOUtil.read(IOUtil.java:171)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:245)
at
org.apache.hadoop.hbase.ipc.HBaseServer.channelRead(HBaseServer.java:1796)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:1179)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:748)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:539)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
13/08/06 15:30:34 ERROR regionserver.HRegionServer:
org.apache.hadoop.hbase.ipc.CallerDisconnectedException: Aborting call
next(1916648340315433886, 1), rpc version=1, client version=29,
methodsFingerPrint=-1368823753 from 192.168.2.209:1150 after 8504 ms, since
caller disconnected
at
org.apache.hadoop.hbase.ipc.HBaseServer$Call.throwExceptionIfCallerDisconnected(HBaseServer.java:436)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3856)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3776)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3768)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2513)
at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
Re: scan very slow in hbase
Posted by Marcos Luis Ortiz Valmaseda <ma...@gmail.com>.
Regards, ch.
I was looking a post about this topic and I found it. It was posted in the
Ericsson Labs´s blog, and it talked about the approach they followed to
improve HBase performance in several ways, including Scans. You can read it
here:
http://labs.ericsson.com/blog/hbase-performance-tuners
Other resources:
Ameya´s talk about the HBase´s use at Groupon (See his last slide):
http://www.slideshare.net/cloudera/case-studies-session-3b<http://es.slideshare.net/cloudera/case-studies-session-3b>
Manoj and Goving talk:
http://www.slideshare.net/cloudera/hbasecon-2013-evolving-a-firstgeneration-apache-hbase-deployment-to-second-generation-and-beyond<http://es.slideshare.net/cloudera/hbasecon-2013-evolving-a-firstgeneration-apache-hbase-deployment-to-second-generation-and-beyond>
2013/8/6 Marcos Luis Ortiz Valmaseda <ma...@gmail.com>
> Regards, ch.
> Which version of HBase are you using?
> HBase´s development group have worked very hard in Scans improvements
>
> http://www.slideshare.net/cloudera/hbase-consistency-and-performance-final<http://es.slideshare.net/cloudera/hbase-consistency-and-performance-final>
>
> http://www.slideshare.net/cloudera/6-real-time-analytics-with-h-base-alex-baranau-sematext-final-3-updated-last-minute<http://es.slideshare.net/cloudera/6-real-time-analytics-with-h-base-alex-baranau-sematext-final-3-updated-last-minute>
>
> http://www.slideshare.net/cloudera/3-learning-h-base-internals-lars-hofhansl-salesforce-final<http://es.slideshare.net/cloudera/3-learning-h-base-internals-lars-hofhansl-salesforce-final>
>
>
> ---------- Forwarded message ----------
> From: ch huang <ju...@gmail.com>
> Date: 2013/8/6
> Subject: Re: scan very slow in hbase
> To: user@hbase.apache.org
>
>
> i find the error is because i stop the running java code,i have not found
> why the scan is so slow
>
> On Tue, Aug 6, 2013 at 10:38 PM, Stack <st...@duboce.net> wrote:
>
> > I would suggest you search the mail archives before posting first (you
> will
> > usually get your answer faster if you go this route).
> >
> > The below has been answered in the recent past. See
> > http://search-hadoop.com/m/5tk8QnhFqw
> >
> > Thanks,
> > St.Ack
> >
> >
> > On Tue, Aug 6, 2013 at 12:39 AM, ch huang <ju...@gmail.com> wrote:
> >
> > > my workmate tell me hbase is very slow for scan something ,i check the
> > > region server find the following information,anyone can help?
> > >
> > >
> > > 13/08/06 15:30:34 WARN ipc.HBaseServer: IPC Server listener on 60020:
> > > readAndProcess threw exception java.io.IOException: Connection reset by
> > > peer. Count of bytes read: 0
> > > java.io.IOException: Connection reset by peer
> > > at sun.nio.ch.FileDispatcher.read0(Native Method)
> > > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
> > > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198)
> > > at sun.nio.ch.IOUtil.read(IOUtil.java:171)
> > > at
> sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:245)
> > > at
> > >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer.channelRead(HBaseServer.java:1796)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:1179)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:748)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:539)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
> > > at
> > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at
> > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 13/08/06 15:30:34 ERROR regionserver.HRegionServer:
> > > org.apache.hadoop.hbase.ipc.CallerDisconnectedException: Aborting call
> > > next(1916648340315433886, 1), rpc version=1, client version=29,
> > > methodsFingerPrint=-1368823753 from 192.168.2.209:1150 after 8504 ms,
> > > since
> > > caller disconnected
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Call.throwExceptionIfCallerDisconnected(HBaseServer.java:436)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3856)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3776)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3768)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2513)
> > > at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
> > > at
> > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > > at java.lang.reflect.Method.invoke(Method.java:597)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
> > > at
> > >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
> > >
> >
>
>
>
> --
> Marcos Ortiz Valmaseda
> Product Manager at PDVSA
> http://about.me/marcosortiz
>
>
--
Marcos Ortiz Valmaseda
Product Manager at PDVSA
http://about.me/marcosortiz
Fwd: scan very slow in hbase
Posted by Marcos Luis Ortiz Valmaseda <ma...@gmail.com>.
Regards, ch.
Which version of HBase are you using?
HBase´s development group have worked very hard in Scans improvements
http://www.slideshare.net/cloudera/hbase-consistency-and-performance-final<http://es.slideshare.net/cloudera/hbase-consistency-and-performance-final>
http://www.slideshare.net/cloudera/6-real-time-analytics-with-h-base-alex-baranau-sematext-final-3-updated-last-minute<http://es.slideshare.net/cloudera/6-real-time-analytics-with-h-base-alex-baranau-sematext-final-3-updated-last-minute>
http://www.slideshare.net/cloudera/3-learning-h-base-internals-lars-hofhansl-salesforce-final<http://es.slideshare.net/cloudera/3-learning-h-base-internals-lars-hofhansl-salesforce-final>
---------- Forwarded message ----------
From: ch huang <ju...@gmail.com>
Date: 2013/8/6
Subject: Re: scan very slow in hbase
To: user@hbase.apache.org
i find the error is because i stop the running java code,i have not found
why the scan is so slow
On Tue, Aug 6, 2013 at 10:38 PM, Stack <st...@duboce.net> wrote:
> I would suggest you search the mail archives before posting first (you
will
> usually get your answer faster if you go this route).
>
> The below has been answered in the recent past. See
> http://search-hadoop.com/m/5tk8QnhFqw
>
> Thanks,
> St.Ack
>
>
> On Tue, Aug 6, 2013 at 12:39 AM, ch huang <ju...@gmail.com> wrote:
>
> > my workmate tell me hbase is very slow for scan something ,i check the
> > region server find the following information,anyone can help?
> >
> >
> > 13/08/06 15:30:34 WARN ipc.HBaseServer: IPC Server listener on 60020:
> > readAndProcess threw exception java.io.IOException: Connection reset by
> > peer. Count of bytes read: 0
> > java.io.IOException: Connection reset by peer
> > at sun.nio.ch.FileDispatcher.read0(Native Method)
> > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
> > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198)
> > at sun.nio.ch.IOUtil.read(IOUtil.java:171)
> > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:245)
> > at
> >
> org.apache.hadoop.hbase.ipc.HBaseServer.channelRead(HBaseServer.java:1796)
> > at
> >
> >
>
org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:1179)
> > at
> >
> >
>
org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:748)
> > at
> >
> >
>
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:539)
> > at
> >
> >
>
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
> > at
> >
> >
>
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > at
> >
> >
>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > at java.lang.Thread.run(Thread.java:662)
> > 13/08/06 15:30:34 ERROR regionserver.HRegionServer:
> > org.apache.hadoop.hbase.ipc.CallerDisconnectedException: Aborting call
> > next(1916648340315433886, 1), rpc version=1, client version=29,
> > methodsFingerPrint=-1368823753 from 192.168.2.209:1150 after 8504 ms,
> > since
> > caller disconnected
> > at
> >
> >
>
org.apache.hadoop.hbase.ipc.HBaseServer$Call.throwExceptionIfCallerDisconnected(HBaseServer.java:436)
> > at
> >
> >
>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3856)
> > at
> >
> >
>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3776)
> > at
> >
> >
>
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3768)
> > at
> >
> >
>
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2513)
> > at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
> > at
> >
> >
>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> >
>
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
> > at
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
> >
>
--
Marcos Ortiz Valmaseda
Product Manager at PDVSA
http://about.me/marcosortiz
Re: scan very slow in hbase
Posted by ch huang <ju...@gmail.com>.
i find the error is because i stop the running java code,i have not found
why the scan is so slow
On Tue, Aug 6, 2013 at 10:38 PM, Stack <st...@duboce.net> wrote:
> I would suggest you search the mail archives before posting first (you will
> usually get your answer faster if you go this route).
>
> The below has been answered in the recent past. See
> http://search-hadoop.com/m/5tk8QnhFqw
>
> Thanks,
> St.Ack
>
>
> On Tue, Aug 6, 2013 at 12:39 AM, ch huang <ju...@gmail.com> wrote:
>
> > my workmate tell me hbase is very slow for scan something ,i check the
> > region server find the following information,anyone can help?
> >
> >
> > 13/08/06 15:30:34 WARN ipc.HBaseServer: IPC Server listener on 60020:
> > readAndProcess threw exception java.io.IOException: Connection reset by
> > peer. Count of bytes read: 0
> > java.io.IOException: Connection reset by peer
> > at sun.nio.ch.FileDispatcher.read0(Native Method)
> > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
> > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198)
> > at sun.nio.ch.IOUtil.read(IOUtil.java:171)
> > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:245)
> > at
> >
> org.apache.hadoop.hbase.ipc.HBaseServer.channelRead(HBaseServer.java:1796)
> > at
> >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:1179)
> > at
> >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:748)
> > at
> >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:539)
> > at
> >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > at java.lang.Thread.run(Thread.java:662)
> > 13/08/06 15:30:34 ERROR regionserver.HRegionServer:
> > org.apache.hadoop.hbase.ipc.CallerDisconnectedException: Aborting call
> > next(1916648340315433886, 1), rpc version=1, client version=29,
> > methodsFingerPrint=-1368823753 from 192.168.2.209:1150 after 8504 ms,
> > since
> > caller disconnected
> > at
> >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Call.throwExceptionIfCallerDisconnected(HBaseServer.java:436)
> > at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3856)
> > at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3776)
> > at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3768)
> > at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2513)
> > at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> >
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
> > at
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
> >
>
Re: scan very slow in hbase
Posted by Azuryy Yu <az...@gmail.com>.
Try to set scan cache to 200. and what size of your block cache? did you
check the GC log during slow scan?
On Wed, Aug 7, 2013 at 9:20 AM, ch huang <ju...@gmail.com> wrote:
> i set the cache still no help ,here is my code
>
>
> HTable table = new HTable(conf, tablename);
> Scan scan = new Scan();
>
> scan.setTimeRange(NumberUtils.toLong("1369108800515"),
> NumberUtils.toLong("1369108810515"));
>
> scan.setCaching(10000);
>
> System.out.println("start get scan");
> ResultScanner rs = table.getScanner(scan);
>
> while(true){
> try {
> Result k = rs.next();
> if(k == null) break;
> System.out.println("get rowkey:" + new String(k.getRow()));
> }catch (Exception e){
> e.printStackTrace();
> break;
> }
> }
>
>
>
> On Wed, Aug 7, 2013 at 12:29 AM, lars hofhansl <la...@apache.org> wrote:
>
> > In addition to that, make sure you have scanner caching enabled
> > (Scan.setCaching).
> > By default (in 0.94) every call to scanner.next() causes its own RPC
> > request.
> >
> > -- Lars
> >
> >
> >
> > ________________________________
> > From: Stack <st...@duboce.net>
> > To: Hbase-User <us...@hbase.apache.org>
> > Sent: Tuesday, August 6, 2013 7:38 AM
> > Subject: Re: scan very slow in hbase
> >
> >
> > I would suggest you search the mail archives before posting first (you
> will
> > usually get your answer faster if you go this route).
> >
> > The below has been answered in the recent past. See
> > http://search-hadoop.com/m/5tk8QnhFqw
> >
> > Thanks,
> > St.Ack
> >
> >
> > On Tue, Aug 6, 2013 at 12:39 AM, ch huang <ju...@gmail.com> wrote:
> >
> > > my workmate tell me hbase is very slow for scan something ,i check the
> > > region server find the following information,anyone can help?
> > >
> > >
> > > 13/08/06 15:30:34 WARN ipc.HBaseServer: IPC Server listener on 60020:
> > > readAndProcess threw exception java.io.IOException: Connection reset by
> > > peer. Count of bytes read: 0
> > > java.io.IOException: Connection reset by peer
> > > at sun.nio.ch.FileDispatcher.read0(Native Method)
> > > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
> > > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198)
> > > at sun.nio.ch.IOUtil.read(IOUtil.java:171)
> > > at
> sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:245)
> > > at
> > >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer.channelRead(HBaseServer.java:1796)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:1179)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:748)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:539)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
> > > at
> > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > at
> > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > at java.lang.Thread.run(Thread.java:662)
> > > 13/08/06 15:30:34 ERROR regionserver.HRegionServer:
> > > org.apache.hadoop.hbase.ipc.CallerDisconnectedException: Aborting call
> > > next(1916648340315433886, 1), rpc version=1, client version=29,
> > > methodsFingerPrint=-1368823753 from 192.168.2.209:1150 after 8504 ms,
> > > since
> > > caller disconnected
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Call.throwExceptionIfCallerDisconnected(HBaseServer.java:436)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3856)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3776)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3768)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2513)
> > > at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
> > > at
> > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > > at java.lang.reflect.Method.invoke(Method.java:597)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
> > > at
> > >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
> > >
> >
>
Re: scan very slow in hbase
Posted by ch huang <ju...@gmail.com>.
i set the cache still no help ,here is my code
HTable table = new HTable(conf, tablename);
Scan scan = new Scan();
scan.setTimeRange(NumberUtils.toLong("1369108800515"),
NumberUtils.toLong("1369108810515"));
scan.setCaching(10000);
System.out.println("start get scan");
ResultScanner rs = table.getScanner(scan);
while(true){
try {
Result k = rs.next();
if(k == null) break;
System.out.println("get rowkey:" + new String(k.getRow()));
}catch (Exception e){
e.printStackTrace();
break;
}
}
On Wed, Aug 7, 2013 at 12:29 AM, lars hofhansl <la...@apache.org> wrote:
> In addition to that, make sure you have scanner caching enabled
> (Scan.setCaching).
> By default (in 0.94) every call to scanner.next() causes its own RPC
> request.
>
> -- Lars
>
>
>
> ________________________________
> From: Stack <st...@duboce.net>
> To: Hbase-User <us...@hbase.apache.org>
> Sent: Tuesday, August 6, 2013 7:38 AM
> Subject: Re: scan very slow in hbase
>
>
> I would suggest you search the mail archives before posting first (you will
> usually get your answer faster if you go this route).
>
> The below has been answered in the recent past. See
> http://search-hadoop.com/m/5tk8QnhFqw
>
> Thanks,
> St.Ack
>
>
> On Tue, Aug 6, 2013 at 12:39 AM, ch huang <ju...@gmail.com> wrote:
>
> > my workmate tell me hbase is very slow for scan something ,i check the
> > region server find the following information,anyone can help?
> >
> >
> > 13/08/06 15:30:34 WARN ipc.HBaseServer: IPC Server listener on 60020:
> > readAndProcess threw exception java.io.IOException: Connection reset by
> > peer. Count of bytes read: 0
> > java.io.IOException: Connection reset by peer
> > at sun.nio.ch.FileDispatcher.read0(Native Method)
> > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
> > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198)
> > at sun.nio.ch.IOUtil.read(IOUtil.java:171)
> > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:245)
> > at
> >
> org.apache.hadoop.hbase.ipc.HBaseServer.channelRead(HBaseServer.java:1796)
> > at
> >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:1179)
> > at
> >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:748)
> > at
> >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:539)
> > at
> >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > at java.lang.Thread.run(Thread.java:662)
> > 13/08/06 15:30:34 ERROR regionserver.HRegionServer:
> > org.apache.hadoop.hbase.ipc.CallerDisconnectedException: Aborting call
> > next(1916648340315433886, 1), rpc version=1, client version=29,
> > methodsFingerPrint=-1368823753 from 192.168.2.209:1150 after 8504 ms,
> > since
> > caller disconnected
> > at
> >
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Call.throwExceptionIfCallerDisconnected(HBaseServer.java:436)
> > at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3856)
> > at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3776)
> > at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3768)
> > at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2513)
> > at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> >
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
> > at
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
> >
>
Re: scan very slow in hbase
Posted by lars hofhansl <la...@apache.org>.
In addition to that, make sure you have scanner caching enabled (Scan.setCaching).
By default (in 0.94) every call to scanner.next() causes its own RPC request.
-- Lars
________________________________
From: Stack <st...@duboce.net>
To: Hbase-User <us...@hbase.apache.org>
Sent: Tuesday, August 6, 2013 7:38 AM
Subject: Re: scan very slow in hbase
I would suggest you search the mail archives before posting first (you will
usually get your answer faster if you go this route).
The below has been answered in the recent past. See
http://search-hadoop.com/m/5tk8QnhFqw
Thanks,
St.Ack
On Tue, Aug 6, 2013 at 12:39 AM, ch huang <ju...@gmail.com> wrote:
> my workmate tell me hbase is very slow for scan something ,i check the
> region server find the following information,anyone can help?
>
>
> 13/08/06 15:30:34 WARN ipc.HBaseServer: IPC Server listener on 60020:
> readAndProcess threw exception java.io.IOException: Connection reset by
> peer. Count of bytes read: 0
> java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcher.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198)
> at sun.nio.ch.IOUtil.read(IOUtil.java:171)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:245)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer.channelRead(HBaseServer.java:1796)
> at
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:1179)
> at
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:748)
> at
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:539)
> at
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 13/08/06 15:30:34 ERROR regionserver.HRegionServer:
> org.apache.hadoop.hbase.ipc.CallerDisconnectedException: Aborting call
> next(1916648340315433886, 1), rpc version=1, client version=29,
> methodsFingerPrint=-1368823753 from 192.168.2.209:1150 after 8504 ms,
> since
> caller disconnected
> at
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Call.throwExceptionIfCallerDisconnected(HBaseServer.java:436)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3856)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3776)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3768)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2513)
> at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
>
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
>
Re: scan very slow in hbase
Posted by Stack <st...@duboce.net>.
I would suggest you search the mail archives before posting first (you will
usually get your answer faster if you go this route).
The below has been answered in the recent past. See
http://search-hadoop.com/m/5tk8QnhFqw
Thanks,
St.Ack
On Tue, Aug 6, 2013 at 12:39 AM, ch huang <ju...@gmail.com> wrote:
> my workmate tell me hbase is very slow for scan something ,i check the
> region server find the following information,anyone can help?
>
>
> 13/08/06 15:30:34 WARN ipc.HBaseServer: IPC Server listener on 60020:
> readAndProcess threw exception java.io.IOException: Connection reset by
> peer. Count of bytes read: 0
> java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcher.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198)
> at sun.nio.ch.IOUtil.read(IOUtil.java:171)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:245)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer.channelRead(HBaseServer.java:1796)
> at
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:1179)
> at
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:748)
> at
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:539)
> at
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 13/08/06 15:30:34 ERROR regionserver.HRegionServer:
> org.apache.hadoop.hbase.ipc.CallerDisconnectedException: Aborting call
> next(1916648340315433886, 1), rpc version=1, client version=29,
> methodsFingerPrint=-1368823753 from 192.168.2.209:1150 after 8504 ms,
> since
> caller disconnected
> at
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Call.throwExceptionIfCallerDisconnected(HBaseServer.java:436)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3856)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3776)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3768)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2513)
> at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
>
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
>