You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by "Sutton, Ben" <Be...@virginmedia.co.uk> on 2015/12/03 22:12:05 UTC
RE: Slow response on HBase REST api using globbing option
We are currently experiencing an issue with HBase 1.1.1 through the REST API whereby we are getting a very slow (> 2minutes) /non-existent response when hitting the Rest API with a globbing option i.e. http://<HBase_Rest>:<HBase_Rest_Port/table/key*<http://%3cHBase_Rest%3e:%3cHBase_Rest_Port/table/key*>
We are able to return each individual row for the key with no problem i.e. http://<HBase_Rest>:<HBase_Rest_Port/table/key01<http://%3cHBase_Rest%3e:%3cHBase_Rest_Port/table/key01> or http://<HBase_Rest>:<HBase_Rest_Port/table/key02<http://%3cHBase_Rest%3e:%3cHBase_Rest_Port/table/key02> etc.
We have also found if we load in a third of the data to hbase we have no trouble with the above.
Facing this issue with 1.7billion row hbase table - we are able to retrieve data for initial rows keys through to 1/10 of the region with a slowly degrading performance after this point until no result.
This is with 19 regionservers with each having at least 5 regions.
Seeing the below error:
Failed after attempts=36, exceptions:
Tue Dec 01 12:51:29 GMT 2015, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60303: row '$ROW' on table '$TABLE' at region=$TABLE,$ROW,1416253073158.2f25cef5708de354950561f4a399da36., hostname=$LOCAL_HOST,60020,1448892673650, seqNum=444047093
(through reference chain: org.apache.hadoop.hbase.rest.CellSetModelStream["Row"])
2015-12-01 12:51:29,405 ERROR [1516838186@qtp-258535644-213] mortbay.log: /$TABLE/KEY.*
java.lang.RuntimeException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=36, exceptions:
Tue Dec 01 12:51:29 GMT 2015, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60303: row $ROW on table $TABLE at region=$TABLE,04079290507.TiVo,1416253073158.2f25cef5708de354950561f4a399da36., hostname=$LOCAL_HOST,60020,1448892673650, seqNum=444047093
at org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:97)
at org.apache.hadoop.hbase.rest.TableScanResource$1$1.hasNext(TableScanResource.java:82)
at com.sun.xml.bind.v2.runtime.reflect.Lister$CollectionLister$1.hasNext(Lister.java:285)
at com.sun.xml.bind.v2.runtime.property.ArrayElementProperty.serializeListBody(ArrayElementProperty.java:137)
at com.sun.xml.bind.v2.runtime.property.ArrayERProperty.serializeBody(ArrayERProperty.java:159)
at com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.serializeBody(ClassBeanInfoImpl.java:344)
at com.sun.xml.bind.v2.runtime.XMLSerializer.childAsSoleContent(XMLSerializer.java:597)
at com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.serializeRoot(ClassBeanInfoImpl.java:328)
at com.sun.xml.bind.v2.runtime.XMLSerializer.childAsRoot(XMLSerializer.java:498)
at com.sun.xml.bind.v2.runtime.MarshallerImpl.write(MarshallerImpl.java:320)
at com.sun.xml.bind.v2.runtime.MarshallerImpl.marshal(MarshallerImpl.java:249)
at javax.xml.bind.helpers.AbstractMarshallerImpl.marshal(AbstractMarshallerImpl.java:95)
at com.sun.jersey.core.provider.jaxb.AbstractRootElementProvider.writeTo(AbstractRootElementProvider.java:179)
at com.sun.jersey.core.provider.jaxb.AbstractRootElementProvider.writeTo(AbstractRootElementProvider.java:157)
at com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:306)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1437)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at org.apache.hadoop.hbase.rest.filter.GzipFilter.doFilter(GzipFilter.java:76)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=36, exceptions:
Anyone have any ideas?
Thanks,
Ben
--------------------------------------------------------------------
Save Paper - Do you really need to print this e-mail?
Visit www.virginmedia.com for more information, and more fun.
This email and any attachments are or may be confidential and legally privileged
and are sent solely for the attention of the addressee(s). If you have received this
email in error, please delete it from your system: its use, disclosure or copying is
unauthorised. Statements and opinions expressed in this email may not represent
those of Virgin Media. Any representations or commitments in this email are
subject to contract.
Registered office: Media House, Bartley Wood Business Park, Hook, Hampshire, RG27 9UP
Registered in England and Wales with number 2591237
Re: Slow response on HBase REST api using globbing option
Posted by Ted Yu <yu...@gmail.com>.
Created HBASE-14928 and attached patch there.
FYI
On Thu, Dec 3, 2015 at 9:05 PM, Ted Yu <yu...@gmail.com> wrote:
> Thanks for the response, Jerry.
>
> I created a patch:
>
> http://pastebin.com/xisGVHt8
>
> All REST tests passed.
>
> I know Ben logged a JIRA on this subject already.
> Not sure if that should be re-opened or, a new JIRA should be created.
>
> Once we have an open JIRA, I will attach my patch there.
>
> Cheers
>
> On Thu, Dec 3, 2015 at 8:16 PM, Jerry He <je...@gmail.com> wrote:
>
>> From HBase 0.98, there have been changes going into the Rest gateway,
>> mainly more scan support.
>>
>> There seems to be a change in the way the url table/rowkey* is executed
>> on the Rest gateway.
>> in pre-0.96, we set the startKey = rowkey and endKey = rowkey + one byte
>> of
>> 255 on Rest gateway in the Scan sent to the servers.
>> In the newer versions, table/rowkey* seems to treated with a prefixFilter.
>> This may explain why you are seeing slower response.
>>
>> Jerry
>>
>>
>>
>> On Thu, Dec 3, 2015 at 1:12 PM, Sutton, Ben <Ben.Sutton@virginmedia.co.uk
>> >
>> wrote:
>>
>> > We are currently experiencing an issue with HBase 1.1.1 through the REST
>> > API whereby we are getting a very slow (> 2minutes) /non-existent
>> response
>> > when hitting the Rest API with a globbing option i.e. http://
>> > <HBase_Rest>:<HBase_Rest_Port/table/key*<http://
>> > %3cHBase_Rest%3e:%3cHBase_Rest_Port/table/key*>
>> >
>> >
>> >
>> > We are able to return each individual row for the key with no problem
>> i.e.
>> > http://<HBase_Rest>:<HBase_Rest_Port/table/key01<http://
>> %3cHBase_Rest%3e:%3cHBase_Rest_Port/table/key01>
>> > or http://<HBase_Rest>:<HBase_Rest_Port/table/key02<http://
>> %3cHBase_Rest%3e:%3cHBase_Rest_Port/table/key02>
>> > etc.
>> >
>> > We have also found if we load in a third of the data to hbase we have no
>> > trouble with the above.
>> >
>> > Facing this issue with 1.7billion row hbase table - we are able to
>> > retrieve data for initial rows keys through to 1/10 of the region with a
>> > slowly degrading performance after this point until no result.
>> >
>> > This is with 19 regionservers with each having at least 5 regions.
>> >
>> > Seeing the below error:
>> > Failed after attempts=36, exceptions:
>> > Tue Dec 01 12:51:29 GMT 2015, null, java.net.SocketTimeoutException:
>> > callTimeout=60000, callDuration=60303: row '$ROW' on table '$TABLE' at
>> > region=$TABLE,$ROW,1416253073158.2f25cef5708de354950561f4a399da36.,
>> > hostname=$LOCAL_HOST,60020,1448892673650, seqNum=444047093
>> > (through reference chain:
>> > org.apache.hadoop.hbase.rest.CellSetModelStream["Row"])
>> > 2015-12-01 12:51:29,405 ERROR [1516838186@qtp-258535644-213]
>> mortbay.log:
>> > /$TABLE/KEY.*
>> > java.lang.RuntimeException:
>> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
>> > attempts=36, exceptions:
>> > Tue Dec 01 12:51:29 GMT 2015, null, java.net.SocketTimeoutException:
>> > callTimeout=60000, callDuration=60303: row $ROW on table $TABLE at
>> >
>> region=$TABLE,04079290507.TiVo,1416253073158.2f25cef5708de354950561f4a399da36.,
>> > hostname=$LOCAL_HOST,60020,1448892673650, seqNum=444047093
>> >
>> > at
>> >
>> org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:97)
>> > at
>> >
>> org.apache.hadoop.hbase.rest.TableScanResource$1$1.hasNext(TableScanResource.java:82)
>> > at
>> >
>> com.sun.xml.bind.v2.runtime.reflect.Lister$CollectionLister$1.hasNext(Lister.java:285)
>> > at
>> >
>> com.sun.xml.bind.v2.runtime.property.ArrayElementProperty.serializeListBody(ArrayElementProperty.java:137)
>> > at
>> >
>> com.sun.xml.bind.v2.runtime.property.ArrayERProperty.serializeBody(ArrayERProperty.java:159)
>> > at
>> >
>> com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.serializeBody(ClassBeanInfoImpl.java:344)
>> > at
>> >
>> com.sun.xml.bind.v2.runtime.XMLSerializer.childAsSoleContent(XMLSerializer.java:597)
>> > at
>> >
>> com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.serializeRoot(ClassBeanInfoImpl.java:328)
>> > at
>> >
>> com.sun.xml.bind.v2.runtime.XMLSerializer.childAsRoot(XMLSerializer.java:498)
>> > at
>> >
>> com.sun.xml.bind.v2.runtime.MarshallerImpl.write(MarshallerImpl.java:320)
>> > at
>> >
>> com.sun.xml.bind.v2.runtime.MarshallerImpl.marshal(MarshallerImpl.java:249)
>> > at
>> >
>> javax.xml.bind.helpers.AbstractMarshallerImpl.marshal(AbstractMarshallerImpl.java:95)
>> > at
>> >
>> com.sun.jersey.core.provider.jaxb.AbstractRootElementProvider.writeTo(AbstractRootElementProvider.java:179)
>> > at
>> >
>> com.sun.jersey.core.provider.jaxb.AbstractRootElementProvider.writeTo(AbstractRootElementProvider.java:157)
>> > at
>> >
>> com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:306)
>> > at
>> >
>> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1437)
>> > at
>> >
>> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
>> > at
>> >
>> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
>> > at
>> >
>> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
>> > at
>> >
>> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
>> > at
>> >
>> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
>> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>> > at
>> > org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>> > at
>> >
>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>> > at
>> >
>> org.apache.hadoop.hbase.rest.filter.GzipFilter.doFilter(GzipFilter.java:76)
>> > at
>> >
>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>> > at
>> > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>> > at
>> > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>> > at
>> >
>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>> > at
>> > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767)
>> > at
>> > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>> > at org.mortbay.jetty.Server.handle(Server.java:326)
>> > at
>> > org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>> > at
>> >
>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>> > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>> > at
>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>> > at
>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>> > at
>> >
>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>> > at
>> >
>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>> > Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException:
>> > Failed after attempts=36, exceptions:
>> >
>> > Anyone have any ideas?
>> >
>> > Thanks,
>> >
>> > Ben
>> >
>> >
>> > --------------------------------------------------------------------
>> > Save Paper - Do you really need to print this e-mail?
>> >
>> > Visit www.virginmedia.com for more information, and more fun.
>> >
>> > This email and any attachments are or may be confidential and legally
>> > privileged
>> > and are sent solely for the attention of the addressee(s). If you have
>> > received this
>> > email in error, please delete it from your system: its use, disclosure
>> or
>> > copying is
>> > unauthorised. Statements and opinions expressed in this email may not
>> > represent
>> > those of Virgin Media. Any representations or commitments in this email
>> are
>> > subject to contract.
>> >
>> > Registered office: Media House, Bartley Wood Business Park, Hook,
>> > Hampshire, RG27 9UP
>> > Registered in England and Wales with number 2591237
>> >
>>
>
>
Re: Slow response on HBase REST api using globbing option
Posted by Ted Yu <yu...@gmail.com>.
Thanks for the response, Jerry.
I created a patch:
http://pastebin.com/xisGVHt8
All REST tests passed.
I know Ben logged a JIRA on this subject already.
Not sure if that should be re-opened or, a new JIRA should be created.
Once we have an open JIRA, I will attach my patch there.
Cheers
On Thu, Dec 3, 2015 at 8:16 PM, Jerry He <je...@gmail.com> wrote:
> From HBase 0.98, there have been changes going into the Rest gateway,
> mainly more scan support.
>
> There seems to be a change in the way the url table/rowkey* is executed
> on the Rest gateway.
> in pre-0.96, we set the startKey = rowkey and endKey = rowkey + one byte of
> 255 on Rest gateway in the Scan sent to the servers.
> In the newer versions, table/rowkey* seems to treated with a prefixFilter.
> This may explain why you are seeing slower response.
>
> Jerry
>
>
>
> On Thu, Dec 3, 2015 at 1:12 PM, Sutton, Ben <Be...@virginmedia.co.uk>
> wrote:
>
> > We are currently experiencing an issue with HBase 1.1.1 through the REST
> > API whereby we are getting a very slow (> 2minutes) /non-existent
> response
> > when hitting the Rest API with a globbing option i.e. http://
> > <HBase_Rest>:<HBase_Rest_Port/table/key*<http://
> > %3cHBase_Rest%3e:%3cHBase_Rest_Port/table/key*>
> >
> >
> >
> > We are able to return each individual row for the key with no problem
> i.e.
> > http://<HBase_Rest>:<HBase_Rest_Port/table/key01<http://
> %3cHBase_Rest%3e:%3cHBase_Rest_Port/table/key01>
> > or http://<HBase_Rest>:<HBase_Rest_Port/table/key02<http://
> %3cHBase_Rest%3e:%3cHBase_Rest_Port/table/key02>
> > etc.
> >
> > We have also found if we load in a third of the data to hbase we have no
> > trouble with the above.
> >
> > Facing this issue with 1.7billion row hbase table - we are able to
> > retrieve data for initial rows keys through to 1/10 of the region with a
> > slowly degrading performance after this point until no result.
> >
> > This is with 19 regionservers with each having at least 5 regions.
> >
> > Seeing the below error:
> > Failed after attempts=36, exceptions:
> > Tue Dec 01 12:51:29 GMT 2015, null, java.net.SocketTimeoutException:
> > callTimeout=60000, callDuration=60303: row '$ROW' on table '$TABLE' at
> > region=$TABLE,$ROW,1416253073158.2f25cef5708de354950561f4a399da36.,
> > hostname=$LOCAL_HOST,60020,1448892673650, seqNum=444047093
> > (through reference chain:
> > org.apache.hadoop.hbase.rest.CellSetModelStream["Row"])
> > 2015-12-01 12:51:29,405 ERROR [1516838186@qtp-258535644-213]
> mortbay.log:
> > /$TABLE/KEY.*
> > java.lang.RuntimeException:
> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
> > attempts=36, exceptions:
> > Tue Dec 01 12:51:29 GMT 2015, null, java.net.SocketTimeoutException:
> > callTimeout=60000, callDuration=60303: row $ROW on table $TABLE at
> >
> region=$TABLE,04079290507.TiVo,1416253073158.2f25cef5708de354950561f4a399da36.,
> > hostname=$LOCAL_HOST,60020,1448892673650, seqNum=444047093
> >
> > at
> >
> org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:97)
> > at
> >
> org.apache.hadoop.hbase.rest.TableScanResource$1$1.hasNext(TableScanResource.java:82)
> > at
> >
> com.sun.xml.bind.v2.runtime.reflect.Lister$CollectionLister$1.hasNext(Lister.java:285)
> > at
> >
> com.sun.xml.bind.v2.runtime.property.ArrayElementProperty.serializeListBody(ArrayElementProperty.java:137)
> > at
> >
> com.sun.xml.bind.v2.runtime.property.ArrayERProperty.serializeBody(ArrayERProperty.java:159)
> > at
> >
> com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.serializeBody(ClassBeanInfoImpl.java:344)
> > at
> >
> com.sun.xml.bind.v2.runtime.XMLSerializer.childAsSoleContent(XMLSerializer.java:597)
> > at
> >
> com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.serializeRoot(ClassBeanInfoImpl.java:328)
> > at
> >
> com.sun.xml.bind.v2.runtime.XMLSerializer.childAsRoot(XMLSerializer.java:498)
> > at
> > com.sun.xml.bind.v2.runtime.MarshallerImpl.write(MarshallerImpl.java:320)
> > at
> >
> com.sun.xml.bind.v2.runtime.MarshallerImpl.marshal(MarshallerImpl.java:249)
> > at
> >
> javax.xml.bind.helpers.AbstractMarshallerImpl.marshal(AbstractMarshallerImpl.java:95)
> > at
> >
> com.sun.jersey.core.provider.jaxb.AbstractRootElementProvider.writeTo(AbstractRootElementProvider.java:179)
> > at
> >
> com.sun.jersey.core.provider.jaxb.AbstractRootElementProvider.writeTo(AbstractRootElementProvider.java:157)
> > at
> >
> com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:306)
> > at
> >
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1437)
> > at
> >
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> > at
> >
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> > at
> >
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> > at
> >
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> > at
> >
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> > at
> > org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> > at
> >
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
> > at
> >
> org.apache.hadoop.hbase.rest.filter.GzipFilter.doFilter(GzipFilter.java:76)
> > at
> >
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> > at
> > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> > at
> > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> > at
> >
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> > at
> > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767)
> > at
> > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> > at org.mortbay.jetty.Server.handle(Server.java:326)
> > at
> > org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
> > at
> >
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
> > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
> > at
> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
> > at
> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> > at
> >
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
> > at
> >
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> > Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException:
> > Failed after attempts=36, exceptions:
> >
> > Anyone have any ideas?
> >
> > Thanks,
> >
> > Ben
> >
> >
> > --------------------------------------------------------------------
> > Save Paper - Do you really need to print this e-mail?
> >
> > Visit www.virginmedia.com for more information, and more fun.
> >
> > This email and any attachments are or may be confidential and legally
> > privileged
> > and are sent solely for the attention of the addressee(s). If you have
> > received this
> > email in error, please delete it from your system: its use, disclosure or
> > copying is
> > unauthorised. Statements and opinions expressed in this email may not
> > represent
> > those of Virgin Media. Any representations or commitments in this email
> are
> > subject to contract.
> >
> > Registered office: Media House, Bartley Wood Business Park, Hook,
> > Hampshire, RG27 9UP
> > Registered in England and Wales with number 2591237
> >
>
RE: Slow response on HBase REST api using globbing option
Posted by "Sutton, Ben" <Be...@virginmedia.co.uk>.
Thanks Jerry - thought it may have been a change in how the Rest gateway scans.
Ben
-----Original Message-----
From: Jerry He [mailto:jerryjch@gmail.com]
Sent: 04 December 2015 04:16
To: user@hbase.apache.org
Subject: Re: Slow response on HBase REST api using globbing option
From HBase 0.98, there have been changes going into the Rest gateway, mainly more scan support.
There seems to be a change in the way the url table/rowkey* is executed on the Rest gateway.
in pre-0.96, we set the startKey = rowkey and endKey = rowkey + one byte of
255 on Rest gateway in the Scan sent to the servers.
In the newer versions, table/rowkey* seems to treated with a prefixFilter.
This may explain why you are seeing slower response.
Jerry
On Thu, Dec 3, 2015 at 1:12 PM, Sutton, Ben <Be...@virginmedia.co.uk>
wrote:
> We are currently experiencing an issue with HBase 1.1.1 through the
> REST API whereby we are getting a very slow (> 2minutes) /non-existent
> response when hitting the Rest API with a globbing option i.e. http://
> <HBase_Rest>:<HBase_Rest_Port/table/key*<http://
> %3cHBase_Rest%3e:%3cHBase_Rest_Port/table/key*>
>
>
>
> We are able to return each individual row for the key with no problem i.e.
> http://<HBase_Rest>:<HBase_Rest_Port/table/key01<http://%3cHBase_Rest%
> 3e:%3cHBase_Rest_Port/table/key01>
> or
> http://<HBase_Rest>:<HBase_Rest_Port/table/key02<http://%3cHBase_Rest%
> 3e:%3cHBase_Rest_Port/table/key02>
> etc.
>
> We have also found if we load in a third of the data to hbase we have
> no trouble with the above.
>
> Facing this issue with 1.7billion row hbase table - we are able to
> retrieve data for initial rows keys through to 1/10 of the region with
> a slowly degrading performance after this point until no result.
>
> This is with 19 regionservers with each having at least 5 regions.
>
> Seeing the below error:
> Failed after attempts=36, exceptions:
> Tue Dec 01 12:51:29 GMT 2015, null, java.net.SocketTimeoutException:
> callTimeout=60000, callDuration=60303: row '$ROW' on table '$TABLE' at
> region=$TABLE,$ROW,1416253073158.2f25cef5708de354950561f4a399da36.,
> hostname=$LOCAL_HOST,60020,1448892673650, seqNum=444047093 (through
> reference chain:
> org.apache.hadoop.hbase.rest.CellSetModelStream["Row"])
> 2015-12-01 12:51:29,405 ERROR [1516838186@qtp-258535644-213] mortbay.log:
> /$TABLE/KEY.*
> java.lang.RuntimeException:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
> attempts=36, exceptions:
> Tue Dec 01 12:51:29 GMT 2015, null, java.net.SocketTimeoutException:
> callTimeout=60000, callDuration=60303: row $ROW on table $TABLE at
> region=$TABLE,04079290507.TiVo,1416253073158.2f25cef5708de354950561f4a
> 399da36., hostname=$LOCAL_HOST,60020,1448892673650, seqNum=444047093
>
> at
> org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:97)
> at
> org.apache.hadoop.hbase.rest.TableScanResource$1$1.hasNext(TableScanResource.java:82)
> at
> com.sun.xml.bind.v2.runtime.reflect.Lister$CollectionLister$1.hasNext(Lister.java:285)
> at
> com.sun.xml.bind.v2.runtime.property.ArrayElementProperty.serializeListBody(ArrayElementProperty.java:137)
> at
> com.sun.xml.bind.v2.runtime.property.ArrayERProperty.serializeBody(ArrayERProperty.java:159)
> at
> com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.serializeBody(ClassBeanInfoImpl.java:344)
> at
> com.sun.xml.bind.v2.runtime.XMLSerializer.childAsSoleContent(XMLSerializer.java:597)
> at
> com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.serializeRoot(ClassBeanInfoImpl.java:328)
> at
> com.sun.xml.bind.v2.runtime.XMLSerializer.childAsRoot(XMLSerializer.java:498)
> at
> com.sun.xml.bind.v2.runtime.MarshallerImpl.write(MarshallerImpl.java:320)
> at
> com.sun.xml.bind.v2.runtime.MarshallerImpl.marshal(MarshallerImpl.java:249)
> at
> javax.xml.bind.helpers.AbstractMarshallerImpl.marshal(AbstractMarshallerImpl.java:95)
> at
> com.sun.jersey.core.provider.jaxb.AbstractRootElementProvider.writeTo(AbstractRootElementProvider.java:179)
> at
> com.sun.jersey.core.provider.jaxb.AbstractRootElementProvider.writeTo(AbstractRootElementProvider.java:157)
> at
> com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:306)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1437)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
> at
> org.apache.hadoop.hbase.rest.filter.GzipFilter.doFilter(GzipFilter.java:76)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> at
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767)
> at
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:326)
> at
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
> at
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> at
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
> at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.ja
> va:582) Caused by:
> org.apache.hadoop.hbase.client.RetriesExhaustedException:
> Failed after attempts=36, exceptions:
>
> Anyone have any ideas?
>
> Thanks,
>
> Ben
>
>
> --------------------------------------------------------------------
> Save Paper - Do you really need to print this e-mail?
>
> Visit www.virginmedia.com for more information, and more fun.
>
> This email and any attachments are or may be confidential and legally
> privileged and are sent solely for the attention of the addressee(s).
> If you have received this email in error, please delete it from your
> system: its use, disclosure or copying is unauthorised. Statements and
> opinions expressed in this email may not represent those of Virgin
> Media. Any representations or commitments in this email are subject to
> contract.
>
> Registered office: Media House, Bartley Wood Business Park, Hook,
> Hampshire, RG27 9UP Registered in England and Wales with number
> 2591237
>
--------------------------------------------------------------------
Save Paper - Do you really need to print this e-mail?
Visit www.virginmedia.com for more information, and more fun.
This email and any attachments are or may be confidential and legally privileged
and are sent solely for the attention of the addressee(s). If you have received this
email in error, please delete it from your system: its use, disclosure or copying is
unauthorised. Statements and opinions expressed in this email may not represent
those of Virgin Media. Any representations or commitments in this email are
subject to contract.
Registered office: Media House, Bartley Wood Business Park, Hook, Hampshire, RG27 9UP
Registered in England and Wales with number 2591237
Re: Slow response on HBase REST api using globbing option
Posted by Jerry He <je...@gmail.com>.
>From HBase 0.98, there have been changes going into the Rest gateway,
mainly more scan support.
There seems to be a change in the way the url table/rowkey* is executed
on the Rest gateway.
in pre-0.96, we set the startKey = rowkey and endKey = rowkey + one byte of
255 on Rest gateway in the Scan sent to the servers.
In the newer versions, table/rowkey* seems to treated with a prefixFilter.
This may explain why you are seeing slower response.
Jerry
On Thu, Dec 3, 2015 at 1:12 PM, Sutton, Ben <Be...@virginmedia.co.uk>
wrote:
> We are currently experiencing an issue with HBase 1.1.1 through the REST
> API whereby we are getting a very slow (> 2minutes) /non-existent response
> when hitting the Rest API with a globbing option i.e. http://
> <HBase_Rest>:<HBase_Rest_Port/table/key*<http://
> %3cHBase_Rest%3e:%3cHBase_Rest_Port/table/key*>
>
>
>
> We are able to return each individual row for the key with no problem i.e.
> http://<HBase_Rest>:<HBase_Rest_Port/table/key01<http://%3cHBase_Rest%3e:%3cHBase_Rest_Port/table/key01>
> or http://<HBase_Rest>:<HBase_Rest_Port/table/key02<http://%3cHBase_Rest%3e:%3cHBase_Rest_Port/table/key02>
> etc.
>
> We have also found if we load in a third of the data to hbase we have no
> trouble with the above.
>
> Facing this issue with 1.7billion row hbase table - we are able to
> retrieve data for initial rows keys through to 1/10 of the region with a
> slowly degrading performance after this point until no result.
>
> This is with 19 regionservers with each having at least 5 regions.
>
> Seeing the below error:
> Failed after attempts=36, exceptions:
> Tue Dec 01 12:51:29 GMT 2015, null, java.net.SocketTimeoutException:
> callTimeout=60000, callDuration=60303: row '$ROW' on table '$TABLE' at
> region=$TABLE,$ROW,1416253073158.2f25cef5708de354950561f4a399da36.,
> hostname=$LOCAL_HOST,60020,1448892673650, seqNum=444047093
> (through reference chain:
> org.apache.hadoop.hbase.rest.CellSetModelStream["Row"])
> 2015-12-01 12:51:29,405 ERROR [1516838186@qtp-258535644-213] mortbay.log:
> /$TABLE/KEY.*
> java.lang.RuntimeException:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
> attempts=36, exceptions:
> Tue Dec 01 12:51:29 GMT 2015, null, java.net.SocketTimeoutException:
> callTimeout=60000, callDuration=60303: row $ROW on table $TABLE at
> region=$TABLE,04079290507.TiVo,1416253073158.2f25cef5708de354950561f4a399da36.,
> hostname=$LOCAL_HOST,60020,1448892673650, seqNum=444047093
>
> at
> org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:97)
> at
> org.apache.hadoop.hbase.rest.TableScanResource$1$1.hasNext(TableScanResource.java:82)
> at
> com.sun.xml.bind.v2.runtime.reflect.Lister$CollectionLister$1.hasNext(Lister.java:285)
> at
> com.sun.xml.bind.v2.runtime.property.ArrayElementProperty.serializeListBody(ArrayElementProperty.java:137)
> at
> com.sun.xml.bind.v2.runtime.property.ArrayERProperty.serializeBody(ArrayERProperty.java:159)
> at
> com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.serializeBody(ClassBeanInfoImpl.java:344)
> at
> com.sun.xml.bind.v2.runtime.XMLSerializer.childAsSoleContent(XMLSerializer.java:597)
> at
> com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.serializeRoot(ClassBeanInfoImpl.java:328)
> at
> com.sun.xml.bind.v2.runtime.XMLSerializer.childAsRoot(XMLSerializer.java:498)
> at
> com.sun.xml.bind.v2.runtime.MarshallerImpl.write(MarshallerImpl.java:320)
> at
> com.sun.xml.bind.v2.runtime.MarshallerImpl.marshal(MarshallerImpl.java:249)
> at
> javax.xml.bind.helpers.AbstractMarshallerImpl.marshal(AbstractMarshallerImpl.java:95)
> at
> com.sun.jersey.core.provider.jaxb.AbstractRootElementProvider.writeTo(AbstractRootElementProvider.java:179)
> at
> com.sun.jersey.core.provider.jaxb.AbstractRootElementProvider.writeTo(AbstractRootElementProvider.java:157)
> at
> com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:306)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1437)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
> at
> org.apache.hadoop.hbase.rest.filter.GzipFilter.doFilter(GzipFilter.java:76)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> at
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767)
> at
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:326)
> at
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
> at
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> at
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
> at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException:
> Failed after attempts=36, exceptions:
>
> Anyone have any ideas?
>
> Thanks,
>
> Ben
>
>
> --------------------------------------------------------------------
> Save Paper - Do you really need to print this e-mail?
>
> Visit www.virginmedia.com for more information, and more fun.
>
> This email and any attachments are or may be confidential and legally
> privileged
> and are sent solely for the attention of the addressee(s). If you have
> received this
> email in error, please delete it from your system: its use, disclosure or
> copying is
> unauthorised. Statements and opinions expressed in this email may not
> represent
> those of Virgin Media. Any representations or commitments in this email are
> subject to contract.
>
> Registered office: Media House, Bartley Wood Business Park, Hook,
> Hampshire, RG27 9UP
> Registered in England and Wales with number 2591237
>