You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Wukang Lin <vb...@gmail.com> on 2013/12/03 17:22:17 UTC
Strange Problem on using HRegion's getScanner Method in RegionServer
Hi all,
I got a trouble in using HRegion's RegionScanner in RegionServer.
In my project, a custom RPC server was started in RegionServer using
coprocessor mechanism, and it work well. In this RPC server, it receive a
call from client, and scan the HRegion specialised by the params setted by
the client like this:
---------------------------------------------------------------------------------------------------------------------
HRegion region = this.getHRegionByEncodeName(encodeName);
Scan scan = new Scan();
RegionScanner scanner = null;
try {
scanner = region.getScanner(scan);
long total = 0;
boolean hasNext = true;
List<KeyValue> kvs = new ArrayList<KeyValue>();
while (hasNext) {
kvs.clear();
hasNext = scanner.next(kvs);
if (kvs.size() > 0) {
total++;
}
} catch (IOException e) {
LOG.warn("Exception thrown when scanning region '" + regionName
+ "', SKIP IT!", e);
} finally {
if (scanner != null) {
try {
scanner.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
LOG.info("Total " + total + " rows scanned.");
---------------------------------------------------------------------------------------------------------------------
Then, I put a row to a HRegion by HTable interface, and make an RPC call of
my custom RPC server to scan the HRegion(the code lines above is executed
in a HRegionServer, a HRegion is scanned), and got nothing, total = 0. But
if I use client side scan like HTable's scan interface or HBase shell, I
can get the row I insert just now. I tried to flush the region and do a
major compaction on the HRegion using hbase shell, Nothing changed.
So, what's wrong with the code above? how to scan a HRegion in
HRegionServer instance?
Any responses is Appreciated!
Re: Strange Problem on using HRegion's getScanner Method in RegionServer
Posted by Wukang Lin <vb...@gmail.com>.
hi Gary,
it is true that it is not a standard usage of coprocessor mechanism. I
use coprocessor as a plugin framework. the endpoint pattern of coprocessor
is pretty good, but not available for me, aggregaate on the client side is
not fast enough for me, and it is not necessary to transfer all the
mid-data to client. What's more, some of my applications is not a simple
Map-Reduce like style. so we try to setup a non-standard and
highly-customed compute framework inside hbase. It works well before we got
this strange trouble.
Any hints and advices are welcome!
Thank you!
Re: Strange Problem on using HRegion's getScanner Method in RegionServer
Posted by Gary Helmling <gh...@gmail.com>.
> hi Asaf,
> Thank you for your response. the rpc server in my application is a
> singleton
> instance. It is started in a region observer, and work as a single server
> in the HRegionServer, just like the RPC servers bring up in the RS's main()
> Method. It not attatched with any Table or regions, It can get all the
> HRegion instance on the HRegionServer.
>
>
That sounds like a pretty non-standard setup. Why not just use the normal
coprocessor endpoint mechanism and execute a scan on the local region,
instead of going through a regionserver-scoped singleton? Then you can
aggregate results on the client.
Within each coprocessor endpoint instance, you can simply call
RegionCoprocessorEnvironment.getRegion() to get a reference to the local
region.
Re: Strange Problem on using HRegion's getScanner Method in RegionServer
Posted by Wukang Lin <vb...@gmail.com>.
hi Asaf,
Thank you for your response. the rpc server in my application is a singleton
instance. It is started in a region observer, and work as a single server
in the HRegionServer, just like the RPC servers bring up in the RS's main()
Method. It not attatched with any Table or regions, It can get all the
HRegion instance on the HRegionServer.
2013/12/5 Asaf Mesika <as...@gmail.com>
> How do you check in your code you only have one region?
> How do you retrieve the HRegion exactly? Since you are running in inside a
> coprocessor I presume you have such rpc server per region since Hbase
> creates an instance of your coprocessor for each region.
>
> On Tuesday, December 3, 2013, Wukang Lin wrote:
>
> > Hi Ted,
> > Thank you for your response. I use hbase-0.94.6-cdh4.4.0.
> > For test, I just have one region in the table, so, it quite sure that
> > the data is inserted in the region. The RPC server is running on each
> > RegionServer, one for one.
> > In my real application, I got the HRegion by HRegion EncodeName which
> > is specialised as a params of the RPC call, the scan operation is carried
> > on, and some old rows were scanned, but some newly inserted rows lost.
> but
> > when i scan the region on client side, such as hbase shell, all rows can
> be
> > got.
> > so, what's the different between the scan operation on RegionServer
> > behind the client side scan and the direct scan on RegionServers side
> using
> > HRegion's getScanner Method?
> >
> > Thank you.
> >
>
Re: Strange Problem on using HRegion's getScanner Method in RegionServer
Posted by Asaf Mesika <as...@gmail.com>.
How do you check in your code you only have one region?
How do you retrieve the HRegion exactly? Since you are running in inside a
coprocessor I presume you have such rpc server per region since Hbase
creates an instance of your coprocessor for each region.
On Tuesday, December 3, 2013, Wukang Lin wrote:
> Hi Ted,
> Thank you for your response. I use hbase-0.94.6-cdh4.4.0.
> For test, I just have one region in the table, so, it quite sure that
> the data is inserted in the region. The RPC server is running on each
> RegionServer, one for one.
> In my real application, I got the HRegion by HRegion EncodeName which
> is specialised as a params of the RPC call, the scan operation is carried
> on, and some old rows were scanned, but some newly inserted rows lost. but
> when i scan the region on client side, such as hbase shell, all rows can be
> got.
> so, what's the different between the scan operation on RegionServer
> behind the client side scan and the direct scan on RegionServers side using
> HRegion's getScanner Method?
>
> Thank you.
>
Re: Strange Problem on using HRegion's getScanner Method in RegionServer
Posted by Wukang Lin <vb...@gmail.com>.
Hi Ted,
Thank you for your response. I use hbase-0.94.6-cdh4.4.0.
For test, I just have one region in the table, so, it quite sure that
the data is inserted in the region. The RPC server is running on each
RegionServer, one for one.
In my real application, I got the HRegion by HRegion EncodeName which
is specialised as a params of the RPC call, the scan operation is carried
on, and some old rows were scanned, but some newly inserted rows lost. but
when i scan the region on client side, such as hbase shell, all rows can be
got.
so, what's the different between the scan operation on RegionServer
behind the client side scan and the direct scan on RegionServers side using
HRegion's getScanner Method?
Thank you.
Re: Strange Problem on using HRegion's getScanner Method in RegionServer
Posted by Ted Yu <yu...@gmail.com>.
bq. make an RPC call of my custom RPC server to scan the HRegion
How many regions do you have ?
How do you direct the RPC call to the region where the data was inserted ?
What version of HBase are you using ?
Cheers
On Tue, Dec 3, 2013 at 8:22 AM, Wukang Lin <vb...@gmail.com> wrote:
> Hi all,
> I got a trouble in using HRegion's RegionScanner in RegionServer.
> In my project, a custom RPC server was started in RegionServer using
> coprocessor mechanism, and it work well. In this RPC server, it receive a
> call from client, and scan the HRegion specialised by the params setted by
> the client like this:
>
> ---------------------------------------------------------------------------------------------------------------------
> HRegion region = this.getHRegionByEncodeName(encodeName);
>
> Scan scan = new Scan();
> RegionScanner scanner = null;
> try {
> scanner = region.getScanner(scan);
> long total = 0;
> boolean hasNext = true;
> List<KeyValue> kvs = new ArrayList<KeyValue>();
> while (hasNext) {
> kvs.clear();
> hasNext = scanner.next(kvs);
> if (kvs.size() > 0) {
> total++;
> }
> } catch (IOException e) {
> LOG.warn("Exception thrown when scanning region '" + regionName
> + "', SKIP IT!", e);
> } finally {
>
> if (scanner != null) {
> try {
> scanner.close();
> } catch (IOException e) {
> e.printStackTrace();
> }
> }
> }
> LOG.info("Total " + total + " rows scanned.");
>
> ---------------------------------------------------------------------------------------------------------------------
> Then, I put a row to a HRegion by HTable interface, and make an RPC call of
> my custom RPC server to scan the HRegion(the code lines above is executed
> in a HRegionServer, a HRegion is scanned), and got nothing, total = 0. But
> if I use client side scan like HTable's scan interface or HBase shell, I
> can get the row I insert just now. I tried to flush the region and do a
> major compaction on the HRegion using hbase shell, Nothing changed.
> So, what's wrong with the code above? how to scan a HRegion in
> HRegionServer instance?
> Any responses is Appreciated!
>