You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@hbase.apache.org by Przemek <pr...@subrecord.org> on 2008/10/11 08:23:19 UTC

scanning with custom filter problem

Hi all,

Scanning works for me when I don't provide my filter. If I do I have
strange behavior, see the code:

HTable table = new HTable("foo");
for (Object row : table.getScanner(Bytes 				.toByteArrays(new String[]
{ "foo:x" }),					HConstants.EMPTY_START_ROW, System.currentTimeMillis(),
					new MyFilter())) {
	...
}

What I have noticed is that only write() method is invoked on my filter
a couple of times and then an exception is thrown:

ERROR 08:20:19 [org.subrecord.repo.storage.MyScanner (57):scan]: Error
scanning table : any
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to
contact region server 127.0.0.1:2167 for region any,,1223705009718, row
'', but failed after 10 attempts.
Exceptions:
java.io.IOException: Call failed on local exception
java.io.IOException: Call failed on local exception
java.io.IOException: Call failed on local exception

and on the console:

2008-10-11 08:20:19,468 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 6 on 2167, call next(-1, 30) from 127.0.0.1:2303: error:
org.apache.hado
op.hbase.UnknownScannerException: Name: -1
org.apache.hadoop.hbase.UnknownScannerException: Name: -1
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1079)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at
org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:622)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
2008-10-11 08:20:22,500 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
startK

HBase - build from trunk
Setup - 'local' mode

What's wrong with that?

-- 
SubRecord - Enterprise Aspects Stack
http://subrecord.org

Re: Does HBase 0.18.0 not support HQL anymore?

Posted by Jean-Daniel Cryans <jd...@apache.org>.

David,

HQL was replaced in 0.2.0 by a ruby shell. See
http://wiki.apache.org/hadoop/Hbase/Shell

Sorry for the inconveniences

J-D

On Tue, Oct 14, 2008 at 9:00 PM, David Wei <we...@kingsoft.com> wrote:

> We are just trying to use HBase and Hadoop in our production and hoping
> them leverage our data manipulation performance. After deploying in 2 nodes,
> everything at first looks just fine and no error logs when starting Hadoop
> and HBase (all version are 0.18).
> But when we start hbase shell for trying to use HQL, it doesn't work! we
> have use "help" in shell for the commands list and find that it seems HBase
> does not support HQL(0.18) anymore.
>
> Does anybody know about this issue?
>
> Thx!
>
> David
>
>

Does HBase 0.18.0 not support HQL anymore?

Posted by David Wei <we...@kingsoft.com>.

We are just trying to use HBase and Hadoop in our production and hoping 
them leverage our data manipulation performance. After deploying in 2 
nodes, everything at first looks just fine and no error logs when 
starting Hadoop and HBase (all version are 0.18).
But when we start hbase shell for trying to use HQL, it doesn't work! we 
have use "help" in shell for the commands list and find that it seems 
HBase does not support HQL(0.18) anymore.

Does anybody know about this issue?

Thx!

David

RE: scanning and cell histories

Posted by "Jim Kellerman (POWERSET)" <Ji...@microsoft.com>.

Jeremy,

As things currently stand, you are correct. However, HBASE-847
and its sub tasks will permit you to get the history you are looking
for from both scanners and HTable.get. We expect this to be
incorporated into hbase-0.19.0, but cannot make any promises at this
point. I have elevated HBASE-847 to critical, so it is on our
highest priority behind "stuff that is really broken", which are
blockers. This is an important issue for several users, and we will
address it as soon as the current fires have been put out.

---
Jim Kellerman, Powerset (Live Search, Microsoft Corporation)


> -----Original Message-----
> From: Jeremy Pinkham [mailto:jeremy@lotame.com]
> Sent: Tuesday, October 14, 2008 1:52 PM
> To: hbase-user@hadoop.apache.org
> Subject: scanning and cell histories
>
> We've been considering using hbase for some research tasks and were
> particularly interested in the versioning mechanism.  Specifically, we
> thought about using the versioning as a way to get access to when things
> happened for a particular row as opposed to just what a cells value was
> at a point in time... in other words, look at the complete history of a
> cell.  From looking through the API and code it appears we can do this
> if we are accessing specific rows (e.g. HTable.get), but when using a
> scanner (e.g. HTable.getScanner or in a map/reduce job) the complete
> history is not provided.  Can anyone confirm this? or correct me if I am
> missing something
>
> Thanks
>
> jeremy
>
> The information transmitted in this email is intended only for the
> person(s) or entity to which it is addressed and may contain confidential
> and/or privileged material. Any review, retransmission, dissemination or
> other use of, or taking of any action in reliance upon, this information
> by persons or entities other than the intended recipient is prohibited. If
> you received this email in error, please contact the sender and
> permanently delete the email from any computer.
>
>
>
>
> The information transmitted in this email is intended only for the
> person(s) or entity to which it is addressed and may contain confidential
> and/or privileged material. Any review, retransmission, dissemination or
> other use of, or taking of any action in reliance upon, this information
> by persons or entities other than the intended recipient is prohibited. If
> you received this email in error, please contact the sender and
> permanently delete the email from any computer.

scanning and cell histories

Posted by Jeremy Pinkham <je...@lotame.com>.

We've been considering using hbase for some research tasks and were
particularly interested in the versioning mechanism. Specifically, we
thought about using the versioning as a way to get access to when things
happened for a particular row as opposed to just what a cells value was
at a point in time... in other words, look at the complete history of a
cell. From looking through the API and code it appears we can do this
if we are accessing specific rows (e.g. HTable.get), but when using a
scanner (e.g. HTable.getScanner or in a map/reduce job) the complete
history is not provided. Can anyone confirm this? or correct me if I am
missing something

Thanks

jeremy

The information transmitted in this email is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this email in error, please contact the sender and permanently delete the email from any computer.

Re: scanning with custom filter problem

Posted by Przemek <pr...@subrecord.org>.

stack wrote:
> You have a default constructor in your filter?  Want to paste code into
> an issue and describe how to the manufacture the issue so others can
> share your pain?
> St.Ack

Stack, Clint thanks guys! Works like a charm :) Yeah, I missed a default
constructor. Those swallowed exceptions misled me. I should have looked
at the sources. I suppose a local instance is created by my non-default
constructor and remote is deserialized, but instantiated by newInstance
so actually by that missing constructor I didn't have.

Przemek

-- 
SubRecord - Enterprise Aspects Stack
http://subrecord.org

Re: scanning with custom filter problem

Posted by stack <st...@duboce.net>.

You have a default constructor in your filter?  Want to paste code into 
an issue and describe how to the manufacture the issue so others can 
share your pain?
St.Ack

Clint Morgan wrote:
> I'm using a couple of custom filters. I seem to recall a similar initial
> problem with them being serialized, and it seems like the errors were being
> swallowed and not making it to the logs (as Stack mentioned). I think it was
> something silly on my part like a missing default constructor, but can't
> remember the specifics...
>
> Sorry I can't be any help...
> -clint
>
> On Wed, Oct 15, 2008 at 12:57 PM, Przemek <pr...@subrecord.org> wrote:
>
>   
>> stack wrote:
>>     
>>> Can you just add static logging to Filter?  Especially around
>>> construction and in the read/write methods used serializing?  Hopefully
>>> will turn up a clue as to whats going on.
>>> St.Ack
>>>       
>> I had done so before the previous post. That's why I knew only write()
>> was invoked. I've got even a test for the serialization (similar to the
>> ones you have in HBase tests) and it works. Same for the constructor.
>> I'm having hard times with that :( I was testing that on Linux and Win32
>> so it has nothing to do with the environment. Is anybody using custom
>> filters? If I could adapt one of the core filters it would be cool, but
>> I doubt any of those can meet my requirements. Basically I wanna apply
>> some expression on each RowResult so I need to be given a RowResult and
>> decide if I accept that one or not. Of course I need to have that
>> elimination not on the client side. The thing is I wanna use any column
>> and eval my own expression (MVEL, OGNL, etc), so ColumnValueFilter is
>> not suitable here I guess.
>>
>> Przemek
>>
>> --
>> SubRecord - Enterprise Aspects Stack
>> http://subrecord.org
>>
>>     
>
>

Re: scanning with custom filter problem

Posted by Clint Morgan <cl...@gmail.com>.

I'm using a couple of custom filters. I seem to recall a similar initial
problem with them being serialized, and it seems like the errors were being
swallowed and not making it to the logs (as Stack mentioned). I think it was
something silly on my part like a missing default constructor, but can't
remember the specifics...

Sorry I can't be any help...
-clint

On Wed, Oct 15, 2008 at 12:57 PM, Przemek <pr...@subrecord.org> wrote:

> stack wrote:
> > Can you just add static logging to Filter?  Especially around
> > construction and in the read/write methods used serializing?  Hopefully
> > will turn up a clue as to whats going on.
> > St.Ack
>
> I had done so before the previous post. That's why I knew only write()
> was invoked. I've got even a test for the serialization (similar to the
> ones you have in HBase tests) and it works. Same for the constructor.
> I'm having hard times with that :( I was testing that on Linux and Win32
> so it has nothing to do with the environment. Is anybody using custom
> filters? If I could adapt one of the core filters it would be cool, but
> I doubt any of those can meet my requirements. Basically I wanna apply
> some expression on each RowResult so I need to be given a RowResult and
> decide if I accept that one or not. Of course I need to have that
> elimination not on the client side. The thing is I wanna use any column
> and eval my own expression (MVEL, OGNL, etc), so ColumnValueFilter is
> not suitable here I guess.
>
> Przemek
>
> --
> SubRecord - Enterprise Aspects Stack
> http://subrecord.org
>

Re: scanning with custom filter problem

Posted by Przemek <pr...@subrecord.org>.

stack wrote:
> Can you just add static logging to Filter?  Especially around
> construction and in the read/write methods used serializing?  Hopefully
> will turn up a clue as to whats going on.
> St.Ack

I had done so before the previous post. That's why I knew only write()
was invoked. I've got even a test for the serialization (similar to the
ones you have in HBase tests) and it works. Same for the constructor.
I'm having hard times with that :( I was testing that on Linux and Win32
so it has nothing to do with the environment. Is anybody using custom
filters? If I could adapt one of the core filters it would be cool, but
I doubt any of those can meet my requirements. Basically I wanna apply
some expression on each RowResult so I need to be given a RowResult and
decide if I accept that one or not. Of course I need to have that
elimination not on the client side. The thing is I wanna use any column
and eval my own expression (MVEL, OGNL, etc), so ColumnValueFilter is
not suitable here I guess.

Przemek

-- 
SubRecord - Enterprise Aspects Stack
http://subrecord.org

Re: scanning with custom filter problem

Posted by stack <st...@duboce.net>.

Can you just add static logging to Filter?  Especially around 
construction and in the read/write methods used serializing?  Hopefully 
will turn up a clue as to whats going on.
St.Ack

Przemek wrote:
> stack wrote:
>   
>> One thought: your new class is on the CLASSPATH on both sides of the
>> communication -- server and client?
>> St.Ack
>>     
>
> yes. I also created my own test extending HBaseClusterTestCase to have
> everything in one process and the behaviour was similar - call to
> write() and then scanning, many logs on the console, write(), scanning,
>  again and again endlessly.
>
> BTW, is there any test case for testing custom filter in action (that is
> passed to getScanner method) ? Couln't find any.
>
> Przemek
>
>

Re: scanning with custom filter problem

Posted by Przemek <pr...@subrecord.org>.

stack wrote:
> One thought: your new class is on the CLASSPATH on both sides of the
> communication -- server and client?
> St.Ack

yes. I also created my own test extending HBaseClusterTestCase to have
everything in one process and the behaviour was similar - call to
write() and then scanning, many logs on the console, write(), scanning,
 again and again endlessly.

BTW, is there any test case for testing custom filter in action (that is
passed to getScanner method) ? Couln't find any.

Przemek

-- 
SubRecord - Enterprise Aspects Stack
http://subrecord.org

Re: scanning with custom filter problem

Posted by stack <st...@duboce.net>.

One thought: your new class is on the CLASSPATH on both sides of the 
communication -- server and client?
St.Ack


Przemek wrote:
> stack wrote:
>   
>> Debugging something else recently, I saw exceptions like the above
>> because of an NPE in a constructor during deserialization of an object
>> passed over RPC.  It was awkward to figure.  Perhaps your combination of
>> filters is doing something similar?  (You might try adding a test to the
>> TestSerializations that serializes and then deserializes your particular
>> filterset as a first check to ensure the issue is not here.  Just a
>> suggestion)
>>
>> St.Ack
>>     
>
> I have noticed something strange - I can pass new StopRowFilter(...) and
> it works. When I paste that code as a body of MyFilter class it doesn't
> work. Seems the same Java code has different binary format and that from
> HBase jar on my classpath works and what is compiled in Eclipse doesn't.
> Are there any special requirements for that? I suppose it is not
> post-processed by something like AOP/whatever compiler what could make
> any difference.
>
> Przemek
>
>

Re: scanning with custom filter problem

Posted by Przemek <pr...@subrecord.org>.

stack wrote:
> Debugging something else recently, I saw exceptions like the above
> because of an NPE in a constructor during deserialization of an object
> passed over RPC.  It was awkward to figure.  Perhaps your combination of
> filters is doing something similar?  (You might try adding a test to the
> TestSerializations that serializes and then deserializes your particular
> filterset as a first check to ensure the issue is not here.  Just a
> suggestion)
> 
> St.Ack

I have noticed something strange - I can pass new StopRowFilter(...) and
it works. When I paste that code as a body of MyFilter class it doesn't
work. Seems the same Java code has different binary format and that from
HBase jar on my classpath works and what is compiled in Eclipse doesn't.
Are there any special requirements for that? I suppose it is not
post-processed by something like AOP/whatever compiler what could make
any difference.

Przemek

-- 
SubRecord - Enterprise Aspects Stack
http://subrecord.org

Re: scanning with custom filter problem

Posted by stack <st...@duboce.net>.

Przemek wrote:
> ...
> Exceptions:
> java.io.IOException: Call failed on local exception
> java.io.IOException: Call failed on local exception
> java.io.IOException: Call failed on local exception
>
>   
Debugging something else recently, I saw exceptions like the above 
because of an NPE in a constructor during deserialization of an object 
passed over RPC.  It was awkward to figure.  Perhaps your combination of 
filters is doing something similar?  (You might try adding a test to the 
TestSerializations that serializes and then deserializes your particular 
filterset as a first check to ensure the issue is not here.  Just a 
suggestion)

St.Ack