You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Wayne <wa...@gmail.com> on 2011/01/24 21:17:31 UTC
.90 Upgrade Problems
We tried to upgrade to .90 and got 2x the nodes listed and saw none of our
old regions showing up in the counts. We assumed the upgrade was not "easy"
so we just re-formated the HDFS thinking it would fix everything and still
see the same problem. Any suggestions? The duplicate region servers listed
all show no heap or anything. requests=0, regions=0, usedHeap=0, maxHeap=0
Thanks.
RE: Scan Question
Posted by Peter Haidinyak <ph...@local.com>.
Thanks, that fixed my problem. I knew it had to be something I did or didn't do. I had just copied the code from one of my working apps but its tables didn't have any columns with empty values.
Thanks again.
-Pete
-----Original Message-----
From: Ryan Rawson [mailto:ryanobjc@gmail.com]
Sent: Monday, January 24, 2011 4:24 PM
To: user@hbase.apache.org
Subject: Re: Scan Question
if you look at the code, in the top of SingleColumnValueFilter, the
javadoc says:
"To prevent the entire row from being emitted if the column is not
found on a row, use setFilterIfMissing. Otherwise, if the column is
found, the entire row will be emitted only if the value passes. If the
value fails, the row will be filtered out."
The default value of filterIfMissing is FALSE, meaning if a row does
NOT contain the column you are looking for, it is included. If it
does contain the column you are looking for AND the comparison is
true, include else exclude.
that is probably the source of the confusion...
setFilterIfMissing(true). API wise it will be a little annoying,
sorry about that.
-ryan
On Mon, Jan 24, 2011 at 4:15 PM, Peter Haidinyak <ph...@local.com> wrote:
> In a scan I setup a filter as follows...
>
> final FilterList filterList = new FilterList();
> final String botFilterString = getFilter(BOT_VALUE);
> if (botFilterString != null)
> {
> m_logger.info("Bot Value Filter: " + botFilterString);
> final SingleColumnValueFilter botFilter = new SingleColumnValueFilter(TRAFFIC_FAMILY_BYTES,
> BOT_VALUE.getColumnAsBytes(),
> CompareFilter.CompareOp.EQUAL,
> botFilterString.getBytes());
> filterList.addFilter(botFilter);
> }
>
> I then add the filterList to the Scan object. The problem I am having is if I setup a filter that I know should not return any values I still get values returned. What is also interesting is that some of the filters work as expected and some don't. I've had several sets of eyes look at the code and couldn't see any obvious problems.
>
>
> Thanks
>
> -Pete
>
> -----Original Message-----
> From: Ryan Rawson [mailto:ryanobjc@gmail.com]
> Sent: Monday, January 24, 2011 3:58 PM
> To: user@hbase.apache.org
> Subject: Re: Scan Question
>
> CompareFilter is just an abstract base of a series of other filters
> that compare specific components, what exactly are you having a
> problem with?
>
>
>
> On Mon, Jan 24, 2011 at 3:53 PM, Peter Haidinyak <ph...@local.com> wrote:
>> That's the way I thought it should work. When I setup a filter with data that I know shouldn't return any data I still get rows back.
>>
>> -----Original Message-----
>> From: Ryan Rawson [mailto:ryanobjc@gmail.com]
>> Sent: Monday, January 24, 2011 3:32 PM
>> To: user@hbase.apache.org
>> Subject: Re: Scan Question
>>
>> no, if you provide a list of columns in your Scan query and a
>> particular row does not actually contain _that specific column_, then
>> the filter does not see anything and nothing for that row is returned.
>>
>> -ryan
>>
>> On Mon, Jan 24, 2011 at 3:20 PM, Peter Haidinyak <ph...@local.com> wrote:
>>> If I have a table where some of the columns might not have values in each row and I do a scan with a CompareFilter.CompareOp.EQUAL type filter on one of those columns will the scan bring back rows where there is no value in the column I am comparing on?
>>
>>>
>>> Thanks
>>>
>>> -Pete
>>>
>>
>
Re: Scan Question
Posted by Ryan Rawson <ry...@gmail.com>.
if you look at the code, in the top of SingleColumnValueFilter, the
javadoc says:
"To prevent the entire row from being emitted if the column is not
found on a row, use setFilterIfMissing. Otherwise, if the column is
found, the entire row will be emitted only if the value passes. If the
value fails, the row will be filtered out."
The default value of filterIfMissing is FALSE, meaning if a row does
NOT contain the column you are looking for, it is included. If it
does contain the column you are looking for AND the comparison is
true, include else exclude.
that is probably the source of the confusion...
setFilterIfMissing(true). API wise it will be a little annoying,
sorry about that.
-ryan
On Mon, Jan 24, 2011 at 4:15 PM, Peter Haidinyak <ph...@local.com> wrote:
> In a scan I setup a filter as follows...
>
> final FilterList filterList = new FilterList();
> final String botFilterString = getFilter(BOT_VALUE);
> if (botFilterString != null)
> {
> m_logger.info("Bot Value Filter: " + botFilterString);
> final SingleColumnValueFilter botFilter = new SingleColumnValueFilter(TRAFFIC_FAMILY_BYTES,
> BOT_VALUE.getColumnAsBytes(),
> CompareFilter.CompareOp.EQUAL,
> botFilterString.getBytes());
> filterList.addFilter(botFilter);
> }
>
> I then add the filterList to the Scan object. The problem I am having is if I setup a filter that I know should not return any values I still get values returned. What is also interesting is that some of the filters work as expected and some don't. I've had several sets of eyes look at the code and couldn't see any obvious problems.
>
>
> Thanks
>
> -Pete
>
> -----Original Message-----
> From: Ryan Rawson [mailto:ryanobjc@gmail.com]
> Sent: Monday, January 24, 2011 3:58 PM
> To: user@hbase.apache.org
> Subject: Re: Scan Question
>
> CompareFilter is just an abstract base of a series of other filters
> that compare specific components, what exactly are you having a
> problem with?
>
>
>
> On Mon, Jan 24, 2011 at 3:53 PM, Peter Haidinyak <ph...@local.com> wrote:
>> That's the way I thought it should work. When I setup a filter with data that I know shouldn't return any data I still get rows back.
>>
>> -----Original Message-----
>> From: Ryan Rawson [mailto:ryanobjc@gmail.com]
>> Sent: Monday, January 24, 2011 3:32 PM
>> To: user@hbase.apache.org
>> Subject: Re: Scan Question
>>
>> no, if you provide a list of columns in your Scan query and a
>> particular row does not actually contain _that specific column_, then
>> the filter does not see anything and nothing for that row is returned.
>>
>> -ryan
>>
>> On Mon, Jan 24, 2011 at 3:20 PM, Peter Haidinyak <ph...@local.com> wrote:
>>> If I have a table where some of the columns might not have values in each row and I do a scan with a CompareFilter.CompareOp.EQUAL type filter on one of those columns will the scan bring back rows where there is no value in the column I am comparing on?
>>
>>>
>>> Thanks
>>>
>>> -Pete
>>>
>>
>
RE: Scan Question
Posted by Peter Haidinyak <ph...@local.com>.
In a scan I setup a filter as follows...
final FilterList filterList = new FilterList();
final String botFilterString = getFilter(BOT_VALUE);
if (botFilterString != null)
{
m_logger.info("Bot Value Filter: " + botFilterString);
final SingleColumnValueFilter botFilter = new SingleColumnValueFilter(TRAFFIC_FAMILY_BYTES,
BOT_VALUE.getColumnAsBytes(),
CompareFilter.CompareOp.EQUAL,
botFilterString.getBytes());
filterList.addFilter(botFilter);
}
I then add the filterList to the Scan object. The problem I am having is if I setup a filter that I know should not return any values I still get values returned. What is also interesting is that some of the filters work as expected and some don't. I've had several sets of eyes look at the code and couldn't see any obvious problems.
Thanks
-Pete
-----Original Message-----
From: Ryan Rawson [mailto:ryanobjc@gmail.com]
Sent: Monday, January 24, 2011 3:58 PM
To: user@hbase.apache.org
Subject: Re: Scan Question
CompareFilter is just an abstract base of a series of other filters
that compare specific components, what exactly are you having a
problem with?
On Mon, Jan 24, 2011 at 3:53 PM, Peter Haidinyak <ph...@local.com> wrote:
> That's the way I thought it should work. When I setup a filter with data that I know shouldn't return any data I still get rows back.
>
> -----Original Message-----
> From: Ryan Rawson [mailto:ryanobjc@gmail.com]
> Sent: Monday, January 24, 2011 3:32 PM
> To: user@hbase.apache.org
> Subject: Re: Scan Question
>
> no, if you provide a list of columns in your Scan query and a
> particular row does not actually contain _that specific column_, then
> the filter does not see anything and nothing for that row is returned.
>
> -ryan
>
> On Mon, Jan 24, 2011 at 3:20 PM, Peter Haidinyak <ph...@local.com> wrote:
>> If I have a table where some of the columns might not have values in each row and I do a scan with a CompareFilter.CompareOp.EQUAL type filter on one of those columns will the scan bring back rows where there is no value in the column I am comparing on?
>
>>
>> Thanks
>>
>> -Pete
>>
>
Re: Scan Question
Posted by Ryan Rawson <ry...@gmail.com>.
CompareFilter is just an abstract base of a series of other filters
that compare specific components, what exactly are you having a
problem with?
On Mon, Jan 24, 2011 at 3:53 PM, Peter Haidinyak <ph...@local.com> wrote:
> That's the way I thought it should work. When I setup a filter with data that I know shouldn't return any data I still get rows back.
>
> -----Original Message-----
> From: Ryan Rawson [mailto:ryanobjc@gmail.com]
> Sent: Monday, January 24, 2011 3:32 PM
> To: user@hbase.apache.org
> Subject: Re: Scan Question
>
> no, if you provide a list of columns in your Scan query and a
> particular row does not actually contain _that specific column_, then
> the filter does not see anything and nothing for that row is returned.
>
> -ryan
>
> On Mon, Jan 24, 2011 at 3:20 PM, Peter Haidinyak <ph...@local.com> wrote:
>> If I have a table where some of the columns might not have values in each row and I do a scan with a CompareFilter.CompareOp.EQUAL type filter on one of those columns will the scan bring back rows where there is no value in the column I am comparing on?
>
>>
>> Thanks
>>
>> -Pete
>>
>
RE: Scan Question
Posted by Peter Haidinyak <ph...@local.com>.
That's the way I thought it should work. When I setup a filter with data that I know shouldn't return any data I still get rows back.
-----Original Message-----
From: Ryan Rawson [mailto:ryanobjc@gmail.com]
Sent: Monday, January 24, 2011 3:32 PM
To: user@hbase.apache.org
Subject: Re: Scan Question
no, if you provide a list of columns in your Scan query and a
particular row does not actually contain _that specific column_, then
the filter does not see anything and nothing for that row is returned.
-ryan
On Mon, Jan 24, 2011 at 3:20 PM, Peter Haidinyak <ph...@local.com> wrote:
> If I have a table where some of the columns might not have values in each row and I do a scan with a CompareFilter.CompareOp.EQUAL type filter on one of those columns will the scan bring back rows where there is no value in the column I am comparing on?
>
> Thanks
>
> -Pete
>
Re: Scan Question
Posted by Ryan Rawson <ry...@gmail.com>.
no, if you provide a list of columns in your Scan query and a
particular row does not actually contain _that specific column_, then
the filter does not see anything and nothing for that row is returned.
-ryan
On Mon, Jan 24, 2011 at 3:20 PM, Peter Haidinyak <ph...@local.com> wrote:
> If I have a table where some of the columns might not have values in each row and I do a scan with a CompareFilter.CompareOp.EQUAL type filter on one of those columns will the scan bring back rows where there is no value in the column I am comparing on?
>
> Thanks
>
> -Pete
>
Scan Question
Posted by Peter Haidinyak <ph...@local.com>.
If I have a table where some of the columns might not have values in each row and I do a scan with a CompareFilter.CompareOp.EQUAL type filter on one of those columns will the scan bring back rows where there is no value in the column I am comparing on?
Thanks
-Pete
Re: .90 Upgrade Problems
Posted by Wayne <wa...@gmail.com>.
lzop problem...I guess 1 month is enough to forget how hard it is to get a
new install up and running.
Thanks.
On Mon, Jan 24, 2011 at 5:40 PM, Wayne <wa...@gmail.com> wrote:
> We have gotten past duplicate region listings, and now when creating a new
> table get an error in the shell noserverforregionexception. We see all 10
> nodes up and running and we can see the new tables but the UI does not
> respond in showing anything for these new tables and they do not show up in
> the total region count.
>
> Any ideas?
>
> On Mon, Jan 24, 2011 at 4:14 PM, Stack <st...@duboce.net> wrote:
>
>> Its a bug that its needed but yeah, what Ted says.
>> St.Ack
>>
>> On Mon, Jan 24, 2011 at 12:59 PM, Ted Dunning <td...@maprtech.com>
>> wrote:
>> > Yes. It is a requirement.
>> >
>> > On Mon, Jan 24, 2011 at 12:27 PM, Wayne <wa...@gmail.com> wrote:
>> >
>> >> Is reverse dns a requirement with .90? It was not with .89.xxx
>> >>
>> >> On Mon, Jan 24, 2011 at 3:17 PM, Wayne <wa...@gmail.com> wrote:
>> >>
>> >> > We tried to upgrade to .90 and got 2x the nodes listed and saw none
>> of
>> >> our
>> >> > old regions showing up in the counts. We assumed the upgrade was not
>> >> "easy"
>> >> > so we just re-formated the HDFS thinking it would fix everything and
>> >> still
>> >> > see the same problem. Any suggestions? The duplicate region servers
>> >> listed
>> >> > all show no heap or anything. requests=0, regions=0, usedHeap=0,
>> >> maxHeap=0
>> >> >
>> >> > Thanks.
>> >> >
>> >>
>> >
>>
>
>
Re: .90 Upgrade Problems
Posted by Wayne <wa...@gmail.com>.
We have gotten past duplicate region listings, and now when creating a new
table get an error in the shell noserverforregionexception. We see all 10
nodes up and running and we can see the new tables but the UI does not
respond in showing anything for these new tables and they do not show up in
the total region count.
Any ideas?
On Mon, Jan 24, 2011 at 4:14 PM, Stack <st...@duboce.net> wrote:
> Its a bug that its needed but yeah, what Ted says.
> St.Ack
>
> On Mon, Jan 24, 2011 at 12:59 PM, Ted Dunning <td...@maprtech.com>
> wrote:
> > Yes. It is a requirement.
> >
> > On Mon, Jan 24, 2011 at 12:27 PM, Wayne <wa...@gmail.com> wrote:
> >
> >> Is reverse dns a requirement with .90? It was not with .89.xxx
> >>
> >> On Mon, Jan 24, 2011 at 3:17 PM, Wayne <wa...@gmail.com> wrote:
> >>
> >> > We tried to upgrade to .90 and got 2x the nodes listed and saw none of
> >> our
> >> > old regions showing up in the counts. We assumed the upgrade was not
> >> "easy"
> >> > so we just re-formated the HDFS thinking it would fix everything and
> >> still
> >> > see the same problem. Any suggestions? The duplicate region servers
> >> listed
> >> > all show no heap or anything. requests=0, regions=0, usedHeap=0,
> >> maxHeap=0
> >> >
> >> > Thanks.
> >> >
> >>
> >
>
Re: .90 Upgrade Problems
Posted by Stack <st...@duboce.net>.
Its a bug that its needed but yeah, what Ted says.
St.Ack
On Mon, Jan 24, 2011 at 12:59 PM, Ted Dunning <td...@maprtech.com> wrote:
> Yes. It is a requirement.
>
> On Mon, Jan 24, 2011 at 12:27 PM, Wayne <wa...@gmail.com> wrote:
>
>> Is reverse dns a requirement with .90? It was not with .89.xxx
>>
>> On Mon, Jan 24, 2011 at 3:17 PM, Wayne <wa...@gmail.com> wrote:
>>
>> > We tried to upgrade to .90 and got 2x the nodes listed and saw none of
>> our
>> > old regions showing up in the counts. We assumed the upgrade was not
>> "easy"
>> > so we just re-formated the HDFS thinking it would fix everything and
>> still
>> > see the same problem. Any suggestions? The duplicate region servers
>> listed
>> > all show no heap or anything. requests=0, regions=0, usedHeap=0,
>> maxHeap=0
>> >
>> > Thanks.
>> >
>>
>
Re: .90 Upgrade Problems
Posted by Ted Dunning <td...@maprtech.com>.
Yes. It is a requirement.
On Mon, Jan 24, 2011 at 12:27 PM, Wayne <wa...@gmail.com> wrote:
> Is reverse dns a requirement with .90? It was not with .89.xxx
>
> On Mon, Jan 24, 2011 at 3:17 PM, Wayne <wa...@gmail.com> wrote:
>
> > We tried to upgrade to .90 and got 2x the nodes listed and saw none of
> our
> > old regions showing up in the counts. We assumed the upgrade was not
> "easy"
> > so we just re-formated the HDFS thinking it would fix everything and
> still
> > see the same problem. Any suggestions? The duplicate region servers
> listed
> > all show no heap or anything. requests=0, regions=0, usedHeap=0,
> maxHeap=0
> >
> > Thanks.
> >
>
Re: .90 Upgrade Problems
Posted by Wayne <wa...@gmail.com>.
Is reverse dns a requirement with .90? It was not with .89.xxx
On Mon, Jan 24, 2011 at 3:17 PM, Wayne <wa...@gmail.com> wrote:
> We tried to upgrade to .90 and got 2x the nodes listed and saw none of our
> old regions showing up in the counts. We assumed the upgrade was not "easy"
> so we just re-formated the HDFS thinking it would fix everything and still
> see the same problem. Any suggestions? The duplicate region servers listed
> all show no heap or anything. requests=0, regions=0, usedHeap=0, maxHeap=0
>
> Thanks.
>