You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Shuja Rehman <sh...@gmail.com> on 2011/01/20 09:22:24 UTC
Scan (Start Row, End Row) vs Scan (Row)
Hi
Consider the following scenario.
Row Key Format = DATETIME_VAR1_VAR2 (where var1 and var2 have fixed
lengths)
and example data could be
20100809041500_abc_xyz
20100809041500_abc_xyw
20100809041500_abc_xyc
*20100809041500_abd_xyz*
20100809041500_abd_xyw
20100809041500_abf_xyz
...
Now if i want to get the rows which only have this row key
20100809041500_abd then is there anyway to achieve through scan without
using filter because if i use filter scan(startrow, filter) where
startrow="20100809041500_abd" then it will scan whole table from start key
to end of table. i want to just scan that part of table which i require. So
if there is any method like this
scan(row) where row ="20100809041500_abd" and it just return the following
results
20100809041500_abd_xyz
20100809041500_abd_xyw
Kindly let me know whether it is achievable or not?
thnx
--
Regards
Shuja-ur-Rehman Baig
<http://pk.linkedin.com/in/shujamughal>
Re: Data loss on clean RS shutdown without WAL?
Posted by Jean-Daniel Cryans <jd...@apache.org>.
The master splits the logs per region before reassigning them. The log
splits are put directly in the region's folder so that when a region
server opens a region that comes from a dead server, it looks for
those files and processes them first before opening the region (to
ensure consistency).
Splitting logs can be slow when you have tons of them, since only one
machine does it, so work is being done to parallelize it just like in
Bigtable: https://issues.apache.org/jira/browse/hbase-1364
J-D
On Sun, Jan 23, 2011 at 11:08 AM, M. C. Srivas <mc...@gmail.com> wrote:
> Hey JD,
>
> when the RS dies, the regions that it was serving are spread out amongst
> the rest of the RS's, correct? But isn't the WAL a per-RS thingy rather
> than a per-region thingy? How do the other RS's then recover the regions
> alloted to them? Do they skip over log-records in the dead RS's WAL that do
> not belong to the regions not allocated to them?
>
> Also, how is the dead RS's WAL garbage-collected?
>
> thanks,
> Srivas.
>
>
Re: Data loss on clean RS shutdown without WAL?
Posted by "M. C. Srivas" <mc...@gmail.com>.
Hey JD,
when the RS dies, the regions that it was serving are spread out amongst
the rest of the RS's, correct? But isn't the WAL a per-RS thingy rather
than a per-region thingy? How do the other RS's then recover the regions
alloted to them? Do they skip over log-records in the dead RS's WAL that do
not belong to the regions not allocated to them?
Also, how is the dead RS's WAL garbage-collected?
thanks,
Srivas.
On Fri, Jan 21, 2011 at 9:32 AM, Jean-Daniel Cryans <jd...@apache.org>wrote:
> If the region servers gets YouAreDeadException, it does an "abort" and
> won't flush the data since another region server could already be
> serving the region. If you're not writing to the WAL, then yes it's
> data loss.
>
> Not sure what you mean by "shuts down cleanly" in your case, if you
> see a log that starts with "Aborting region server" then it's not
> really "clean".
>
> J-D
>
> On Fri, Jan 21, 2011 at 2:38 AM, Friso van Vollenhoven
> <fv...@xebia.com> wrote:
> > Hi all,
> >
> > Question: when a regionserver shuts down cleanly after a
> YouAreDeadException and the master nicely reassigns the regions, will you
> loose any data that was written to the memstore of the dead RS when not
> using WAL?
> >
> > There was no hard crash and not a single error in any of the logs (except
> for the FATAL: YouAreDeadException). The RS lost its zookeeper session after
> a timeout, probably GC combined with some other starvation on heavy load. I
> think the memstore flushes on shutdown, but I am not entirely sure what
> happens in the situation where regions are already opened by other
> regionservers when the dying executes the shutdown code. Can I assume that
> the RS that gets reassigned a region creates a new HFile and that this will
> be compacted together with the one left by the dead RS at the next
> compaction run?
> >
> >
> > Thanks,
> > Friso
> >
> >
>
RE: Data loss on clean RS shutdown without WAL?
Posted by Friso van Vollenhoven <fv...@xebia.com>.
Thanks J-D.
With cleanly, I meant that it did not produce any errors other than the YouAreDeadException. When a very long pause causes this I often also see HDFS client related errors because of leases expiring, but this time that was not the case. It's not "cleanly" in the sense that I issued the shutdown myself.
Friso
________________________________________
Van: jdcryans@gmail.com [jdcryans@gmail.com] namens Jean-Daniel Cryans [jdcryans@apache.org]
Verzonden: vrijdag 21 januari 2011 18:32
Aan: user@hbase.apache.org
Onderwerp: Re: Data loss on clean RS shutdown without WAL?
If the region servers gets YouAreDeadException, it does an "abort" and
won't flush the data since another region server could already be
serving the region. If you're not writing to the WAL, then yes it's
data loss.
Not sure what you mean by "shuts down cleanly" in your case, if you
see a log that starts with "Aborting region server" then it's not
really "clean".
J-D
On Fri, Jan 21, 2011 at 2:38 AM, Friso van Vollenhoven
<fv...@xebia.com> wrote:
> Hi all,
>
> Question: when a regionserver shuts down cleanly after a YouAreDeadException and the master nicely reassigns the regions, will you loose any data that was written to the memstore of the dead RS when not using WAL?
>
> There was no hard crash and not a single error in any of the logs (except for the FATAL: YouAreDeadException). The RS lost its zookeeper session after a timeout, probably GC combined with some other starvation on heavy load. I think the memstore flushes on shutdown, but I am not entirely sure what happens in the situation where regions are already opened by other regionservers when the dying executes the shutdown code. Can I assume that the RS that gets reassigned a region creates a new HFile and that this will be compacted together with the one left by the dead RS at the next compaction run?
>
>
> Thanks,
> Friso
>
>
Re: Data loss on clean RS shutdown without WAL?
Posted by Jean-Daniel Cryans <jd...@apache.org>.
If the region servers gets YouAreDeadException, it does an "abort" and
won't flush the data since another region server could already be
serving the region. If you're not writing to the WAL, then yes it's
data loss.
Not sure what you mean by "shuts down cleanly" in your case, if you
see a log that starts with "Aborting region server" then it's not
really "clean".
J-D
On Fri, Jan 21, 2011 at 2:38 AM, Friso van Vollenhoven
<fv...@xebia.com> wrote:
> Hi all,
>
> Question: when a regionserver shuts down cleanly after a YouAreDeadException and the master nicely reassigns the regions, will you loose any data that was written to the memstore of the dead RS when not using WAL?
>
> There was no hard crash and not a single error in any of the logs (except for the FATAL: YouAreDeadException). The RS lost its zookeeper session after a timeout, probably GC combined with some other starvation on heavy load. I think the memstore flushes on shutdown, but I am not entirely sure what happens in the situation where regions are already opened by other regionservers when the dying executes the shutdown code. Can I assume that the RS that gets reassigned a region creates a new HFile and that this will be compacted together with the one left by the dead RS at the next compaction run?
>
>
> Thanks,
> Friso
>
>
Data loss on clean RS shutdown without WAL?
Posted by Friso van Vollenhoven <fv...@xebia.com>.
Hi all,
Question: when a regionserver shuts down cleanly after a YouAreDeadException and the master nicely reassigns the regions, will you loose any data that was written to the memstore of the dead RS when not using WAL?
There was no hard crash and not a single error in any of the logs (except for the FATAL: YouAreDeadException). The RS lost its zookeeper session after a timeout, probably GC combined with some other starvation on heavy load. I think the memstore flushes on shutdown, but I am not entirely sure what happens in the situation where regions are already opened by other regionservers when the dying executes the shutdown code. Can I assume that the RS that gets reassigned a region creates a new HFile and that this will be compacted together with the one left by the dead RS at the next compaction run?
Thanks,
Friso
RE: Scan (Start Row, End Row) vs Scan (Row)
Posted by Peter Haidinyak <ph...@local.com>.
Thanks
-----Original Message-----
From: Jonathan Gray [mailto:jgray@fb.com]
Sent: Thursday, January 20, 2011 8:44 AM
To: user@hbase.apache.org
Subject: RE: Scan (Start Row, End Row) vs Scan (Row)
Yes.
> -----Original Message-----
> From: Peter Haidinyak [mailto:phaidinyak@local.com]
> Sent: Thursday, January 20, 2011 8:41 AM
> To: user@hbase.apache.org
> Subject: RE: Scan (Start Row, End Row) vs Scan (Row)
>
> Question, does HBase stop scanning after it hits the end row? I thought it
> does.
>
> Thanks
>
> -Pete
>
> -----Original Message-----
> From: Jonathan Gray [mailto:jgray@fb.com]
> Sent: Thursday, January 20, 2011 8:09 AM
> To: user@hbase.apache.org
> Subject: RE: Scan (Start Row, End Row) vs Scan (Row)
>
> The best way to do this is as Friso describes, using the existing stopRow
> parameter in Scan.
>
> There is another way to do it with startRow + a filter. There is a PrefixFilter
> which could be used here. Looking at the code, it seems as though the
> PrefixFilter does an early out and stops the scan once passed the prefix.
>
> If not, you can wrap any filter in a WhileMatchFilter. That wrapping filter will
> make it so once the underlying filter fails once, all further things will fail and
> the scan will early out.
>
> JG
>
> > -----Original Message-----
> > From: Friso van Vollenhoven [mailto:fvanvollenhoven@xebia.com]
> > Sent: Thursday, January 20, 2011 12:45 AM
> > To: <us...@hbase.apache.org>
> > Subject: Re: Scan (Start Row, End Row) vs Scan (Row)
> >
> > Performing a scan with
> >
> > start row = 20100809041500_abd
> > end row = 20100809041500_abe
> >
> > will give you just that. The end row is exclusive, so it will only
> > return rows with VAR1 = abd. You need to compute the 'abe' yourself,
> > though (which is basically taking 'abd' and increasing the right most
> > byte by 1 unless it's at max byte value, then set it to 0 and increase
> > the byte left to that by 1, etc.). There is no scan method that has 'starts
> with' semantics, AFAIK.
> >
> > See here:
> >
> http://hbase.apache.org/docs/r0.89.20100924/apidocs/org/apache/hadoop/
> > hbase/client/Scan.html#Scan(byte[],
> >
> byte[])<http://hbase.apache.org/docs/r0.89.20100924/apidocs/org/apache
> > / hadoop/hbase/client/Scan.html#Scan(byte%5B%5D,%20byte%5B%5D)>
> >
> >
> > Friso
> >
> >
> >
> >
> > On 20 jan 2011, at 09:22, Shuja Rehman wrote:
> >
> > Hi
> > Consider the following scenario.
> >
> > Row Key Format = DATETIME_VAR1_VAR2 (where var1 and var2 have
> fixed
> > lengths)
> >
> > and example data could be
> >
> > 20100809041500_abc_xyz
> > 20100809041500_abc_xyw
> > 20100809041500_abc_xyc
> > *20100809041500_abd_xyz*
> > 20100809041500_abd_xyw
> > 20100809041500_abf_xyz
> > ...
> >
> > Now if i want to get the rows which only have this row key
> > 20100809041500_abd then is there anyway to achieve through scan
> > without using filter because if i use filter scan(startrow, filter)
> > where startrow="20100809041500_abd" then it will scan whole table from
> > start key to end of table. i want to just scan that part of table
> > which i require. So if there is any method like this
> >
> > scan(row) where row ="20100809041500_abd" and it just return the
> > following results
> >
> > 20100809041500_abd_xyz
> > 20100809041500_abd_xyw
> >
> > Kindly let me know whether it is achievable or not?
> > thnx
> > --
> > Regards
> > Shuja-ur-Rehman Baig
> > <http://pk.linkedin.com/in/shujamughal>
RE: Scan (Start Row, End Row) vs Scan (Row)
Posted by Jonathan Gray <jg...@fb.com>.
Yes.
> -----Original Message-----
> From: Peter Haidinyak [mailto:phaidinyak@local.com]
> Sent: Thursday, January 20, 2011 8:41 AM
> To: user@hbase.apache.org
> Subject: RE: Scan (Start Row, End Row) vs Scan (Row)
>
> Question, does HBase stop scanning after it hits the end row? I thought it
> does.
>
> Thanks
>
> -Pete
>
> -----Original Message-----
> From: Jonathan Gray [mailto:jgray@fb.com]
> Sent: Thursday, January 20, 2011 8:09 AM
> To: user@hbase.apache.org
> Subject: RE: Scan (Start Row, End Row) vs Scan (Row)
>
> The best way to do this is as Friso describes, using the existing stopRow
> parameter in Scan.
>
> There is another way to do it with startRow + a filter. There is a PrefixFilter
> which could be used here. Looking at the code, it seems as though the
> PrefixFilter does an early out and stops the scan once passed the prefix.
>
> If not, you can wrap any filter in a WhileMatchFilter. That wrapping filter will
> make it so once the underlying filter fails once, all further things will fail and
> the scan will early out.
>
> JG
>
> > -----Original Message-----
> > From: Friso van Vollenhoven [mailto:fvanvollenhoven@xebia.com]
> > Sent: Thursday, January 20, 2011 12:45 AM
> > To: <us...@hbase.apache.org>
> > Subject: Re: Scan (Start Row, End Row) vs Scan (Row)
> >
> > Performing a scan with
> >
> > start row = 20100809041500_abd
> > end row = 20100809041500_abe
> >
> > will give you just that. The end row is exclusive, so it will only
> > return rows with VAR1 = abd. You need to compute the 'abe' yourself,
> > though (which is basically taking 'abd' and increasing the right most
> > byte by 1 unless it's at max byte value, then set it to 0 and increase
> > the byte left to that by 1, etc.). There is no scan method that has 'starts
> with' semantics, AFAIK.
> >
> > See here:
> >
> http://hbase.apache.org/docs/r0.89.20100924/apidocs/org/apache/hadoop/
> > hbase/client/Scan.html#Scan(byte[],
> >
> byte[])<http://hbase.apache.org/docs/r0.89.20100924/apidocs/org/apache
> > / hadoop/hbase/client/Scan.html#Scan(byte%5B%5D,%20byte%5B%5D)>
> >
> >
> > Friso
> >
> >
> >
> >
> > On 20 jan 2011, at 09:22, Shuja Rehman wrote:
> >
> > Hi
> > Consider the following scenario.
> >
> > Row Key Format = DATETIME_VAR1_VAR2 (where var1 and var2 have
> fixed
> > lengths)
> >
> > and example data could be
> >
> > 20100809041500_abc_xyz
> > 20100809041500_abc_xyw
> > 20100809041500_abc_xyc
> > *20100809041500_abd_xyz*
> > 20100809041500_abd_xyw
> > 20100809041500_abf_xyz
> > ...
> >
> > Now if i want to get the rows which only have this row key
> > 20100809041500_abd then is there anyway to achieve through scan
> > without using filter because if i use filter scan(startrow, filter)
> > where startrow="20100809041500_abd" then it will scan whole table from
> > start key to end of table. i want to just scan that part of table
> > which i require. So if there is any method like this
> >
> > scan(row) where row ="20100809041500_abd" and it just return the
> > following results
> >
> > 20100809041500_abd_xyz
> > 20100809041500_abd_xyw
> >
> > Kindly let me know whether it is achievable or not?
> > thnx
> > --
> > Regards
> > Shuja-ur-Rehman Baig
> > <http://pk.linkedin.com/in/shujamughal>
RE: Scan (Start Row, End Row) vs Scan (Row)
Posted by Peter Haidinyak <ph...@local.com>.
Question, does HBase stop scanning after it hits the end row? I thought it does.
Thanks
-Pete
-----Original Message-----
From: Jonathan Gray [mailto:jgray@fb.com]
Sent: Thursday, January 20, 2011 8:09 AM
To: user@hbase.apache.org
Subject: RE: Scan (Start Row, End Row) vs Scan (Row)
The best way to do this is as Friso describes, using the existing stopRow parameter in Scan.
There is another way to do it with startRow + a filter. There is a PrefixFilter which could be used here. Looking at the code, it seems as though the PrefixFilter does an early out and stops the scan once passed the prefix.
If not, you can wrap any filter in a WhileMatchFilter. That wrapping filter will make it so once the underlying filter fails once, all further things will fail and the scan will early out.
JG
> -----Original Message-----
> From: Friso van Vollenhoven [mailto:fvanvollenhoven@xebia.com]
> Sent: Thursday, January 20, 2011 12:45 AM
> To: <us...@hbase.apache.org>
> Subject: Re: Scan (Start Row, End Row) vs Scan (Row)
>
> Performing a scan with
>
> start row = 20100809041500_abd
> end row = 20100809041500_abe
>
> will give you just that. The end row is exclusive, so it will only return rows
> with VAR1 = abd. You need to compute the 'abe' yourself, though (which is
> basically taking 'abd' and increasing the right most byte by 1 unless it's at max
> byte value, then set it to 0 and increase the byte left to that by 1, etc.). There
> is no scan method that has 'starts with' semantics, AFAIK.
>
> See here:
> http://hbase.apache.org/docs/r0.89.20100924/apidocs/org/apache/hadoop/
> hbase/client/Scan.html#Scan(byte[],
> byte[])<http://hbase.apache.org/docs/r0.89.20100924/apidocs/org/apache/
> hadoop/hbase/client/Scan.html#Scan(byte%5B%5D,%20byte%5B%5D)>
>
>
> Friso
>
>
>
>
> On 20 jan 2011, at 09:22, Shuja Rehman wrote:
>
> Hi
> Consider the following scenario.
>
> Row Key Format = DATETIME_VAR1_VAR2 (where var1 and var2 have fixed
> lengths)
>
> and example data could be
>
> 20100809041500_abc_xyz
> 20100809041500_abc_xyw
> 20100809041500_abc_xyc
> *20100809041500_abd_xyz*
> 20100809041500_abd_xyw
> 20100809041500_abf_xyz
> ...
>
> Now if i want to get the rows which only have this row key
> 20100809041500_abd then is there anyway to achieve through scan without
> using filter because if i use filter scan(startrow, filter) where
> startrow="20100809041500_abd" then it will scan whole table from start key
> to end of table. i want to just scan that part of table which i require. So if
> there is any method like this
>
> scan(row) where row ="20100809041500_abd" and it just return the
> following results
>
> 20100809041500_abd_xyz
> 20100809041500_abd_xyw
>
> Kindly let me know whether it is achievable or not?
> thnx
> --
> Regards
> Shuja-ur-Rehman Baig
> <http://pk.linkedin.com/in/shujamughal>
RE: Scan (Start Row, End Row) vs Scan (Row)
Posted by Jonathan Gray <jg...@fb.com>.
The best way to do this is as Friso describes, using the existing stopRow parameter in Scan.
There is another way to do it with startRow + a filter. There is a PrefixFilter which could be used here. Looking at the code, it seems as though the PrefixFilter does an early out and stops the scan once passed the prefix.
If not, you can wrap any filter in a WhileMatchFilter. That wrapping filter will make it so once the underlying filter fails once, all further things will fail and the scan will early out.
JG
> -----Original Message-----
> From: Friso van Vollenhoven [mailto:fvanvollenhoven@xebia.com]
> Sent: Thursday, January 20, 2011 12:45 AM
> To: <us...@hbase.apache.org>
> Subject: Re: Scan (Start Row, End Row) vs Scan (Row)
>
> Performing a scan with
>
> start row = 20100809041500_abd
> end row = 20100809041500_abe
>
> will give you just that. The end row is exclusive, so it will only return rows
> with VAR1 = abd. You need to compute the 'abe' yourself, though (which is
> basically taking 'abd' and increasing the right most byte by 1 unless it's at max
> byte value, then set it to 0 and increase the byte left to that by 1, etc.). There
> is no scan method that has 'starts with' semantics, AFAIK.
>
> See here:
> http://hbase.apache.org/docs/r0.89.20100924/apidocs/org/apache/hadoop/
> hbase/client/Scan.html#Scan(byte[],
> byte[])<http://hbase.apache.org/docs/r0.89.20100924/apidocs/org/apache/
> hadoop/hbase/client/Scan.html#Scan(byte%5B%5D,%20byte%5B%5D)>
>
>
> Friso
>
>
>
>
> On 20 jan 2011, at 09:22, Shuja Rehman wrote:
>
> Hi
> Consider the following scenario.
>
> Row Key Format = DATETIME_VAR1_VAR2 (where var1 and var2 have fixed
> lengths)
>
> and example data could be
>
> 20100809041500_abc_xyz
> 20100809041500_abc_xyw
> 20100809041500_abc_xyc
> *20100809041500_abd_xyz*
> 20100809041500_abd_xyw
> 20100809041500_abf_xyz
> ...
>
> Now if i want to get the rows which only have this row key
> 20100809041500_abd then is there anyway to achieve through scan without
> using filter because if i use filter scan(startrow, filter) where
> startrow="20100809041500_abd" then it will scan whole table from start key
> to end of table. i want to just scan that part of table which i require. So if
> there is any method like this
>
> scan(row) where row ="20100809041500_abd" and it just return the
> following results
>
> 20100809041500_abd_xyz
> 20100809041500_abd_xyw
>
> Kindly let me know whether it is achievable or not?
> thnx
> --
> Regards
> Shuja-ur-Rehman Baig
> <http://pk.linkedin.com/in/shujamughal>
Re: Scan (Start Row, End Row) vs Scan (Row)
Posted by Friso van Vollenhoven <fv...@xebia.com>.
Performing a scan with
start row = 20100809041500_abd
end row = 20100809041500_abe
will give you just that. The end row is exclusive, so it will only return rows with VAR1 = abd. You need to compute the 'abe' yourself, though (which is basically taking 'abd' and increasing the right most byte by 1 unless it's at max byte value, then set it to 0 and increase the byte left to that by 1, etc.). There is no scan method that has 'starts with' semantics, AFAIK.
See here: http://hbase.apache.org/docs/r0.89.20100924/apidocs/org/apache/hadoop/hbase/client/Scan.html#Scan(byte[], byte[])<http://hbase.apache.org/docs/r0.89.20100924/apidocs/org/apache/hadoop/hbase/client/Scan.html#Scan(byte%5B%5D,%20byte%5B%5D)>
Friso
On 20 jan 2011, at 09:22, Shuja Rehman wrote:
Hi
Consider the following scenario.
Row Key Format = DATETIME_VAR1_VAR2 (where var1 and var2 have fixed
lengths)
and example data could be
20100809041500_abc_xyz
20100809041500_abc_xyw
20100809041500_abc_xyc
*20100809041500_abd_xyz*
20100809041500_abd_xyw
20100809041500_abf_xyz
...
Now if i want to get the rows which only have this row key
20100809041500_abd then is there anyway to achieve through scan without
using filter because if i use filter scan(startrow, filter) where
startrow="20100809041500_abd" then it will scan whole table from start key
to end of table. i want to just scan that part of table which i require. So
if there is any method like this
scan(row) where row ="20100809041500_abd" and it just return the following
results
20100809041500_abd_xyz
20100809041500_abd_xyw
Kindly let me know whether it is achievable or not?
thnx
--
Regards
Shuja-ur-Rehman Baig
<http://pk.linkedin.com/in/shujamughal>