You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Marc Hoppins <ma...@eset.sk> on 2020/12/15 15:46:32 UTC
Removal of table rows.
Hi all,
I am an HBASE newbie so I apologise if I am being repetitious.
Apologies also if this is not the right group. Am not sure if this may be more suited to 'dev' list. However,
A problem question and a technical/wishlist question.
Problem:
I have inherited a problem with an HBASE table. The original issue may have erupted due to a network outage. A table has 48 region in transition operations, stuck that way for several weeks. A previous attempt to fix things with hbck failed. An attempt to DISABLE then DROP the table also failed. The four or five attempts to work the table ALSO now had stuck procedures. Subsequent DFS and ZOO operations left the situation where there was no data and no real table: just a 6K file in an empty structure.
When I got to the problem my knowledge of HBASE was nil. It is little better than that now but anyway...
Fortunately for me this is a testing/dev cluster. The 'owner' was content that the table can be removed - and appeared to have already been done...of a kind.
Reading and reading of others' similar issues lead me to the point I also was going to clean the HDFS data and ZK data for this table. I shut down HBASE, cleaned HDFS and ZK node data, deleted the masterprocwals and restarted HBASE.
When all came up I was happy to see that the affected table appeared nowhere and that the procedures had all disappeared.
However, when I hopped to hbase master, even though no table of that name existed, 48 regions were still in transition. Further research steered me toward hbase:meta and sure enough, the references to the RITs lived happily among other data for other tables.
The solution offered by hbase-operator-tools - extraRegionsinMeta - offered hope. Once again, however, another problem has surfaced: this tools command for extra regions is incompatible with the hbase version we are running.
So...
How can I remove the references to namespace:kaput_table from hbase:meta?
Technical:
Is there to be any implementation of such a fix within HBASe itself where table manipulation can be performed by Eg.,
delete hbase:meta namespace
delete hbase:meta namespace:table
or even
scan hbase:meta filter = namespace:table | deleterow
or some such?
Thanks
M
Re: Removal of table rows.
Posted by Wellington Chevreuil <we...@gmail.com>.
>
> So, once again, I ask: is a method to remove rows via namespace or table
> name part of any development plan? Or part of hbase operator tools
> development?
For normal operations, this could be done via "hbase shell" *drop* command.
In this specific case of yours, you can use it for removing the remaining
cells with table state.
As for the fix: apparently all is well. Balancing has restarted, no RITs
> anywhere.
>
Glad to hear that!
Can anyone clarify what other operations are carried out other than
> removing meta rows? Eg., is zookeeper shell called to deal with any
> outstanding items? I saw zk information on screen when I ran tools.
>
ZK is used for orchestration of several internal hbase activities that are
carried in a distributed way. Sometimes, some of these actions can get into
an inconsistent state, causing it to fail to progress, so cleaning out ZK
info entirely basically does a *reset *of hbase internal admin state. In
general, it's safe to clean out hbase info in ZK, but there are few
features that may break if you do so, such as replication.
The meta table is a different story, in short, it's much less safe to
manipulate it manually. It works as an index of all tables in hbase,
containing info about which RegionServer is hosting which portions of
different tables, so that clients can be redirected accordingly when
querying specific tables. Mistakes while manipulating it could cause
temporary/indefinite partial/total user data unavailability, so it's only
advisable to do so if you have a deep understanding of the implications.
Em sex., 18 de dez. de 2020 às 07:55, Marc Hoppins <ma...@eset.sk>
escreveu:
> As for the fix: apparently all is well. Balancing has restarted, no RITs
> anywhere.
>
> Can anyone clarify what other operations are carried out other than
> removing meta rows? Eg., is zookeeper shell called to deal with any
> outstanding items? I saw zk information on screen when I ran tools.
>
> -----Original Message-----
> From: Wellington Chevreuil <we...@gmail.com>
> Sent: Thursday, December 17, 2020 8:15 PM
> To: Hbase-User <us...@hbase.apache.org>
> Subject: Re: Removal of table rows.
>
> EXTERNAL
>
> >
> > alfa:extgen,F,1608197544264.ba22f6113a0bb9520cee1f7b30050fa7.
> > column=info:state, timestamp=1608197544904, value=OPEN
> > alfa:rfilenameext column=table:state, timestamp=1604493139455,
> > value=\x08\x00
> > alfa:rfiles column=table:state, timestamp=1602600776355,
> > value=\x08\x00
> >
> These are all table state rows, not table regions rows, so
> extraRegionsInMeta would not remove it.
>
>
> > I did restart 'instances' on master(s) so I was surprised to see a
> > lingering row.
> >
> The restart should have cleared out the RITs, but not the table state rows.
> Can you confirm if you still see those regions in Master UI even after
> restarting masters?
>
> Em qui., 17 de dez. de 2020 às 15:19, <sc...@comcast.net> escreveu:
>
> > Marc, what type of data are loading into hbase? curious..
> > > On 12/17/2020 8:46 AM Marc Hoppins <ma...@eset.sk> wrote:
> > >
> > >
> > > I did restart 'instances' on master(s) so I was surprised to see a
> > lingering row.
> > >
> > > -----Original Message-----
> > > From: Wellington Chevreuil <we...@gmail.com>
> > > Sent: Thursday, December 17, 2020 1:34 PM
> > > To: Hbase-User <us...@hbase.apache.org>
> > > Subject: Re: Removal of table rows.
> > >
> > > EXTERNAL
> > >
> > > Yes, depending on hbade version, you would need a master restart to
> > effectively clean master cache.
> > >
> > > On Thu, 17 Dec 2020, 10:12 Marc Hoppins, <ma...@eset.sk> wrote:
> > >
> > > > OK. I had a meandering circuit to get a version of operator-tools
> > > > built and running.
> > > >
> > > > After running it on the meta table
> > > >
> > > > sudo hbase hbck -j /tmp/hbase-hbck2-1.1.0-SNAPSHOT.jar
> > > > extraRegionsInMeta alfa:rfilenameext --fix
> > > >
> > > > I have ended up with:-
> > > >
> > > > alfa:extgen,F,1608197544264.ba22f6113a0bb9520cee1f7b30050fa7.
> > > > column=info:state, timestamp=1608197544904, value=OPEN
> > > > alfa:rfilenameext column=table:state, timestamp=1604493139455,
> > > > value=\x08\x00
> > > > alfa:rfiles column=table:state, timestamp=1602600776355,
> > > > value=\x08\x00
> > > >
> > > > I still have one row referring to the missing table. Do I need to
> > > > restart hbase service to remove this or will it vanish at some time?
> > > >
> > > > HBASE on the master still shows the 48 regions in transition when
> > > > I open the interface. I assume this is because the service has
> > > > not been
> > restarted.
> > > >
> > > > M
> > > >
> > > > -----Original Message-----
> > > > From: Wellington Chevreuil <we...@gmail.com>
> > > > Sent: Wednesday, December 16, 2020 3:39 PM
> > > > To: Hbase-User <us...@hbase.apache.org>
> > > > Subject: Re: Removal of table rows.
> > > >
> > > > EXTERNAL
> > > >
> > > > >
> > > > > Do I build hbck2-tools on similar OS & java version? I have
> > > > > been informed we have 'maven' installed on one host on a
> > > > > cluster, which is centos and
> > > > > (probably) a different java, and the build itself is needed for
> > Ubuntu16.
> > > >
> > > > I would stick to the same java major version. OS wise, this module
> > > > doesn't rely on any OS native call, AFAIK, so should work fine
> > > > between these different versions.
> > > >
> > > > Em qua., 16 de dez. de 2020 às 13:30, Marc Hoppins
> > > > <ma...@eset.sk>
> > > > escreveu:
> > > >
> > > > > Thanks. Once again, a newbie in this regard but,
> > > > >
> > > > > Do I build hbck2-tools on similar OS & java version? I have
> > > > > been informed we have 'maven' installed on one host on a
> > > > > cluster, which is centos and
> > > > > (probably) a different java, and the build itself is needed for
> > Ubuntu16.
> > > > >
> > > > > -----Original Message-----
> > > > > From: Wellington Chevreuil <we...@gmail.com>
> > > > > Sent: Wednesday, December 16, 2020 10:08 AM
> > > > > To: Hbase-User <us...@hbase.apache.org>
> > > > > Subject: Re: Removal of table rows.
> > > > >
> > > > > EXTERNAL
> > > > >
> > > > > >
> > > > > > Hbase:meta has info for one table region which is NOT on a
> master.
> > > > > > Is that correct? I would have expected all meta info to be
> > > > > > stored on a
> > > > > master.
> > > > > >
> > > > > meta table is an "hbase system table" that has info about which
> > > > > regions are assigned to which region servers in your cluster. As
> > > > > any user table, it also has a region, and this region must be
> > > > > hosted on a region server in the cluster. Master is just
> > > > > responsible for the coordination of regions and some other
> housekeeping actions.
> > > > >
> > > > > As I state below, a simple method to remove everything connected
> > > > > to
> > > > > > alfa:rfilenameext would seem the easiest of tasks as the
> > > > > > namespace:name consistently appears on every relevant row.
> > > > >
> > > > > Looks like hbck2 *extraRegionsinMeta *is what you need here,
> > > > > and it should work with hbase 2.1. It's not available on hbck2
> > > > > 1.0 release, but you can build a new hbck2 jar out of current
> > > > > master branch, and that would give you an hbck2 with
> > > > > extraRegionsinMeta. You can do it by cloning the repo below,
> > > > > then do a *mvn install* from the main
> > module:
> > > > >
> > > > > https://github.com/apache/hbase-operator-tools.git
> > > > >
> > > > > Em qua., 16 de dez. de 2020 às 08:02, Marc Hoppins
> > > > > <ma...@eset.sk>
> > > > > escreveu:
> > > > >
> > > > > > Hbase is 2.1.0 (via Cloudera CDH 6.3.2)
> > > > > >
> > > > > > The UI page for the master shows NO report links/tabs.
> > > > > >
> > > > > > Hbase:meta has info for one table region which is NOT on a
> master.
> > > > > > Is that correct? I would have expected all meta info to be
> > > > > > stored on a
> > > > > master.
> > > > > >
> > > > > > Name Region Server
> > > > > > Read Requests Write
> Requests
> > > > > > Num.Storefiles MemSize Locality
> > > > > > hbase:meta,,1.1588230740
> ba-wtmp05.asgardalfa.hq.com:16030
> > > > > > 219,168,603 35,419 1 MB 4
> 0
> > B
> > > > > > 1.0
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Wellington Chevreuil <we...@gmail.com>
> > > > > > Sent: Tuesday, December 15, 2020 7:10 PM
> > > > > > To: Hbase-User <us...@hbase.apache.org>
> > > > > > Subject: Re: Removal of table rows.
> > > > > >
> > > > > > EXTERNAL
> > > > > >
> > > > > > >
> > > > > > > I am an HBASE newbie so I apologise if I am being repetitious.
> > > > > > >
> > > > > > > Apologies also if this is not the right group. Am not sure
> > > > > > > if this may be more suited to 'dev' list.
> > > > > > >
> > > > > > Welcome, this is the right channel for this kind of questions.
> > > > > >
> > > > > >
> > > > > > > The solution offered by hbase-operator-tools -
> > > > > > > extraRegionsinMeta
> > > > > > > - offered hope. Once again, however, another problem has
> > surfaced:
> > > > > > > this tools command for extra regions is incompatible with
> > > > > > > the hbase version we are running.
> > > > > > >
> > > > > > This command does not rely on any Master/RegionServer
> > > > > > interface, so it should not have any incompatibility issues.
> > > > > > It only uses public client API to cleanup meta table, so maybe
> > > > > > you just need to build latest hbck2 master branch version? It
> > > > > > would work for any hbase 2, in theory, it could even work with
> > > > > > hbase 1, but that was
> > never tested.
> > > > > > Luckly, you got Stack's attention, so yeah, if you confirm on
> > > > > > his previous questions we might be able to help with further
> > directions.
> > > > > >
> > > > > > Em ter., 15 de dez. de 2020 às 17:23, Stack <st...@duboce.net>
> > > > escreveu:
> > > > > >
> > > > > > > On Tue, Dec 15, 2020 at 7:46 AM Marc Hoppins
> > > > > > > <ma...@eset.sk>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > I am an HBASE newbie so I apologise if I am being
> repetitious.
> > > > > > > >
> > > > > > > > Apologies also if this is not the right group. Am not sure
> > > > > > > > if this may be more suited to 'dev' list. However,
> > > > > > > >
> > > > > > > > A problem question and a technical/wishlist question.
> > > > > > > >
> > > > > > > > Problem:
> > > > > > > >
> > > > > > > > I have inherited a problem with an HBASE table. The
> > > > > > > > original issue may have erupted due to a network outage.
> > > > > > > > A table has
> > > > > > > > 48 region in
> > > > > > > transition
> > > > > > > > operations, stuck that way for several weeks. A previous
> > > > > > > > attempt to fix things with hbck failed. An attempt to
> > > > > > > > DISABLE then DROP the table also failed. The four or five
> > > > > > > > attempts to work the table ALSO now had stuck procedures.
> > > > > > > > Subsequent DFS and ZOO operations left the situation where
> > > > > > > > there was no data and no real table: just a 6K file in an
> > > > > > > > empty
> > > > > > > structure.
> > > > > > > >
> > > > > > > >
> > > > > > > Which version of hbase, do you know Marc? (Look at the base
> > > > > > > of the master UI. It'll tell you. Sounds like it is an
> hbase-2.x).
> > > > > > >
> > > > > > > What is the name of the 6k file? (I am trying to understand
> > > > > > > what the file you are referencing is).
> > > > > > >
> > > > > > > The table directory was removed from hdfs and zookeeper?
> > > > > > >
> > > > > > Yes. The data had already been removed from the previous tech
> > > > attempts
> > > > > > to fix the issue.
> > > > > >
> > > > > > Thus, the HDFS 6K file structure contained no actual
> > > > > > information, just empty files and directories. Zkshell had
> > > > > > been used to remove
> > > > znodes.
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > > When I got to the problem my knowledge of HBASE was nil.
> > > > > > > > It is little better than that now but anyway...
> > > > > > > >
> > > > > > > > Fortunately for me this is a testing/dev cluster. The 'owner'
> > > > > > > > was content that the table can be removed - and appeared
> > > > > > > > to have already been
> > > > > > > done...of
> > > > > > > > a kind.
> > > > > > > >
> > > > > > > > Reading and reading of others' similar issues lead me to
> > > > > > > > the point I also was going to clean the HDFS data and ZK
> > > > > > > > data for this table. I shut down HBASE, cleaned HDFS and
> > > > > > > > ZK node data, deleted the masterprocwals and restarted HBASE.
> > > > > > > >
> > > > > > > > When all came up I was happy to see that the affected
> > > > > > > > table appeared nowhere and that the procedures had all
> disappeared.
> > > > > > > >
> > > > > > > > However, when I hopped to hbase master, even though no
> > > > > > > > table of that name existed, 48 regions were still in
> transition.
> > > > > > > > Further research steered
> > > > > > > me
> > > > > > > > toward hbase:meta and sure enough, the references to the
> > > > > > > > RITs lived
> > > > > > > happily
> > > > > > > > among other data for other tables.
> > > > > > > >
> > > > > > > >
> > > > > > > Do you have a 'Procedures and Locks' tab on your master home
> > page?
> > > > > > > If so, what does it report? Is there an "HBCK Report" tab?
> > > > > > > If so, what does it show?
> > > > > > >
> > > > > >
> > > > > > When I got to the problem procedures & locks showed one item
> > > > > > for each problem on that table. After removal of
> > > > > > masterprocwals they all disappeared. The only entry now is one
> > > > > > normal-looking WAL log file entry whose timestamp changes
> > > > > > according to operation time (currently 8.25am today).
> > > > > >
> > > > > > >
> > > > > > > > The solution offered by hbase-operator-tools -
> > > > > > > > extraRegionsinMeta
> > > > > > > > - offered hope. Once again, however, another problem has
> > surfaced:
> > > > > > > > this tools command for extra regions is incompatible with
> > > > > > > > the hbase version we are running.
> > > > > > > >
> > > > > > > > So...
> > > > > > > >
> > > > > > > > How can I remove the references to namespace:kaput_table
> > > > > > > > from
> > > > > > hbase:meta?
> > > > > > > >
> > > > > > > >
> > > > > > > > Sounds like an hbase-2.1.x or a 2.0.x.
> > > > > > >
> > > > > > > Will wait on your answers to the above... It might be a
> > > > > > > crass delete of each row from the hbase:meta table then
> > > > > > > restart (even then, if procedures in the procedure store,
> > > > > > > you may have to clear it again as you did above before the
> > > > > > > restart to purge the procedures as you don't have tooling to
> > > > > > > do it from cmdline... do you have an 'hbck2 bypass
> > > > > > --override'?).
> > > > > > >
> > > > > > A crass delete of each row would work for me if I could
> > > > > > Identify how to remove the rows. Sample data:-
> > > > > >
> > > > > > alfa:rfilenameext column=table:state, timestamp=1604493139455,
> > > > > > value=\x08\x00
> > > > > >
> alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > > > > > column=info:regioninfo, timestamp=1604388258225,
> > > > > > value={ENCODED => 35925292c25898671e5a894ce387e167, NAME =>
> > > > > >
> > 'alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.'
> > > > > > ,
> > > > > > STARTKEY => '', ENDKEY => '0'}
> > > > > >
> alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > > > > > column=info:seqnumDuringOpen, timestamp=1601269814633,
> > > > > > value=\x00\x00\x00\x00\x00\x00\x008
> > > > > >
> alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > > > > > column=info:server, timestamp=1601269814633, value=
> > > > > > ba-wtmp04.asgardalfa.hq.com:16020
> > > > > >
> alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > > > > > column=info:serverstartcode, timestamp=1601269814633,
> > > > > > value=1601061167123
> > > > > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > > > > > column=info:sn, timestamp=1604388258050,
> > > > > > value=ba-wtmp04.asgardalfa.hq.com
> > > > > > ,16020,1601061167123
> > > > > >
> alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > > > > > column=info:state, timestamp=1604388258225, value=CLOSED
> > > > > >
> alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > > > > column=info:regioninfo, timestamp=1600969938610,
> > > > > > value={ENCODED => 787d1455b84f2d846ce6089392f01fd2, NAME =>
> > > > > >
> > 'alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > > > > ',
> > > > > > STARTKEY => '0', ENDKEY => '1'}
> > > > > >
> alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > > > > column=info:seqnumDuringOpen, timestamp=1600780187722,
> > > > > > value=\x00\x00\x00\x00\x00\x02^\xBB
> > > > > >
> alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > > > > column=info:server, timestamp=1600780187722, value=
> > > > > > ba-wtmp08.asgardalfa.hq.com:16020
> > > > > >
> alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > > > > column=info:serverstartcode, timestamp=1600780187722,
> > > > > > value=1600780162556
> > > > > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > > > > column=info:sn, timestamp=1600969938610,
> > > > > > value=ba-wtmp07.asgardalfa.hq.com
> > > > > > ,16020,1600936054386
> > > > > >
> alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > > > > column=info:state, timestamp=1600969938610, value=OPENING
> > > > > >
> alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > > > > > column=info:regioninfo, timestamp=1601060563980,
> > > > > > value={ENCODED => aa9d89b40a9def31a080fdd1776acb4e, NAME =>
> > > > > >
> > 'alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > > > > > ',
> > > > > > STARTKEY => '1', ENDKEY => '2'}
> > > > > >
> alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > > > > > column=info:seqnumDuringOpen, timestamp=1600780186976,
> > > > > > value=\x00\x00\x00\x00\x00\x02^\xA9
> > > > > >
> alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > > > > > column=info:server, timestamp=1600780186976, value=
> > > > > > dr1-wtmp02.asgardalfa.hq.com:16020
> > > > > >
> alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > > > > > column=info:serverstartcode, timestamp=1600780186976,
> > > > > > value=1600780163021
> > > > > alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > > > > > column=info:sn, timestamp=1601060563980,
> > > > > > value=ba-wtmp05.asgardalfa.hq.com
> > > > > > ,16020,1601049145467
> > > > > >
> > > > > >
> > > > > > As I state below, a simple method to remove everything
> > > > > > connected to alfa:rfilenameext would seem the easiest of tasks
> > > > > > as the namespace:name consistently appears on every relevant row.
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > > > Technical:
> > > > > > > >
> > > > > > > > Is there to be any implementation of such a fix within
> > > > > > > > HBASe itself where table manipulation can be performed by
> > > > > > > > Eg.,
> > > > > > > >
> > > > > > > > delete hbase:meta namespace delete hbase:meta
> > > > > > > > namespace:table
> > > > > > > >
> > > > > > > > or even
> > > > > > > >
> > > > > > > > scan hbase:meta filter = namespace:table | deleterow
> > > > > > > >
> > > > > > > > or some such?
> > > > > > > >
> > > > > > > >
> > > > > > > Scan doesn't return result (unfortunately) so you can assign
> > > > > > > to a shell variable; it dumps scan output2 on stdout/stderr.
> > > > > > >
> > > > > > > I'm sure there is a better way but something like below
> > > > > > > (don't
> > laugh!
> > > > > > > and I've not really tried it so be careful):
> > > > > > >
> > > > > > > # Get rows for the BAD_TABLENAME from hbase:meta table $
> > > > > > > echo 'scan "hbase:meta", {ROWPREFIXFILTER =>
> > > > > > > "BAD_TABLENAME"}'| hbase shell > /tmp/out.txt # Need to get
> > > > > > > the row only. Above got rows and
> > > > columns.
> > > > > > > We don't want to filter on column # because we do not know
> > > > > > > which columns are still in hbase:meta. Need to filter on
> > > > > > > table # again to cut the shell preamble and footer out. Unique
> the rows.
> > > > > > > $ cat /tmp/out.txt |awk '{print $1}'|grep BAD_TABLENAME |
> > > > > > > sort -u
> > > > > > > > /tmp/unique_rows.txt # Now you have rows to delete. For
> > > > > > > > each in
> > > > > > > shell do... 'deleteall "ROW_X"'... Could # edit
> > > > > > > /tmp/unique_rows.txt and per for add .... deleteall " prefix
> > > > > > > and
> > a "
> > > > > > > suffix.... to make
> > > > > > > # a file with lines of deleteall "ROW1", etc. then pass it
> > > > > > > to the
> > > > > shell:
> > > > > > > hbase shell /tmp/delete_rows.txt.
> > > > > > >
> > > > > > > But before messing w/ the above, lets get the answers to
> > > > > > > above
> > > > > questions.
> > > > > > > Thanks,
> > > > > > > S
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > > Thanks
> > > > > > > >
> > > > > > > > M
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
>
RE: Removal of table rows.
Posted by Marc Hoppins <ma...@eset.sk>.
As for the fix: apparently all is well. Balancing has restarted, no RITs anywhere.
Can anyone clarify what other operations are carried out other than removing meta rows? Eg., is zookeeper shell called to deal with any outstanding items? I saw zk information on screen when I ran tools.
-----Original Message-----
From: Wellington Chevreuil <we...@gmail.com>
Sent: Thursday, December 17, 2020 8:15 PM
To: Hbase-User <us...@hbase.apache.org>
Subject: Re: Removal of table rows.
EXTERNAL
>
> alfa:extgen,F,1608197544264.ba22f6113a0bb9520cee1f7b30050fa7.
> column=info:state, timestamp=1608197544904, value=OPEN
> alfa:rfilenameext column=table:state, timestamp=1604493139455,
> value=\x08\x00
> alfa:rfiles column=table:state, timestamp=1602600776355,
> value=\x08\x00
>
These are all table state rows, not table regions rows, so extraRegionsInMeta would not remove it.
> I did restart 'instances' on master(s) so I was surprised to see a
> lingering row.
>
The restart should have cleared out the RITs, but not the table state rows.
Can you confirm if you still see those regions in Master UI even after restarting masters?
Em qui., 17 de dez. de 2020 às 15:19, <sc...@comcast.net> escreveu:
> Marc, what type of data are loading into hbase? curious..
> > On 12/17/2020 8:46 AM Marc Hoppins <ma...@eset.sk> wrote:
> >
> >
> > I did restart 'instances' on master(s) so I was surprised to see a
> lingering row.
> >
> > -----Original Message-----
> > From: Wellington Chevreuil <we...@gmail.com>
> > Sent: Thursday, December 17, 2020 1:34 PM
> > To: Hbase-User <us...@hbase.apache.org>
> > Subject: Re: Removal of table rows.
> >
> > EXTERNAL
> >
> > Yes, depending on hbade version, you would need a master restart to
> effectively clean master cache.
> >
> > On Thu, 17 Dec 2020, 10:12 Marc Hoppins, <ma...@eset.sk> wrote:
> >
> > > OK. I had a meandering circuit to get a version of operator-tools
> > > built and running.
> > >
> > > After running it on the meta table
> > >
> > > sudo hbase hbck -j /tmp/hbase-hbck2-1.1.0-SNAPSHOT.jar
> > > extraRegionsInMeta alfa:rfilenameext --fix
> > >
> > > I have ended up with:-
> > >
> > > alfa:extgen,F,1608197544264.ba22f6113a0bb9520cee1f7b30050fa7.
> > > column=info:state, timestamp=1608197544904, value=OPEN
> > > alfa:rfilenameext column=table:state, timestamp=1604493139455,
> > > value=\x08\x00
> > > alfa:rfiles column=table:state, timestamp=1602600776355,
> > > value=\x08\x00
> > >
> > > I still have one row referring to the missing table. Do I need to
> > > restart hbase service to remove this or will it vanish at some time?
> > >
> > > HBASE on the master still shows the 48 regions in transition when
> > > I open the interface. I assume this is because the service has
> > > not been
> restarted.
> > >
> > > M
> > >
> > > -----Original Message-----
> > > From: Wellington Chevreuil <we...@gmail.com>
> > > Sent: Wednesday, December 16, 2020 3:39 PM
> > > To: Hbase-User <us...@hbase.apache.org>
> > > Subject: Re: Removal of table rows.
> > >
> > > EXTERNAL
> > >
> > > >
> > > > Do I build hbck2-tools on similar OS & java version? I have
> > > > been informed we have 'maven' installed on one host on a
> > > > cluster, which is centos and
> > > > (probably) a different java, and the build itself is needed for
> Ubuntu16.
> > >
> > > I would stick to the same java major version. OS wise, this module
> > > doesn't rely on any OS native call, AFAIK, so should work fine
> > > between these different versions.
> > >
> > > Em qua., 16 de dez. de 2020 às 13:30, Marc Hoppins
> > > <ma...@eset.sk>
> > > escreveu:
> > >
> > > > Thanks. Once again, a newbie in this regard but,
> > > >
> > > > Do I build hbck2-tools on similar OS & java version? I have
> > > > been informed we have 'maven' installed on one host on a
> > > > cluster, which is centos and
> > > > (probably) a different java, and the build itself is needed for
> Ubuntu16.
> > > >
> > > > -----Original Message-----
> > > > From: Wellington Chevreuil <we...@gmail.com>
> > > > Sent: Wednesday, December 16, 2020 10:08 AM
> > > > To: Hbase-User <us...@hbase.apache.org>
> > > > Subject: Re: Removal of table rows.
> > > >
> > > > EXTERNAL
> > > >
> > > > >
> > > > > Hbase:meta has info for one table region which is NOT on a master.
> > > > > Is that correct? I would have expected all meta info to be
> > > > > stored on a
> > > > master.
> > > > >
> > > > meta table is an "hbase system table" that has info about which
> > > > regions are assigned to which region servers in your cluster. As
> > > > any user table, it also has a region, and this region must be
> > > > hosted on a region server in the cluster. Master is just
> > > > responsible for the coordination of regions and some other housekeeping actions.
> > > >
> > > > As I state below, a simple method to remove everything connected
> > > > to
> > > > > alfa:rfilenameext would seem the easiest of tasks as the
> > > > > namespace:name consistently appears on every relevant row.
> > > >
> > > > Looks like hbck2 *extraRegionsinMeta *is what you need here,
> > > > and it should work with hbase 2.1. It's not available on hbck2
> > > > 1.0 release, but you can build a new hbck2 jar out of current
> > > > master branch, and that would give you an hbck2 with
> > > > extraRegionsinMeta. You can do it by cloning the repo below,
> > > > then do a *mvn install* from the main
> module:
> > > >
> > > > https://github.com/apache/hbase-operator-tools.git
> > > >
> > > > Em qua., 16 de dez. de 2020 às 08:02, Marc Hoppins
> > > > <ma...@eset.sk>
> > > > escreveu:
> > > >
> > > > > Hbase is 2.1.0 (via Cloudera CDH 6.3.2)
> > > > >
> > > > > The UI page for the master shows NO report links/tabs.
> > > > >
> > > > > Hbase:meta has info for one table region which is NOT on a master.
> > > > > Is that correct? I would have expected all meta info to be
> > > > > stored on a
> > > > master.
> > > > >
> > > > > Name Region Server
> > > > > Read Requests Write Requests
> > > > > Num.Storefiles MemSize Locality
> > > > > hbase:meta,,1.1588230740 ba-wtmp05.asgardalfa.hq.com:16030
> > > > > 219,168,603 35,419 1 MB 4 0
> B
> > > > > 1.0
> > > > >
> > > > > -----Original Message-----
> > > > > From: Wellington Chevreuil <we...@gmail.com>
> > > > > Sent: Tuesday, December 15, 2020 7:10 PM
> > > > > To: Hbase-User <us...@hbase.apache.org>
> > > > > Subject: Re: Removal of table rows.
> > > > >
> > > > > EXTERNAL
> > > > >
> > > > > >
> > > > > > I am an HBASE newbie so I apologise if I am being repetitious.
> > > > > >
> > > > > > Apologies also if this is not the right group. Am not sure
> > > > > > if this may be more suited to 'dev' list.
> > > > > >
> > > > > Welcome, this is the right channel for this kind of questions.
> > > > >
> > > > >
> > > > > > The solution offered by hbase-operator-tools -
> > > > > > extraRegionsinMeta
> > > > > > - offered hope. Once again, however, another problem has
> surfaced:
> > > > > > this tools command for extra regions is incompatible with
> > > > > > the hbase version we are running.
> > > > > >
> > > > > This command does not rely on any Master/RegionServer
> > > > > interface, so it should not have any incompatibility issues.
> > > > > It only uses public client API to cleanup meta table, so maybe
> > > > > you just need to build latest hbck2 master branch version? It
> > > > > would work for any hbase 2, in theory, it could even work with
> > > > > hbase 1, but that was
> never tested.
> > > > > Luckly, you got Stack's attention, so yeah, if you confirm on
> > > > > his previous questions we might be able to help with further
> directions.
> > > > >
> > > > > Em ter., 15 de dez. de 2020 às 17:23, Stack <st...@duboce.net>
> > > escreveu:
> > > > >
> > > > > > On Tue, Dec 15, 2020 at 7:46 AM Marc Hoppins
> > > > > > <ma...@eset.sk>
> > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I am an HBASE newbie so I apologise if I am being repetitious.
> > > > > > >
> > > > > > > Apologies also if this is not the right group. Am not sure
> > > > > > > if this may be more suited to 'dev' list. However,
> > > > > > >
> > > > > > > A problem question and a technical/wishlist question.
> > > > > > >
> > > > > > > Problem:
> > > > > > >
> > > > > > > I have inherited a problem with an HBASE table. The
> > > > > > > original issue may have erupted due to a network outage.
> > > > > > > A table has
> > > > > > > 48 region in
> > > > > > transition
> > > > > > > operations, stuck that way for several weeks. A previous
> > > > > > > attempt to fix things with hbck failed. An attempt to
> > > > > > > DISABLE then DROP the table also failed. The four or five
> > > > > > > attempts to work the table ALSO now had stuck procedures.
> > > > > > > Subsequent DFS and ZOO operations left the situation where
> > > > > > > there was no data and no real table: just a 6K file in an
> > > > > > > empty
> > > > > > structure.
> > > > > > >
> > > > > > >
> > > > > > Which version of hbase, do you know Marc? (Look at the base
> > > > > > of the master UI. It'll tell you. Sounds like it is an hbase-2.x).
> > > > > >
> > > > > > What is the name of the 6k file? (I am trying to understand
> > > > > > what the file you are referencing is).
> > > > > >
> > > > > > The table directory was removed from hdfs and zookeeper?
> > > > > >
> > > > > Yes. The data had already been removed from the previous tech
> > > attempts
> > > > > to fix the issue.
> > > > >
> > > > > Thus, the HDFS 6K file structure contained no actual
> > > > > information, just empty files and directories. Zkshell had
> > > > > been used to remove
> > > znodes.
> > > > >
> > > > >
> > > > > >
> > > > > > > When I got to the problem my knowledge of HBASE was nil.
> > > > > > > It is little better than that now but anyway...
> > > > > > >
> > > > > > > Fortunately for me this is a testing/dev cluster. The 'owner'
> > > > > > > was content that the table can be removed - and appeared
> > > > > > > to have already been
> > > > > > done...of
> > > > > > > a kind.
> > > > > > >
> > > > > > > Reading and reading of others' similar issues lead me to
> > > > > > > the point I also was going to clean the HDFS data and ZK
> > > > > > > data for this table. I shut down HBASE, cleaned HDFS and
> > > > > > > ZK node data, deleted the masterprocwals and restarted HBASE.
> > > > > > >
> > > > > > > When all came up I was happy to see that the affected
> > > > > > > table appeared nowhere and that the procedures had all disappeared.
> > > > > > >
> > > > > > > However, when I hopped to hbase master, even though no
> > > > > > > table of that name existed, 48 regions were still in transition.
> > > > > > > Further research steered
> > > > > > me
> > > > > > > toward hbase:meta and sure enough, the references to the
> > > > > > > RITs lived
> > > > > > happily
> > > > > > > among other data for other tables.
> > > > > > >
> > > > > > >
> > > > > > Do you have a 'Procedures and Locks' tab on your master home
> page?
> > > > > > If so, what does it report? Is there an "HBCK Report" tab?
> > > > > > If so, what does it show?
> > > > > >
> > > > >
> > > > > When I got to the problem procedures & locks showed one item
> > > > > for each problem on that table. After removal of
> > > > > masterprocwals they all disappeared. The only entry now is one
> > > > > normal-looking WAL log file entry whose timestamp changes
> > > > > according to operation time (currently 8.25am today).
> > > > >
> > > > > >
> > > > > > > The solution offered by hbase-operator-tools -
> > > > > > > extraRegionsinMeta
> > > > > > > - offered hope. Once again, however, another problem has
> surfaced:
> > > > > > > this tools command for extra regions is incompatible with
> > > > > > > the hbase version we are running.
> > > > > > >
> > > > > > > So...
> > > > > > >
> > > > > > > How can I remove the references to namespace:kaput_table
> > > > > > > from
> > > > > hbase:meta?
> > > > > > >
> > > > > > >
> > > > > > > Sounds like an hbase-2.1.x or a 2.0.x.
> > > > > >
> > > > > > Will wait on your answers to the above... It might be a
> > > > > > crass delete of each row from the hbase:meta table then
> > > > > > restart (even then, if procedures in the procedure store,
> > > > > > you may have to clear it again as you did above before the
> > > > > > restart to purge the procedures as you don't have tooling to
> > > > > > do it from cmdline... do you have an 'hbck2 bypass
> > > > > --override'?).
> > > > > >
> > > > > A crass delete of each row would work for me if I could
> > > > > Identify how to remove the rows. Sample data:-
> > > > >
> > > > > alfa:rfilenameext column=table:state, timestamp=1604493139455,
> > > > > value=\x08\x00
> > > > > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > > > > column=info:regioninfo, timestamp=1604388258225,
> > > > > value={ENCODED => 35925292c25898671e5a894ce387e167, NAME =>
> > > > >
> 'alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.'
> > > > > ,
> > > > > STARTKEY => '', ENDKEY => '0'}
> > > > > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > > > > column=info:seqnumDuringOpen, timestamp=1601269814633,
> > > > > value=\x00\x00\x00\x00\x00\x00\x008
> > > > > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > > > > column=info:server, timestamp=1601269814633, value=
> > > > > ba-wtmp04.asgardalfa.hq.com:16020
> > > > > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > > > > column=info:serverstartcode, timestamp=1601269814633,
> > > > > value=1601061167123
> > > > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > > > > column=info:sn, timestamp=1604388258050,
> > > > > value=ba-wtmp04.asgardalfa.hq.com
> > > > > ,16020,1601061167123
> > > > > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > > > > column=info:state, timestamp=1604388258225, value=CLOSED
> > > > > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > > > column=info:regioninfo, timestamp=1600969938610,
> > > > > value={ENCODED => 787d1455b84f2d846ce6089392f01fd2, NAME =>
> > > > >
> 'alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > > > ',
> > > > > STARTKEY => '0', ENDKEY => '1'}
> > > > > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > > > column=info:seqnumDuringOpen, timestamp=1600780187722,
> > > > > value=\x00\x00\x00\x00\x00\x02^\xBB
> > > > > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > > > column=info:server, timestamp=1600780187722, value=
> > > > > ba-wtmp08.asgardalfa.hq.com:16020
> > > > > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > > > column=info:serverstartcode, timestamp=1600780187722,
> > > > > value=1600780162556
> > > > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > > > column=info:sn, timestamp=1600969938610,
> > > > > value=ba-wtmp07.asgardalfa.hq.com
> > > > > ,16020,1600936054386
> > > > > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > > > column=info:state, timestamp=1600969938610, value=OPENING
> > > > > alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > > > > column=info:regioninfo, timestamp=1601060563980,
> > > > > value={ENCODED => aa9d89b40a9def31a080fdd1776acb4e, NAME =>
> > > > >
> 'alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > > > > ',
> > > > > STARTKEY => '1', ENDKEY => '2'}
> > > > > alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > > > > column=info:seqnumDuringOpen, timestamp=1600780186976,
> > > > > value=\x00\x00\x00\x00\x00\x02^\xA9
> > > > > alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > > > > column=info:server, timestamp=1600780186976, value=
> > > > > dr1-wtmp02.asgardalfa.hq.com:16020
> > > > > alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > > > > column=info:serverstartcode, timestamp=1600780186976,
> > > > > value=1600780163021
> > > > alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > > > > column=info:sn, timestamp=1601060563980,
> > > > > value=ba-wtmp05.asgardalfa.hq.com
> > > > > ,16020,1601049145467
> > > > >
> > > > >
> > > > > As I state below, a simple method to remove everything
> > > > > connected to alfa:rfilenameext would seem the easiest of tasks
> > > > > as the namespace:name consistently appears on every relevant row.
> > > > >
> > > > > >
> > > > > >
> > > > > > > Technical:
> > > > > > >
> > > > > > > Is there to be any implementation of such a fix within
> > > > > > > HBASe itself where table manipulation can be performed by
> > > > > > > Eg.,
> > > > > > >
> > > > > > > delete hbase:meta namespace delete hbase:meta
> > > > > > > namespace:table
> > > > > > >
> > > > > > > or even
> > > > > > >
> > > > > > > scan hbase:meta filter = namespace:table | deleterow
> > > > > > >
> > > > > > > or some such?
> > > > > > >
> > > > > > >
> > > > > > Scan doesn't return result (unfortunately) so you can assign
> > > > > > to a shell variable; it dumps scan output2 on stdout/stderr.
> > > > > >
> > > > > > I'm sure there is a better way but something like below
> > > > > > (don't
> laugh!
> > > > > > and I've not really tried it so be careful):
> > > > > >
> > > > > > # Get rows for the BAD_TABLENAME from hbase:meta table $
> > > > > > echo 'scan "hbase:meta", {ROWPREFIXFILTER =>
> > > > > > "BAD_TABLENAME"}'| hbase shell > /tmp/out.txt # Need to get
> > > > > > the row only. Above got rows and
> > > columns.
> > > > > > We don't want to filter on column # because we do not know
> > > > > > which columns are still in hbase:meta. Need to filter on
> > > > > > table # again to cut the shell preamble and footer out. Unique the rows.
> > > > > > $ cat /tmp/out.txt |awk '{print $1}'|grep BAD_TABLENAME |
> > > > > > sort -u
> > > > > > > /tmp/unique_rows.txt # Now you have rows to delete. For
> > > > > > > each in
> > > > > > shell do... 'deleteall "ROW_X"'... Could # edit
> > > > > > /tmp/unique_rows.txt and per for add .... deleteall " prefix
> > > > > > and
> a "
> > > > > > suffix.... to make
> > > > > > # a file with lines of deleteall "ROW1", etc. then pass it
> > > > > > to the
> > > > shell:
> > > > > > hbase shell /tmp/delete_rows.txt.
> > > > > >
> > > > > > But before messing w/ the above, lets get the answers to
> > > > > > above
> > > > questions.
> > > > > > Thanks,
> > > > > > S
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > > > M
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>
RE: Removal of table rows.
Posted by Marc Hoppins <ma...@eset.sk>.
😊 Which brings me back - full circle - to my original question: a way to remove rows from meta table.
I am no coder or database wallah but, my first reaction to seeing the metadata table and how each table is identified with (at the least) a table name on every row, was "why is there no method to remove row data based on namespace:table, or namespace alone?"
There will always be a case where some operator/techie will want to remove an entire table that may have stuck entities and will affect a fix by removing the data via non database methods first, leaving lingering table data in meta.
So, once again, I ask: is a method to remove rows via namespace or table name part of any development plan? Or part of hbase operator tools development?
-----Original Message-----
From: Wellington Chevreuil <we...@gmail.com>
Sent: Thursday, December 17, 2020 8:15 PM
To: Hbase-User <us...@hbase.apache.org>
Subject: Re: Removal of table rows.
EXTERNAL
>
> alfa:extgen,F,1608197544264.ba22f6113a0bb9520cee1f7b30050fa7.
> column=info:state, timestamp=1608197544904, value=OPEN
> alfa:rfilenameext column=table:state, timestamp=1604493139455,
> value=\x08\x00
> alfa:rfiles column=table:state, timestamp=1602600776355,
> value=\x08\x00
>
These are all table state rows, not table regions rows, so extraRegionsInMeta would not remove it.
> I did restart 'instances' on master(s) so I was surprised to see a
> lingering row.
>
The restart should have cleared out the RITs, but not the table state rows.
Can you confirm if you still see those regions in Master UI even after restarting masters?
Em qui., 17 de dez. de 2020 às 15:19, <sc...@comcast.net> escreveu:
> Marc, what type of data are loading into hbase? curious..
> > On 12/17/2020 8:46 AM Marc Hoppins <ma...@eset.sk> wrote:
> >
> >
> > I did restart 'instances' on master(s) so I was surprised to see a
> lingering row.
> >
> > -----Original Message-----
> > From: Wellington Chevreuil <we...@gmail.com>
> > Sent: Thursday, December 17, 2020 1:34 PM
> > To: Hbase-User <us...@hbase.apache.org>
> > Subject: Re: Removal of table rows.
> >
> > EXTERNAL
> >
> > Yes, depending on hbade version, you would need a master restart to
> effectively clean master cache.
> >
> > On Thu, 17 Dec 2020, 10:12 Marc Hoppins, <ma...@eset.sk> wrote:
> >
> > > OK. I had a meandering circuit to get a version of operator-tools
> > > built and running.
> > >
> > > After running it on the meta table
> > >
> > > sudo hbase hbck -j /tmp/hbase-hbck2-1.1.0-SNAPSHOT.jar
> > > extraRegionsInMeta alfa:rfilenameext --fix
> > >
> > > I have ended up with:-
> > >
> > > alfa:extgen,F,1608197544264.ba22f6113a0bb9520cee1f7b30050fa7.
> > > column=info:state, timestamp=1608197544904, value=OPEN
> > > alfa:rfilenameext column=table:state, timestamp=1604493139455,
> > > value=\x08\x00
> > > alfa:rfiles column=table:state, timestamp=1602600776355,
> > > value=\x08\x00
> > >
> > > I still have one row referring to the missing table. Do I need to
> > > restart hbase service to remove this or will it vanish at some time?
> > >
> > > HBASE on the master still shows the 48 regions in transition when
> > > I open the interface. I assume this is because the service has
> > > not been
> restarted.
> > >
> > > M
> > >
> > > -----Original Message-----
> > > From: Wellington Chevreuil <we...@gmail.com>
> > > Sent: Wednesday, December 16, 2020 3:39 PM
> > > To: Hbase-User <us...@hbase.apache.org>
> > > Subject: Re: Removal of table rows.
> > >
> > > EXTERNAL
> > >
> > > >
> > > > Do I build hbck2-tools on similar OS & java version? I have
> > > > been informed we have 'maven' installed on one host on a
> > > > cluster, which is centos and
> > > > (probably) a different java, and the build itself is needed for
> Ubuntu16.
> > >
> > > I would stick to the same java major version. OS wise, this module
> > > doesn't rely on any OS native call, AFAIK, so should work fine
> > > between these different versions.
> > >
> > > Em qua., 16 de dez. de 2020 às 13:30, Marc Hoppins
> > > <ma...@eset.sk>
> > > escreveu:
> > >
> > > > Thanks. Once again, a newbie in this regard but,
> > > >
> > > > Do I build hbck2-tools on similar OS & java version? I have
> > > > been informed we have 'maven' installed on one host on a
> > > > cluster, which is centos and
> > > > (probably) a different java, and the build itself is needed for
> Ubuntu16.
> > > >
> > > > -----Original Message-----
> > > > From: Wellington Chevreuil <we...@gmail.com>
> > > > Sent: Wednesday, December 16, 2020 10:08 AM
> > > > To: Hbase-User <us...@hbase.apache.org>
> > > > Subject: Re: Removal of table rows.
> > > >
> > > > EXTERNAL
> > > >
> > > > >
> > > > > Hbase:meta has info for one table region which is NOT on a master.
> > > > > Is that correct? I would have expected all meta info to be
> > > > > stored on a
> > > > master.
> > > > >
> > > > meta table is an "hbase system table" that has info about which
> > > > regions are assigned to which region servers in your cluster. As
> > > > any user table, it also has a region, and this region must be
> > > > hosted on a region server in the cluster. Master is just
> > > > responsible for the coordination of regions and some other housekeeping actions.
> > > >
> > > > As I state below, a simple method to remove everything connected
> > > > to
> > > > > alfa:rfilenameext would seem the easiest of tasks as the
> > > > > namespace:name consistently appears on every relevant row.
> > > >
> > > > Looks like hbck2 *extraRegionsinMeta *is what you need here,
> > > > and it should work with hbase 2.1. It's not available on hbck2
> > > > 1.0 release, but you can build a new hbck2 jar out of current
> > > > master branch, and that would give you an hbck2 with
> > > > extraRegionsinMeta. You can do it by cloning the repo below,
> > > > then do a *mvn install* from the main
> module:
> > > >
> > > > https://github.com/apache/hbase-operator-tools.git
> > > >
> > > > Em qua., 16 de dez. de 2020 às 08:02, Marc Hoppins
> > > > <ma...@eset.sk>
> > > > escreveu:
> > > >
> > > > > Hbase is 2.1.0 (via Cloudera CDH 6.3.2)
> > > > >
> > > > > The UI page for the master shows NO report links/tabs.
> > > > >
> > > > > Hbase:meta has info for one table region which is NOT on a master.
> > > > > Is that correct? I would have expected all meta info to be
> > > > > stored on a
> > > > master.
> > > > >
> > > > > Name Region Server
> > > > > Read Requests Write Requests
> > > > > Num.Storefiles MemSize Locality
> > > > > hbase:meta,,1.1588230740 ba-wtmp05.asgardalfa.hq.com:16030
> > > > > 219,168,603 35,419 1 MB 4 0
> B
> > > > > 1.0
> > > > >
> > > > > -----Original Message-----
> > > > > From: Wellington Chevreuil <we...@gmail.com>
> > > > > Sent: Tuesday, December 15, 2020 7:10 PM
> > > > > To: Hbase-User <us...@hbase.apache.org>
> > > > > Subject: Re: Removal of table rows.
> > > > >
> > > > > EXTERNAL
> > > > >
> > > > > >
> > > > > > I am an HBASE newbie so I apologise if I am being repetitious.
> > > > > >
> > > > > > Apologies also if this is not the right group. Am not sure
> > > > > > if this may be more suited to 'dev' list.
> > > > > >
> > > > > Welcome, this is the right channel for this kind of questions.
> > > > >
> > > > >
> > > > > > The solution offered by hbase-operator-tools -
> > > > > > extraRegionsinMeta
> > > > > > - offered hope. Once again, however, another problem has
> surfaced:
> > > > > > this tools command for extra regions is incompatible with
> > > > > > the hbase version we are running.
> > > > > >
> > > > > This command does not rely on any Master/RegionServer
> > > > > interface, so it should not have any incompatibility issues.
> > > > > It only uses public client API to cleanup meta table, so maybe
> > > > > you just need to build latest hbck2 master branch version? It
> > > > > would work for any hbase 2, in theory, it could even work with
> > > > > hbase 1, but that was
> never tested.
> > > > > Luckly, you got Stack's attention, so yeah, if you confirm on
> > > > > his previous questions we might be able to help with further
> directions.
> > > > >
> > > > > Em ter., 15 de dez. de 2020 às 17:23, Stack <st...@duboce.net>
> > > escreveu:
> > > > >
> > > > > > On Tue, Dec 15, 2020 at 7:46 AM Marc Hoppins
> > > > > > <ma...@eset.sk>
> > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I am an HBASE newbie so I apologise if I am being repetitious.
> > > > > > >
> > > > > > > Apologies also if this is not the right group. Am not sure
> > > > > > > if this may be more suited to 'dev' list. However,
> > > > > > >
> > > > > > > A problem question and a technical/wishlist question.
> > > > > > >
> > > > > > > Problem:
> > > > > > >
> > > > > > > I have inherited a problem with an HBASE table. The
> > > > > > > original issue may have erupted due to a network outage.
> > > > > > > A table has
> > > > > > > 48 region in
> > > > > > transition
> > > > > > > operations, stuck that way for several weeks. A previous
> > > > > > > attempt to fix things with hbck failed. An attempt to
> > > > > > > DISABLE then DROP the table also failed. The four or five
> > > > > > > attempts to work the table ALSO now had stuck procedures.
> > > > > > > Subsequent DFS and ZOO operations left the situation where
> > > > > > > there was no data and no real table: just a 6K file in an
> > > > > > > empty
> > > > > > structure.
> > > > > > >
> > > > > > >
> > > > > > Which version of hbase, do you know Marc? (Look at the base
> > > > > > of the master UI. It'll tell you. Sounds like it is an hbase-2.x).
> > > > > >
> > > > > > What is the name of the 6k file? (I am trying to understand
> > > > > > what the file you are referencing is).
> > > > > >
> > > > > > The table directory was removed from hdfs and zookeeper?
> > > > > >
> > > > > Yes. The data had already been removed from the previous tech
> > > attempts
> > > > > to fix the issue.
> > > > >
> > > > > Thus, the HDFS 6K file structure contained no actual
> > > > > information, just empty files and directories. Zkshell had
> > > > > been used to remove
> > > znodes.
> > > > >
> > > > >
> > > > > >
> > > > > > > When I got to the problem my knowledge of HBASE was nil.
> > > > > > > It is little better than that now but anyway...
> > > > > > >
> > > > > > > Fortunately for me this is a testing/dev cluster. The 'owner'
> > > > > > > was content that the table can be removed - and appeared
> > > > > > > to have already been
> > > > > > done...of
> > > > > > > a kind.
> > > > > > >
> > > > > > > Reading and reading of others' similar issues lead me to
> > > > > > > the point I also was going to clean the HDFS data and ZK
> > > > > > > data for this table. I shut down HBASE, cleaned HDFS and
> > > > > > > ZK node data, deleted the masterprocwals and restarted HBASE.
> > > > > > >
> > > > > > > When all came up I was happy to see that the affected
> > > > > > > table appeared nowhere and that the procedures had all disappeared.
> > > > > > >
> > > > > > > However, when I hopped to hbase master, even though no
> > > > > > > table of that name existed, 48 regions were still in transition.
> > > > > > > Further research steered
> > > > > > me
> > > > > > > toward hbase:meta and sure enough, the references to the
> > > > > > > RITs lived
> > > > > > happily
> > > > > > > among other data for other tables.
> > > > > > >
> > > > > > >
> > > > > > Do you have a 'Procedures and Locks' tab on your master home
> page?
> > > > > > If so, what does it report? Is there an "HBCK Report" tab?
> > > > > > If so, what does it show?
> > > > > >
> > > > >
> > > > > When I got to the problem procedures & locks showed one item
> > > > > for each problem on that table. After removal of
> > > > > masterprocwals they all disappeared. The only entry now is one
> > > > > normal-looking WAL log file entry whose timestamp changes
> > > > > according to operation time (currently 8.25am today).
> > > > >
> > > > > >
> > > > > > > The solution offered by hbase-operator-tools -
> > > > > > > extraRegionsinMeta
> > > > > > > - offered hope. Once again, however, another problem has
> surfaced:
> > > > > > > this tools command for extra regions is incompatible with
> > > > > > > the hbase version we are running.
> > > > > > >
> > > > > > > So...
> > > > > > >
> > > > > > > How can I remove the references to namespace:kaput_table
> > > > > > > from
> > > > > hbase:meta?
> > > > > > >
> > > > > > >
> > > > > > > Sounds like an hbase-2.1.x or a 2.0.x.
> > > > > >
> > > > > > Will wait on your answers to the above... It might be a
> > > > > > crass delete of each row from the hbase:meta table then
> > > > > > restart (even then, if procedures in the procedure store,
> > > > > > you may have to clear it again as you did above before the
> > > > > > restart to purge the procedures as you don't have tooling to
> > > > > > do it from cmdline... do you have an 'hbck2 bypass
> > > > > --override'?).
> > > > > >
> > > > > A crass delete of each row would work for me if I could
> > > > > Identify how to remove the rows. Sample data:-
> > > > >
> > > > > alfa:rfilenameext column=table:state, timestamp=1604493139455,
> > > > > value=\x08\x00
> > > > > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > > > > column=info:regioninfo, timestamp=1604388258225,
> > > > > value={ENCODED => 35925292c25898671e5a894ce387e167, NAME =>
> > > > >
> 'alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.'
> > > > > ,
> > > > > STARTKEY => '', ENDKEY => '0'}
> > > > > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > > > > column=info:seqnumDuringOpen, timestamp=1601269814633,
> > > > > value=\x00\x00\x00\x00\x00\x00\x008
> > > > > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > > > > column=info:server, timestamp=1601269814633, value=
> > > > > ba-wtmp04.asgardalfa.hq.com:16020
> > > > > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > > > > column=info:serverstartcode, timestamp=1601269814633,
> > > > > value=1601061167123
> > > > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > > > > column=info:sn, timestamp=1604388258050,
> > > > > value=ba-wtmp04.asgardalfa.hq.com
> > > > > ,16020,1601061167123
> > > > > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > > > > column=info:state, timestamp=1604388258225, value=CLOSED
> > > > > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > > > column=info:regioninfo, timestamp=1600969938610,
> > > > > value={ENCODED => 787d1455b84f2d846ce6089392f01fd2, NAME =>
> > > > >
> 'alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > > > ',
> > > > > STARTKEY => '0', ENDKEY => '1'}
> > > > > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > > > column=info:seqnumDuringOpen, timestamp=1600780187722,
> > > > > value=\x00\x00\x00\x00\x00\x02^\xBB
> > > > > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > > > column=info:server, timestamp=1600780187722, value=
> > > > > ba-wtmp08.asgardalfa.hq.com:16020
> > > > > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > > > column=info:serverstartcode, timestamp=1600780187722,
> > > > > value=1600780162556
> > > > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > > > column=info:sn, timestamp=1600969938610,
> > > > > value=ba-wtmp07.asgardalfa.hq.com
> > > > > ,16020,1600936054386
> > > > > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > > > column=info:state, timestamp=1600969938610, value=OPENING
> > > > > alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > > > > column=info:regioninfo, timestamp=1601060563980,
> > > > > value={ENCODED => aa9d89b40a9def31a080fdd1776acb4e, NAME =>
> > > > >
> 'alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > > > > ',
> > > > > STARTKEY => '1', ENDKEY => '2'}
> > > > > alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > > > > column=info:seqnumDuringOpen, timestamp=1600780186976,
> > > > > value=\x00\x00\x00\x00\x00\x02^\xA9
> > > > > alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > > > > column=info:server, timestamp=1600780186976, value=
> > > > > dr1-wtmp02.asgardalfa.hq.com:16020
> > > > > alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > > > > column=info:serverstartcode, timestamp=1600780186976,
> > > > > value=1600780163021
> > > > alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > > > > column=info:sn, timestamp=1601060563980,
> > > > > value=ba-wtmp05.asgardalfa.hq.com
> > > > > ,16020,1601049145467
> > > > >
> > > > >
> > > > > As I state below, a simple method to remove everything
> > > > > connected to alfa:rfilenameext would seem the easiest of tasks
> > > > > as the namespace:name consistently appears on every relevant row.
> > > > >
> > > > > >
> > > > > >
> > > > > > > Technical:
> > > > > > >
> > > > > > > Is there to be any implementation of such a fix within
> > > > > > > HBASe itself where table manipulation can be performed by
> > > > > > > Eg.,
> > > > > > >
> > > > > > > delete hbase:meta namespace delete hbase:meta
> > > > > > > namespace:table
> > > > > > >
> > > > > > > or even
> > > > > > >
> > > > > > > scan hbase:meta filter = namespace:table | deleterow
> > > > > > >
> > > > > > > or some such?
> > > > > > >
> > > > > > >
> > > > > > Scan doesn't return result (unfortunately) so you can assign
> > > > > > to a shell variable; it dumps scan output2 on stdout/stderr.
> > > > > >
> > > > > > I'm sure there is a better way but something like below
> > > > > > (don't
> laugh!
> > > > > > and I've not really tried it so be careful):
> > > > > >
> > > > > > # Get rows for the BAD_TABLENAME from hbase:meta table $
> > > > > > echo 'scan "hbase:meta", {ROWPREFIXFILTER =>
> > > > > > "BAD_TABLENAME"}'| hbase shell > /tmp/out.txt # Need to get
> > > > > > the row only. Above got rows and
> > > columns.
> > > > > > We don't want to filter on column # because we do not know
> > > > > > which columns are still in hbase:meta. Need to filter on
> > > > > > table # again to cut the shell preamble and footer out. Unique the rows.
> > > > > > $ cat /tmp/out.txt |awk '{print $1}'|grep BAD_TABLENAME |
> > > > > > sort -u
> > > > > > > /tmp/unique_rows.txt # Now you have rows to delete. For
> > > > > > > each in
> > > > > > shell do... 'deleteall "ROW_X"'... Could # edit
> > > > > > /tmp/unique_rows.txt and per for add .... deleteall " prefix
> > > > > > and
> a "
> > > > > > suffix.... to make
> > > > > > # a file with lines of deleteall "ROW1", etc. then pass it
> > > > > > to the
> > > > shell:
> > > > > > hbase shell /tmp/delete_rows.txt.
> > > > > >
> > > > > > But before messing w/ the above, lets get the answers to
> > > > > > above
> > > > questions.
> > > > > > Thanks,
> > > > > > S
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > > > M
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>
Re: Removal of table rows.
Posted by Wellington Chevreuil <we...@gmail.com>.
>
> alfa:extgen,F,1608197544264.ba22f6113a0bb9520cee1f7b30050fa7.
> column=info:state, timestamp=1608197544904, value=OPEN
> alfa:rfilenameext column=table:state, timestamp=1604493139455,
> value=\x08\x00
> alfa:rfiles column=table:state, timestamp=1602600776355, value=\x08\x00
>
These are all table state rows, not table regions rows, so
extraRegionsInMeta would not remove it.
> I did restart 'instances' on master(s) so I was surprised to see a
> lingering row.
>
The restart should have cleared out the RITs, but not the table state rows.
Can you confirm if you still see those regions in Master UI even after
restarting masters?
Em qui., 17 de dez. de 2020 às 15:19, <sc...@comcast.net> escreveu:
> Marc, what type of data are loading into hbase? curious..
> > On 12/17/2020 8:46 AM Marc Hoppins <ma...@eset.sk> wrote:
> >
> >
> > I did restart 'instances' on master(s) so I was surprised to see a
> lingering row.
> >
> > -----Original Message-----
> > From: Wellington Chevreuil <we...@gmail.com>
> > Sent: Thursday, December 17, 2020 1:34 PM
> > To: Hbase-User <us...@hbase.apache.org>
> > Subject: Re: Removal of table rows.
> >
> > EXTERNAL
> >
> > Yes, depending on hbade version, you would need a master restart to
> effectively clean master cache.
> >
> > On Thu, 17 Dec 2020, 10:12 Marc Hoppins, <ma...@eset.sk> wrote:
> >
> > > OK. I had a meandering circuit to get a version of operator-tools
> > > built and running.
> > >
> > > After running it on the meta table
> > >
> > > sudo hbase hbck -j /tmp/hbase-hbck2-1.1.0-SNAPSHOT.jar
> > > extraRegionsInMeta alfa:rfilenameext --fix
> > >
> > > I have ended up with:-
> > >
> > > alfa:extgen,F,1608197544264.ba22f6113a0bb9520cee1f7b30050fa7.
> > > column=info:state, timestamp=1608197544904, value=OPEN
> > > alfa:rfilenameext column=table:state, timestamp=1604493139455,
> > > value=\x08\x00
> > > alfa:rfiles column=table:state, timestamp=1602600776355,
> > > value=\x08\x00
> > >
> > > I still have one row referring to the missing table. Do I need to
> > > restart hbase service to remove this or will it vanish at some time?
> > >
> > > HBASE on the master still shows the 48 regions in transition when I
> > > open the interface. I assume this is because the service has not been
> restarted.
> > >
> > > M
> > >
> > > -----Original Message-----
> > > From: Wellington Chevreuil <we...@gmail.com>
> > > Sent: Wednesday, December 16, 2020 3:39 PM
> > > To: Hbase-User <us...@hbase.apache.org>
> > > Subject: Re: Removal of table rows.
> > >
> > > EXTERNAL
> > >
> > > >
> > > > Do I build hbck2-tools on similar OS & java version? I have been
> > > > informed we have 'maven' installed on one host on a cluster, which
> > > > is centos and
> > > > (probably) a different java, and the build itself is needed for
> Ubuntu16.
> > >
> > > I would stick to the same java major version. OS wise, this module
> > > doesn't rely on any OS native call, AFAIK, so should work fine between
> > > these different versions.
> > >
> > > Em qua., 16 de dez. de 2020 às 13:30, Marc Hoppins
> > > <ma...@eset.sk>
> > > escreveu:
> > >
> > > > Thanks. Once again, a newbie in this regard but,
> > > >
> > > > Do I build hbck2-tools on similar OS & java version? I have been
> > > > informed we have 'maven' installed on one host on a cluster, which
> > > > is centos and
> > > > (probably) a different java, and the build itself is needed for
> Ubuntu16.
> > > >
> > > > -----Original Message-----
> > > > From: Wellington Chevreuil <we...@gmail.com>
> > > > Sent: Wednesday, December 16, 2020 10:08 AM
> > > > To: Hbase-User <us...@hbase.apache.org>
> > > > Subject: Re: Removal of table rows.
> > > >
> > > > EXTERNAL
> > > >
> > > > >
> > > > > Hbase:meta has info for one table region which is NOT on a master.
> > > > > Is that correct? I would have expected all meta info to be stored
> > > > > on a
> > > > master.
> > > > >
> > > > meta table is an "hbase system table" that has info about which
> > > > regions are assigned to which region servers in your cluster. As any
> > > > user table, it also has a region, and this region must be hosted on
> > > > a region server in the cluster. Master is just responsible for the
> > > > coordination of regions and some other housekeeping actions.
> > > >
> > > > As I state below, a simple method to remove everything connected to
> > > > > alfa:rfilenameext would seem the easiest of tasks as the
> > > > > namespace:name consistently appears on every relevant row.
> > > >
> > > > Looks like hbck2 *extraRegionsinMeta *is what you need here, and it
> > > > should work with hbase 2.1. It's not available on hbck2 1.0 release,
> > > > but you can build a new hbck2 jar out of current master branch, and
> > > > that would give you an hbck2 with extraRegionsinMeta. You can do it
> > > > by cloning the repo below, then do a *mvn install* from the main
> module:
> > > >
> > > > https://github.com/apache/hbase-operator-tools.git
> > > >
> > > > Em qua., 16 de dez. de 2020 às 08:02, Marc Hoppins
> > > > <ma...@eset.sk>
> > > > escreveu:
> > > >
> > > > > Hbase is 2.1.0 (via Cloudera CDH 6.3.2)
> > > > >
> > > > > The UI page for the master shows NO report links/tabs.
> > > > >
> > > > > Hbase:meta has info for one table region which is NOT on a master.
> > > > > Is that correct? I would have expected all meta info to be stored
> > > > > on a
> > > > master.
> > > > >
> > > > > Name Region Server
> > > > > Read Requests Write Requests
> > > > > Num.Storefiles MemSize Locality
> > > > > hbase:meta,,1.1588230740 ba-wtmp05.asgardalfa.hq.com:16030
> > > > > 219,168,603 35,419 1 MB 4 0
> B
> > > > > 1.0
> > > > >
> > > > > -----Original Message-----
> > > > > From: Wellington Chevreuil <we...@gmail.com>
> > > > > Sent: Tuesday, December 15, 2020 7:10 PM
> > > > > To: Hbase-User <us...@hbase.apache.org>
> > > > > Subject: Re: Removal of table rows.
> > > > >
> > > > > EXTERNAL
> > > > >
> > > > > >
> > > > > > I am an HBASE newbie so I apologise if I am being repetitious.
> > > > > >
> > > > > > Apologies also if this is not the right group. Am not sure if
> > > > > > this may be more suited to 'dev' list.
> > > > > >
> > > > > Welcome, this is the right channel for this kind of questions.
> > > > >
> > > > >
> > > > > > The solution offered by hbase-operator-tools -
> > > > > > extraRegionsinMeta
> > > > > > - offered hope. Once again, however, another problem has
> surfaced:
> > > > > > this tools command for extra regions is incompatible with the
> > > > > > hbase version we are running.
> > > > > >
> > > > > This command does not rely on any Master/RegionServer interface,
> > > > > so it should not have any incompatibility issues. It only uses
> > > > > public client API to cleanup meta table, so maybe you just need to
> > > > > build latest hbck2 master branch version? It would work for any
> > > > > hbase 2, in theory, it could even work with hbase 1, but that was
> never tested.
> > > > > Luckly, you got Stack's attention, so yeah, if you confirm on his
> > > > > previous questions we might be able to help with further
> directions.
> > > > >
> > > > > Em ter., 15 de dez. de 2020 às 17:23, Stack <st...@duboce.net>
> > > escreveu:
> > > > >
> > > > > > On Tue, Dec 15, 2020 at 7:46 AM Marc Hoppins
> > > > > > <ma...@eset.sk>
> > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I am an HBASE newbie so I apologise if I am being repetitious.
> > > > > > >
> > > > > > > Apologies also if this is not the right group. Am not sure if
> > > > > > > this may be more suited to 'dev' list. However,
> > > > > > >
> > > > > > > A problem question and a technical/wishlist question.
> > > > > > >
> > > > > > > Problem:
> > > > > > >
> > > > > > > I have inherited a problem with an HBASE table. The original
> > > > > > > issue may have erupted due to a network outage. A table has
> > > > > > > 48 region in
> > > > > > transition
> > > > > > > operations, stuck that way for several weeks. A previous
> > > > > > > attempt to fix things with hbck failed. An attempt to DISABLE
> > > > > > > then DROP the table also failed. The four or five attempts to
> > > > > > > work the table ALSO now had stuck procedures. Subsequent DFS
> > > > > > > and ZOO operations left the situation where there was no data
> > > > > > > and no real table: just a 6K file in an empty
> > > > > > structure.
> > > > > > >
> > > > > > >
> > > > > > Which version of hbase, do you know Marc? (Look at the base of
> > > > > > the master UI. It'll tell you. Sounds like it is an hbase-2.x).
> > > > > >
> > > > > > What is the name of the 6k file? (I am trying to understand what
> > > > > > the file you are referencing is).
> > > > > >
> > > > > > The table directory was removed from hdfs and zookeeper?
> > > > > >
> > > > > Yes. The data had already been removed from the previous tech
> > > attempts
> > > > > to fix the issue.
> > > > >
> > > > > Thus, the HDFS 6K file structure contained no actual information,
> > > > > just empty files and directories. Zkshell had been used to remove
> > > znodes.
> > > > >
> > > > >
> > > > > >
> > > > > > > When I got to the problem my knowledge of HBASE was nil. It is
> > > > > > > little better than that now but anyway...
> > > > > > >
> > > > > > > Fortunately for me this is a testing/dev cluster. The 'owner'
> > > > > > > was content that the table can be removed - and appeared to
> > > > > > > have already been
> > > > > > done...of
> > > > > > > a kind.
> > > > > > >
> > > > > > > Reading and reading of others' similar issues lead me to the
> > > > > > > point I also was going to clean the HDFS data and ZK data for
> > > > > > > this table. I shut down HBASE, cleaned HDFS and ZK node data,
> > > > > > > deleted the masterprocwals and restarted HBASE.
> > > > > > >
> > > > > > > When all came up I was happy to see that the affected table
> > > > > > > appeared nowhere and that the procedures had all disappeared.
> > > > > > >
> > > > > > > However, when I hopped to hbase master, even though no table
> > > > > > > of that name existed, 48 regions were still in transition.
> > > > > > > Further research steered
> > > > > > me
> > > > > > > toward hbase:meta and sure enough, the references to the RITs
> > > > > > > lived
> > > > > > happily
> > > > > > > among other data for other tables.
> > > > > > >
> > > > > > >
> > > > > > Do you have a 'Procedures and Locks' tab on your master home
> page?
> > > > > > If so, what does it report? Is there an "HBCK Report" tab? If
> > > > > > so, what does it show?
> > > > > >
> > > > >
> > > > > When I got to the problem procedures & locks showed one item for
> > > > > each problem on that table. After removal of masterprocwals they
> > > > > all disappeared. The only entry now is one normal-looking WAL log
> > > > > file entry whose timestamp changes according to operation time
> > > > > (currently 8.25am today).
> > > > >
> > > > > >
> > > > > > > The solution offered by hbase-operator-tools -
> > > > > > > extraRegionsinMeta
> > > > > > > - offered hope. Once again, however, another problem has
> surfaced:
> > > > > > > this tools command for extra regions is incompatible with the
> > > > > > > hbase version we are running.
> > > > > > >
> > > > > > > So...
> > > > > > >
> > > > > > > How can I remove the references to namespace:kaput_table from
> > > > > hbase:meta?
> > > > > > >
> > > > > > >
> > > > > > > Sounds like an hbase-2.1.x or a 2.0.x.
> > > > > >
> > > > > > Will wait on your answers to the above... It might be a crass
> > > > > > delete of each row from the hbase:meta table then restart (even
> > > > > > then, if procedures in the procedure store, you may have to
> > > > > > clear it again as you did above before the restart to purge the
> > > > > > procedures as you don't have tooling to do it from cmdline... do
> > > > > > you have an 'hbck2 bypass
> > > > > --override'?).
> > > > > >
> > > > > A crass delete of each row would work for me if I could Identify
> > > > > how to remove the rows. Sample data:-
> > > > >
> > > > > alfa:rfilenameext column=table:state, timestamp=1604493139455,
> > > > > value=\x08\x00
> > > > > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > > > > column=info:regioninfo, timestamp=1604388258225, value={ENCODED =>
> > > > > 35925292c25898671e5a894ce387e167, NAME =>
> > > > >
> 'alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.'
> > > > > ,
> > > > > STARTKEY => '', ENDKEY => '0'}
> > > > > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > > > > column=info:seqnumDuringOpen, timestamp=1601269814633,
> > > > > value=\x00\x00\x00\x00\x00\x00\x008
> > > > > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > > > > column=info:server, timestamp=1601269814633, value=
> > > > > ba-wtmp04.asgardalfa.hq.com:16020
> > > > > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > > > > column=info:serverstartcode, timestamp=1601269814633,
> > > > > value=1601061167123
> > > > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > > > > column=info:sn, timestamp=1604388258050,
> > > > > value=ba-wtmp04.asgardalfa.hq.com
> > > > > ,16020,1601061167123
> > > > > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > > > > column=info:state, timestamp=1604388258225, value=CLOSED
> > > > > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > > > column=info:regioninfo, timestamp=1600969938610, value={ENCODED =>
> > > > > 787d1455b84f2d846ce6089392f01fd2, NAME =>
> > > > >
> 'alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > > > ',
> > > > > STARTKEY => '0', ENDKEY => '1'}
> > > > > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > > > column=info:seqnumDuringOpen, timestamp=1600780187722,
> > > > > value=\x00\x00\x00\x00\x00\x02^\xBB
> > > > > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > > > column=info:server, timestamp=1600780187722, value=
> > > > > ba-wtmp08.asgardalfa.hq.com:16020
> > > > > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > > > column=info:serverstartcode, timestamp=1600780187722,
> > > > > value=1600780162556
> > > > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > > > column=info:sn, timestamp=1600969938610,
> > > > > value=ba-wtmp07.asgardalfa.hq.com
> > > > > ,16020,1600936054386
> > > > > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > > > column=info:state, timestamp=1600969938610, value=OPENING
> > > > > alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > > > > column=info:regioninfo, timestamp=1601060563980, value={ENCODED =>
> > > > > aa9d89b40a9def31a080fdd1776acb4e, NAME =>
> > > > >
> 'alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > > > > ',
> > > > > STARTKEY => '1', ENDKEY => '2'}
> > > > > alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > > > > column=info:seqnumDuringOpen, timestamp=1600780186976,
> > > > > value=\x00\x00\x00\x00\x00\x02^\xA9
> > > > > alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > > > > column=info:server, timestamp=1600780186976, value=
> > > > > dr1-wtmp02.asgardalfa.hq.com:16020
> > > > > alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > > > > column=info:serverstartcode, timestamp=1600780186976,
> > > > > value=1600780163021
> > > > alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > > > > column=info:sn, timestamp=1601060563980,
> > > > > value=ba-wtmp05.asgardalfa.hq.com
> > > > > ,16020,1601049145467
> > > > >
> > > > >
> > > > > As I state below, a simple method to remove everything connected
> > > > > to alfa:rfilenameext would seem the easiest of tasks as the
> > > > > namespace:name consistently appears on every relevant row.
> > > > >
> > > > > >
> > > > > >
> > > > > > > Technical:
> > > > > > >
> > > > > > > Is there to be any implementation of such a fix within HBASe
> > > > > > > itself where table manipulation can be performed by Eg.,
> > > > > > >
> > > > > > > delete hbase:meta namespace
> > > > > > > delete hbase:meta namespace:table
> > > > > > >
> > > > > > > or even
> > > > > > >
> > > > > > > scan hbase:meta filter = namespace:table | deleterow
> > > > > > >
> > > > > > > or some such?
> > > > > > >
> > > > > > >
> > > > > > Scan doesn't return result (unfortunately) so you can assign to
> > > > > > a shell variable; it dumps scan output2 on stdout/stderr.
> > > > > >
> > > > > > I'm sure there is a better way but something like below (don't
> laugh!
> > > > > > and I've not really tried it so be careful):
> > > > > >
> > > > > > # Get rows for the BAD_TABLENAME from hbase:meta table $ echo
> > > > > > 'scan "hbase:meta", {ROWPREFIXFILTER => "BAD_TABLENAME"}'| hbase
> > > > > > shell > /tmp/out.txt # Need to get the row only. Above got rows
> > > > > > and
> > > columns.
> > > > > > We don't want to filter on column # because we do not know which
> > > > > > columns are still in hbase:meta. Need to filter on table # again
> > > > > > to cut the shell preamble and footer out. Unique the rows.
> > > > > > $ cat /tmp/out.txt |awk '{print $1}'|grep BAD_TABLENAME | sort
> > > > > > -u
> > > > > > > /tmp/unique_rows.txt # Now you have rows to delete. For each
> > > > > > > in
> > > > > > shell do... 'deleteall "ROW_X"'... Could # edit
> > > > > > /tmp/unique_rows.txt and per for add .... deleteall " prefix and
> a "
> > > > > > suffix.... to make
> > > > > > # a file with lines of deleteall "ROW1", etc. then pass it to
> > > > > > the
> > > > shell:
> > > > > > hbase shell /tmp/delete_rows.txt.
> > > > > >
> > > > > > But before messing w/ the above, lets get the answers to above
> > > > questions.
> > > > > > Thanks,
> > > > > > S
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > > > M
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>
RE: Removal of table rows.
Posted by sc...@comcast.net.
Marc, what type of data are loading into hbase? curious..
> On 12/17/2020 8:46 AM Marc Hoppins <ma...@eset.sk> wrote:
>
>
> I did restart 'instances' on master(s) so I was surprised to see a lingering row.
>
> -----Original Message-----
> From: Wellington Chevreuil <we...@gmail.com>
> Sent: Thursday, December 17, 2020 1:34 PM
> To: Hbase-User <us...@hbase.apache.org>
> Subject: Re: Removal of table rows.
>
> EXTERNAL
>
> Yes, depending on hbade version, you would need a master restart to effectively clean master cache.
>
> On Thu, 17 Dec 2020, 10:12 Marc Hoppins, <ma...@eset.sk> wrote:
>
> > OK. I had a meandering circuit to get a version of operator-tools
> > built and running.
> >
> > After running it on the meta table
> >
> > sudo hbase hbck -j /tmp/hbase-hbck2-1.1.0-SNAPSHOT.jar
> > extraRegionsInMeta alfa:rfilenameext --fix
> >
> > I have ended up with:-
> >
> > alfa:extgen,F,1608197544264.ba22f6113a0bb9520cee1f7b30050fa7.
> > column=info:state, timestamp=1608197544904, value=OPEN
> > alfa:rfilenameext column=table:state, timestamp=1604493139455,
> > value=\x08\x00
> > alfa:rfiles column=table:state, timestamp=1602600776355,
> > value=\x08\x00
> >
> > I still have one row referring to the missing table. Do I need to
> > restart hbase service to remove this or will it vanish at some time?
> >
> > HBASE on the master still shows the 48 regions in transition when I
> > open the interface. I assume this is because the service has not been restarted.
> >
> > M
> >
> > -----Original Message-----
> > From: Wellington Chevreuil <we...@gmail.com>
> > Sent: Wednesday, December 16, 2020 3:39 PM
> > To: Hbase-User <us...@hbase.apache.org>
> > Subject: Re: Removal of table rows.
> >
> > EXTERNAL
> >
> > >
> > > Do I build hbck2-tools on similar OS & java version? I have been
> > > informed we have 'maven' installed on one host on a cluster, which
> > > is centos and
> > > (probably) a different java, and the build itself is needed for Ubuntu16.
> >
> > I would stick to the same java major version. OS wise, this module
> > doesn't rely on any OS native call, AFAIK, so should work fine between
> > these different versions.
> >
> > Em qua., 16 de dez. de 2020 às 13:30, Marc Hoppins
> > <ma...@eset.sk>
> > escreveu:
> >
> > > Thanks. Once again, a newbie in this regard but,
> > >
> > > Do I build hbck2-tools on similar OS & java version? I have been
> > > informed we have 'maven' installed on one host on a cluster, which
> > > is centos and
> > > (probably) a different java, and the build itself is needed for Ubuntu16.
> > >
> > > -----Original Message-----
> > > From: Wellington Chevreuil <we...@gmail.com>
> > > Sent: Wednesday, December 16, 2020 10:08 AM
> > > To: Hbase-User <us...@hbase.apache.org>
> > > Subject: Re: Removal of table rows.
> > >
> > > EXTERNAL
> > >
> > > >
> > > > Hbase:meta has info for one table region which is NOT on a master.
> > > > Is that correct? I would have expected all meta info to be stored
> > > > on a
> > > master.
> > > >
> > > meta table is an "hbase system table" that has info about which
> > > regions are assigned to which region servers in your cluster. As any
> > > user table, it also has a region, and this region must be hosted on
> > > a region server in the cluster. Master is just responsible for the
> > > coordination of regions and some other housekeeping actions.
> > >
> > > As I state below, a simple method to remove everything connected to
> > > > alfa:rfilenameext would seem the easiest of tasks as the
> > > > namespace:name consistently appears on every relevant row.
> > >
> > > Looks like hbck2 *extraRegionsinMeta *is what you need here, and it
> > > should work with hbase 2.1. It's not available on hbck2 1.0 release,
> > > but you can build a new hbck2 jar out of current master branch, and
> > > that would give you an hbck2 with extraRegionsinMeta. You can do it
> > > by cloning the repo below, then do a *mvn install* from the main module:
> > >
> > > https://github.com/apache/hbase-operator-tools.git
> > >
> > > Em qua., 16 de dez. de 2020 às 08:02, Marc Hoppins
> > > <ma...@eset.sk>
> > > escreveu:
> > >
> > > > Hbase is 2.1.0 (via Cloudera CDH 6.3.2)
> > > >
> > > > The UI page for the master shows NO report links/tabs.
> > > >
> > > > Hbase:meta has info for one table region which is NOT on a master.
> > > > Is that correct? I would have expected all meta info to be stored
> > > > on a
> > > master.
> > > >
> > > > Name Region Server
> > > > Read Requests Write Requests
> > > > Num.Storefiles MemSize Locality
> > > > hbase:meta,,1.1588230740 ba-wtmp05.asgardalfa.hq.com:16030
> > > > 219,168,603 35,419 1 MB 4 0 B
> > > > 1.0
> > > >
> > > > -----Original Message-----
> > > > From: Wellington Chevreuil <we...@gmail.com>
> > > > Sent: Tuesday, December 15, 2020 7:10 PM
> > > > To: Hbase-User <us...@hbase.apache.org>
> > > > Subject: Re: Removal of table rows.
> > > >
> > > > EXTERNAL
> > > >
> > > > >
> > > > > I am an HBASE newbie so I apologise if I am being repetitious.
> > > > >
> > > > > Apologies also if this is not the right group. Am not sure if
> > > > > this may be more suited to 'dev' list.
> > > > >
> > > > Welcome, this is the right channel for this kind of questions.
> > > >
> > > >
> > > > > The solution offered by hbase-operator-tools -
> > > > > extraRegionsinMeta
> > > > > - offered hope. Once again, however, another problem has surfaced:
> > > > > this tools command for extra regions is incompatible with the
> > > > > hbase version we are running.
> > > > >
> > > > This command does not rely on any Master/RegionServer interface,
> > > > so it should not have any incompatibility issues. It only uses
> > > > public client API to cleanup meta table, so maybe you just need to
> > > > build latest hbck2 master branch version? It would work for any
> > > > hbase 2, in theory, it could even work with hbase 1, but that was never tested.
> > > > Luckly, you got Stack's attention, so yeah, if you confirm on his
> > > > previous questions we might be able to help with further directions.
> > > >
> > > > Em ter., 15 de dez. de 2020 às 17:23, Stack <st...@duboce.net>
> > escreveu:
> > > >
> > > > > On Tue, Dec 15, 2020 at 7:46 AM Marc Hoppins
> > > > > <ma...@eset.sk>
> > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I am an HBASE newbie so I apologise if I am being repetitious.
> > > > > >
> > > > > > Apologies also if this is not the right group. Am not sure if
> > > > > > this may be more suited to 'dev' list. However,
> > > > > >
> > > > > > A problem question and a technical/wishlist question.
> > > > > >
> > > > > > Problem:
> > > > > >
> > > > > > I have inherited a problem with an HBASE table. The original
> > > > > > issue may have erupted due to a network outage. A table has
> > > > > > 48 region in
> > > > > transition
> > > > > > operations, stuck that way for several weeks. A previous
> > > > > > attempt to fix things with hbck failed. An attempt to DISABLE
> > > > > > then DROP the table also failed. The four or five attempts to
> > > > > > work the table ALSO now had stuck procedures. Subsequent DFS
> > > > > > and ZOO operations left the situation where there was no data
> > > > > > and no real table: just a 6K file in an empty
> > > > > structure.
> > > > > >
> > > > > >
> > > > > Which version of hbase, do you know Marc? (Look at the base of
> > > > > the master UI. It'll tell you. Sounds like it is an hbase-2.x).
> > > > >
> > > > > What is the name of the 6k file? (I am trying to understand what
> > > > > the file you are referencing is).
> > > > >
> > > > > The table directory was removed from hdfs and zookeeper?
> > > > >
> > > > Yes. The data had already been removed from the previous tech
> > attempts
> > > > to fix the issue.
> > > >
> > > > Thus, the HDFS 6K file structure contained no actual information,
> > > > just empty files and directories. Zkshell had been used to remove
> > znodes.
> > > >
> > > >
> > > > >
> > > > > > When I got to the problem my knowledge of HBASE was nil. It is
> > > > > > little better than that now but anyway...
> > > > > >
> > > > > > Fortunately for me this is a testing/dev cluster. The 'owner'
> > > > > > was content that the table can be removed - and appeared to
> > > > > > have already been
> > > > > done...of
> > > > > > a kind.
> > > > > >
> > > > > > Reading and reading of others' similar issues lead me to the
> > > > > > point I also was going to clean the HDFS data and ZK data for
> > > > > > this table. I shut down HBASE, cleaned HDFS and ZK node data,
> > > > > > deleted the masterprocwals and restarted HBASE.
> > > > > >
> > > > > > When all came up I was happy to see that the affected table
> > > > > > appeared nowhere and that the procedures had all disappeared.
> > > > > >
> > > > > > However, when I hopped to hbase master, even though no table
> > > > > > of that name existed, 48 regions were still in transition.
> > > > > > Further research steered
> > > > > me
> > > > > > toward hbase:meta and sure enough, the references to the RITs
> > > > > > lived
> > > > > happily
> > > > > > among other data for other tables.
> > > > > >
> > > > > >
> > > > > Do you have a 'Procedures and Locks' tab on your master home page?
> > > > > If so, what does it report? Is there an "HBCK Report" tab? If
> > > > > so, what does it show?
> > > > >
> > > >
> > > > When I got to the problem procedures & locks showed one item for
> > > > each problem on that table. After removal of masterprocwals they
> > > > all disappeared. The only entry now is one normal-looking WAL log
> > > > file entry whose timestamp changes according to operation time
> > > > (currently 8.25am today).
> > > >
> > > > >
> > > > > > The solution offered by hbase-operator-tools -
> > > > > > extraRegionsinMeta
> > > > > > - offered hope. Once again, however, another problem has surfaced:
> > > > > > this tools command for extra regions is incompatible with the
> > > > > > hbase version we are running.
> > > > > >
> > > > > > So...
> > > > > >
> > > > > > How can I remove the references to namespace:kaput_table from
> > > > hbase:meta?
> > > > > >
> > > > > >
> > > > > > Sounds like an hbase-2.1.x or a 2.0.x.
> > > > >
> > > > > Will wait on your answers to the above... It might be a crass
> > > > > delete of each row from the hbase:meta table then restart (even
> > > > > then, if procedures in the procedure store, you may have to
> > > > > clear it again as you did above before the restart to purge the
> > > > > procedures as you don't have tooling to do it from cmdline... do
> > > > > you have an 'hbck2 bypass
> > > > --override'?).
> > > > >
> > > > A crass delete of each row would work for me if I could Identify
> > > > how to remove the rows. Sample data:-
> > > >
> > > > alfa:rfilenameext column=table:state, timestamp=1604493139455,
> > > > value=\x08\x00
> > > > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > > > column=info:regioninfo, timestamp=1604388258225, value={ENCODED =>
> > > > 35925292c25898671e5a894ce387e167, NAME =>
> > > > 'alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.'
> > > > ,
> > > > STARTKEY => '', ENDKEY => '0'}
> > > > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > > > column=info:seqnumDuringOpen, timestamp=1601269814633,
> > > > value=\x00\x00\x00\x00\x00\x00\x008
> > > > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > > > column=info:server, timestamp=1601269814633, value=
> > > > ba-wtmp04.asgardalfa.hq.com:16020
> > > > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > > > column=info:serverstartcode, timestamp=1601269814633,
> > > > value=1601061167123
> > > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > > > column=info:sn, timestamp=1604388258050,
> > > > value=ba-wtmp04.asgardalfa.hq.com
> > > > ,16020,1601061167123
> > > > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > > > column=info:state, timestamp=1604388258225, value=CLOSED
> > > > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > > column=info:regioninfo, timestamp=1600969938610, value={ENCODED =>
> > > > 787d1455b84f2d846ce6089392f01fd2, NAME =>
> > > > 'alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > > ',
> > > > STARTKEY => '0', ENDKEY => '1'}
> > > > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > > column=info:seqnumDuringOpen, timestamp=1600780187722,
> > > > value=\x00\x00\x00\x00\x00\x02^\xBB
> > > > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > > column=info:server, timestamp=1600780187722, value=
> > > > ba-wtmp08.asgardalfa.hq.com:16020
> > > > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > > column=info:serverstartcode, timestamp=1600780187722,
> > > > value=1600780162556
> > > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > > column=info:sn, timestamp=1600969938610,
> > > > value=ba-wtmp07.asgardalfa.hq.com
> > > > ,16020,1600936054386
> > > > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > > column=info:state, timestamp=1600969938610, value=OPENING
> > > > alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > > > column=info:regioninfo, timestamp=1601060563980, value={ENCODED =>
> > > > aa9d89b40a9def31a080fdd1776acb4e, NAME =>
> > > > 'alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > > > ',
> > > > STARTKEY => '1', ENDKEY => '2'}
> > > > alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > > > column=info:seqnumDuringOpen, timestamp=1600780186976,
> > > > value=\x00\x00\x00\x00\x00\x02^\xA9
> > > > alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > > > column=info:server, timestamp=1600780186976, value=
> > > > dr1-wtmp02.asgardalfa.hq.com:16020
> > > > alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > > > column=info:serverstartcode, timestamp=1600780186976,
> > > > value=1600780163021
> > > alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > > > column=info:sn, timestamp=1601060563980,
> > > > value=ba-wtmp05.asgardalfa.hq.com
> > > > ,16020,1601049145467
> > > >
> > > >
> > > > As I state below, a simple method to remove everything connected
> > > > to alfa:rfilenameext would seem the easiest of tasks as the
> > > > namespace:name consistently appears on every relevant row.
> > > >
> > > > >
> > > > >
> > > > > > Technical:
> > > > > >
> > > > > > Is there to be any implementation of such a fix within HBASe
> > > > > > itself where table manipulation can be performed by Eg.,
> > > > > >
> > > > > > delete hbase:meta namespace
> > > > > > delete hbase:meta namespace:table
> > > > > >
> > > > > > or even
> > > > > >
> > > > > > scan hbase:meta filter = namespace:table | deleterow
> > > > > >
> > > > > > or some such?
> > > > > >
> > > > > >
> > > > > Scan doesn't return result (unfortunately) so you can assign to
> > > > > a shell variable; it dumps scan output2 on stdout/stderr.
> > > > >
> > > > > I'm sure there is a better way but something like below (don't laugh!
> > > > > and I've not really tried it so be careful):
> > > > >
> > > > > # Get rows for the BAD_TABLENAME from hbase:meta table $ echo
> > > > > 'scan "hbase:meta", {ROWPREFIXFILTER => "BAD_TABLENAME"}'| hbase
> > > > > shell > /tmp/out.txt # Need to get the row only. Above got rows
> > > > > and
> > columns.
> > > > > We don't want to filter on column # because we do not know which
> > > > > columns are still in hbase:meta. Need to filter on table # again
> > > > > to cut the shell preamble and footer out. Unique the rows.
> > > > > $ cat /tmp/out.txt |awk '{print $1}'|grep BAD_TABLENAME | sort
> > > > > -u
> > > > > > /tmp/unique_rows.txt # Now you have rows to delete. For each
> > > > > > in
> > > > > shell do... 'deleteall "ROW_X"'... Could # edit
> > > > > /tmp/unique_rows.txt and per for add .... deleteall " prefix and a "
> > > > > suffix.... to make
> > > > > # a file with lines of deleteall "ROW1", etc. then pass it to
> > > > > the
> > > shell:
> > > > > hbase shell /tmp/delete_rows.txt.
> > > > >
> > > > > But before messing w/ the above, lets get the answers to above
> > > questions.
> > > > > Thanks,
> > > > > S
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > > Thanks
> > > > > >
> > > > > > M
> > > > > >
> > > > >
> > > >
> > >
> >
RE: Removal of table rows.
Posted by Marc Hoppins <ma...@eset.sk>.
I did restart 'instances' on master(s) so I was surprised to see a lingering row.
-----Original Message-----
From: Wellington Chevreuil <we...@gmail.com>
Sent: Thursday, December 17, 2020 1:34 PM
To: Hbase-User <us...@hbase.apache.org>
Subject: Re: Removal of table rows.
EXTERNAL
Yes, depending on hbade version, you would need a master restart to effectively clean master cache.
On Thu, 17 Dec 2020, 10:12 Marc Hoppins, <ma...@eset.sk> wrote:
> OK. I had a meandering circuit to get a version of operator-tools
> built and running.
>
> After running it on the meta table
>
> sudo hbase hbck -j /tmp/hbase-hbck2-1.1.0-SNAPSHOT.jar
> extraRegionsInMeta alfa:rfilenameext --fix
>
> I have ended up with:-
>
> alfa:extgen,F,1608197544264.ba22f6113a0bb9520cee1f7b30050fa7.
> column=info:state, timestamp=1608197544904, value=OPEN
> alfa:rfilenameext column=table:state, timestamp=1604493139455,
> value=\x08\x00
> alfa:rfiles column=table:state, timestamp=1602600776355,
> value=\x08\x00
>
> I still have one row referring to the missing table. Do I need to
> restart hbase service to remove this or will it vanish at some time?
>
> HBASE on the master still shows the 48 regions in transition when I
> open the interface. I assume this is because the service has not been restarted.
>
> M
>
> -----Original Message-----
> From: Wellington Chevreuil <we...@gmail.com>
> Sent: Wednesday, December 16, 2020 3:39 PM
> To: Hbase-User <us...@hbase.apache.org>
> Subject: Re: Removal of table rows.
>
> EXTERNAL
>
> >
> > Do I build hbck2-tools on similar OS & java version? I have been
> > informed we have 'maven' installed on one host on a cluster, which
> > is centos and
> > (probably) a different java, and the build itself is needed for Ubuntu16.
>
> I would stick to the same java major version. OS wise, this module
> doesn't rely on any OS native call, AFAIK, so should work fine between
> these different versions.
>
> Em qua., 16 de dez. de 2020 às 13:30, Marc Hoppins
> <ma...@eset.sk>
> escreveu:
>
> > Thanks. Once again, a newbie in this regard but,
> >
> > Do I build hbck2-tools on similar OS & java version? I have been
> > informed we have 'maven' installed on one host on a cluster, which
> > is centos and
> > (probably) a different java, and the build itself is needed for Ubuntu16.
> >
> > -----Original Message-----
> > From: Wellington Chevreuil <we...@gmail.com>
> > Sent: Wednesday, December 16, 2020 10:08 AM
> > To: Hbase-User <us...@hbase.apache.org>
> > Subject: Re: Removal of table rows.
> >
> > EXTERNAL
> >
> > >
> > > Hbase:meta has info for one table region which is NOT on a master.
> > > Is that correct? I would have expected all meta info to be stored
> > > on a
> > master.
> > >
> > meta table is an "hbase system table" that has info about which
> > regions are assigned to which region servers in your cluster. As any
> > user table, it also has a region, and this region must be hosted on
> > a region server in the cluster. Master is just responsible for the
> > coordination of regions and some other housekeeping actions.
> >
> > As I state below, a simple method to remove everything connected to
> > > alfa:rfilenameext would seem the easiest of tasks as the
> > > namespace:name consistently appears on every relevant row.
> >
> > Looks like hbck2 *extraRegionsinMeta *is what you need here, and it
> > should work with hbase 2.1. It's not available on hbck2 1.0 release,
> > but you can build a new hbck2 jar out of current master branch, and
> > that would give you an hbck2 with extraRegionsinMeta. You can do it
> > by cloning the repo below, then do a *mvn install* from the main module:
> >
> > https://github.com/apache/hbase-operator-tools.git
> >
> > Em qua., 16 de dez. de 2020 às 08:02, Marc Hoppins
> > <ma...@eset.sk>
> > escreveu:
> >
> > > Hbase is 2.1.0 (via Cloudera CDH 6.3.2)
> > >
> > > The UI page for the master shows NO report links/tabs.
> > >
> > > Hbase:meta has info for one table region which is NOT on a master.
> > > Is that correct? I would have expected all meta info to be stored
> > > on a
> > master.
> > >
> > > Name Region Server
> > > Read Requests Write Requests
> > > Num.Storefiles MemSize Locality
> > > hbase:meta,,1.1588230740 ba-wtmp05.asgardalfa.hq.com:16030
> > > 219,168,603 35,419 1 MB 4 0 B
> > > 1.0
> > >
> > > -----Original Message-----
> > > From: Wellington Chevreuil <we...@gmail.com>
> > > Sent: Tuesday, December 15, 2020 7:10 PM
> > > To: Hbase-User <us...@hbase.apache.org>
> > > Subject: Re: Removal of table rows.
> > >
> > > EXTERNAL
> > >
> > > >
> > > > I am an HBASE newbie so I apologise if I am being repetitious.
> > > >
> > > > Apologies also if this is not the right group. Am not sure if
> > > > this may be more suited to 'dev' list.
> > > >
> > > Welcome, this is the right channel for this kind of questions.
> > >
> > >
> > > > The solution offered by hbase-operator-tools -
> > > > extraRegionsinMeta
> > > > - offered hope. Once again, however, another problem has surfaced:
> > > > this tools command for extra regions is incompatible with the
> > > > hbase version we are running.
> > > >
> > > This command does not rely on any Master/RegionServer interface,
> > > so it should not have any incompatibility issues. It only uses
> > > public client API to cleanup meta table, so maybe you just need to
> > > build latest hbck2 master branch version? It would work for any
> > > hbase 2, in theory, it could even work with hbase 1, but that was never tested.
> > > Luckly, you got Stack's attention, so yeah, if you confirm on his
> > > previous questions we might be able to help with further directions.
> > >
> > > Em ter., 15 de dez. de 2020 às 17:23, Stack <st...@duboce.net>
> escreveu:
> > >
> > > > On Tue, Dec 15, 2020 at 7:46 AM Marc Hoppins
> > > > <ma...@eset.sk>
> > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I am an HBASE newbie so I apologise if I am being repetitious.
> > > > >
> > > > > Apologies also if this is not the right group. Am not sure if
> > > > > this may be more suited to 'dev' list. However,
> > > > >
> > > > > A problem question and a technical/wishlist question.
> > > > >
> > > > > Problem:
> > > > >
> > > > > I have inherited a problem with an HBASE table. The original
> > > > > issue may have erupted due to a network outage. A table has
> > > > > 48 region in
> > > > transition
> > > > > operations, stuck that way for several weeks. A previous
> > > > > attempt to fix things with hbck failed. An attempt to DISABLE
> > > > > then DROP the table also failed. The four or five attempts to
> > > > > work the table ALSO now had stuck procedures. Subsequent DFS
> > > > > and ZOO operations left the situation where there was no data
> > > > > and no real table: just a 6K file in an empty
> > > > structure.
> > > > >
> > > > >
> > > > Which version of hbase, do you know Marc? (Look at the base of
> > > > the master UI. It'll tell you. Sounds like it is an hbase-2.x).
> > > >
> > > > What is the name of the 6k file? (I am trying to understand what
> > > > the file you are referencing is).
> > > >
> > > > The table directory was removed from hdfs and zookeeper?
> > > >
> > > Yes. The data had already been removed from the previous tech
> attempts
> > > to fix the issue.
> > >
> > > Thus, the HDFS 6K file structure contained no actual information,
> > > just empty files and directories. Zkshell had been used to remove
> znodes.
> > >
> > >
> > > >
> > > > > When I got to the problem my knowledge of HBASE was nil. It is
> > > > > little better than that now but anyway...
> > > > >
> > > > > Fortunately for me this is a testing/dev cluster. The 'owner'
> > > > > was content that the table can be removed - and appeared to
> > > > > have already been
> > > > done...of
> > > > > a kind.
> > > > >
> > > > > Reading and reading of others' similar issues lead me to the
> > > > > point I also was going to clean the HDFS data and ZK data for
> > > > > this table. I shut down HBASE, cleaned HDFS and ZK node data,
> > > > > deleted the masterprocwals and restarted HBASE.
> > > > >
> > > > > When all came up I was happy to see that the affected table
> > > > > appeared nowhere and that the procedures had all disappeared.
> > > > >
> > > > > However, when I hopped to hbase master, even though no table
> > > > > of that name existed, 48 regions were still in transition.
> > > > > Further research steered
> > > > me
> > > > > toward hbase:meta and sure enough, the references to the RITs
> > > > > lived
> > > > happily
> > > > > among other data for other tables.
> > > > >
> > > > >
> > > > Do you have a 'Procedures and Locks' tab on your master home page?
> > > > If so, what does it report? Is there an "HBCK Report" tab? If
> > > > so, what does it show?
> > > >
> > >
> > > When I got to the problem procedures & locks showed one item for
> > > each problem on that table. After removal of masterprocwals they
> > > all disappeared. The only entry now is one normal-looking WAL log
> > > file entry whose timestamp changes according to operation time
> > > (currently 8.25am today).
> > >
> > > >
> > > > > The solution offered by hbase-operator-tools -
> > > > > extraRegionsinMeta
> > > > > - offered hope. Once again, however, another problem has surfaced:
> > > > > this tools command for extra regions is incompatible with the
> > > > > hbase version we are running.
> > > > >
> > > > > So...
> > > > >
> > > > > How can I remove the references to namespace:kaput_table from
> > > hbase:meta?
> > > > >
> > > > >
> > > > > Sounds like an hbase-2.1.x or a 2.0.x.
> > > >
> > > > Will wait on your answers to the above... It might be a crass
> > > > delete of each row from the hbase:meta table then restart (even
> > > > then, if procedures in the procedure store, you may have to
> > > > clear it again as you did above before the restart to purge the
> > > > procedures as you don't have tooling to do it from cmdline... do
> > > > you have an 'hbck2 bypass
> > > --override'?).
> > > >
> > > A crass delete of each row would work for me if I could Identify
> > > how to remove the rows. Sample data:-
> > >
> > > alfa:rfilenameext column=table:state, timestamp=1604493139455,
> > > value=\x08\x00
> > > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > > column=info:regioninfo, timestamp=1604388258225, value={ENCODED =>
> > > 35925292c25898671e5a894ce387e167, NAME =>
> > > 'alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.'
> > > ,
> > > STARTKEY => '', ENDKEY => '0'}
> > > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > > column=info:seqnumDuringOpen, timestamp=1601269814633,
> > > value=\x00\x00\x00\x00\x00\x00\x008
> > > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > > column=info:server, timestamp=1601269814633, value=
> > > ba-wtmp04.asgardalfa.hq.com:16020
> > > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > > column=info:serverstartcode, timestamp=1601269814633,
> > > value=1601061167123
> > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > > column=info:sn, timestamp=1604388258050,
> > > value=ba-wtmp04.asgardalfa.hq.com
> > > ,16020,1601061167123
> > > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > > column=info:state, timestamp=1604388258225, value=CLOSED
> > > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > column=info:regioninfo, timestamp=1600969938610, value={ENCODED =>
> > > 787d1455b84f2d846ce6089392f01fd2, NAME =>
> > > 'alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > ',
> > > STARTKEY => '0', ENDKEY => '1'}
> > > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > column=info:seqnumDuringOpen, timestamp=1600780187722,
> > > value=\x00\x00\x00\x00\x00\x02^\xBB
> > > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > column=info:server, timestamp=1600780187722, value=
> > > ba-wtmp08.asgardalfa.hq.com:16020
> > > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > column=info:serverstartcode, timestamp=1600780187722,
> > > value=1600780162556
> > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > column=info:sn, timestamp=1600969938610,
> > > value=ba-wtmp07.asgardalfa.hq.com
> > > ,16020,1600936054386
> > > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > column=info:state, timestamp=1600969938610, value=OPENING
> > > alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > > column=info:regioninfo, timestamp=1601060563980, value={ENCODED =>
> > > aa9d89b40a9def31a080fdd1776acb4e, NAME =>
> > > 'alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > > ',
> > > STARTKEY => '1', ENDKEY => '2'}
> > > alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > > column=info:seqnumDuringOpen, timestamp=1600780186976,
> > > value=\x00\x00\x00\x00\x00\x02^\xA9
> > > alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > > column=info:server, timestamp=1600780186976, value=
> > > dr1-wtmp02.asgardalfa.hq.com:16020
> > > alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > > column=info:serverstartcode, timestamp=1600780186976,
> > > value=1600780163021
> > alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > > column=info:sn, timestamp=1601060563980,
> > > value=ba-wtmp05.asgardalfa.hq.com
> > > ,16020,1601049145467
> > >
> > >
> > > As I state below, a simple method to remove everything connected
> > > to alfa:rfilenameext would seem the easiest of tasks as the
> > > namespace:name consistently appears on every relevant row.
> > >
> > > >
> > > >
> > > > > Technical:
> > > > >
> > > > > Is there to be any implementation of such a fix within HBASe
> > > > > itself where table manipulation can be performed by Eg.,
> > > > >
> > > > > delete hbase:meta namespace
> > > > > delete hbase:meta namespace:table
> > > > >
> > > > > or even
> > > > >
> > > > > scan hbase:meta filter = namespace:table | deleterow
> > > > >
> > > > > or some such?
> > > > >
> > > > >
> > > > Scan doesn't return result (unfortunately) so you can assign to
> > > > a shell variable; it dumps scan output2 on stdout/stderr.
> > > >
> > > > I'm sure there is a better way but something like below (don't laugh!
> > > > and I've not really tried it so be careful):
> > > >
> > > > # Get rows for the BAD_TABLENAME from hbase:meta table $ echo
> > > > 'scan "hbase:meta", {ROWPREFIXFILTER => "BAD_TABLENAME"}'| hbase
> > > > shell > /tmp/out.txt # Need to get the row only. Above got rows
> > > > and
> columns.
> > > > We don't want to filter on column # because we do not know which
> > > > columns are still in hbase:meta. Need to filter on table # again
> > > > to cut the shell preamble and footer out. Unique the rows.
> > > > $ cat /tmp/out.txt |awk '{print $1}'|grep BAD_TABLENAME | sort
> > > > -u
> > > > > /tmp/unique_rows.txt # Now you have rows to delete. For each
> > > > > in
> > > > shell do... 'deleteall "ROW_X"'... Could # edit
> > > > /tmp/unique_rows.txt and per for add .... deleteall " prefix and a "
> > > > suffix.... to make
> > > > # a file with lines of deleteall "ROW1", etc. then pass it to
> > > > the
> > shell:
> > > > hbase shell /tmp/delete_rows.txt.
> > > >
> > > > But before messing w/ the above, lets get the answers to above
> > questions.
> > > > Thanks,
> > > > S
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > > Thanks
> > > > >
> > > > > M
> > > > >
> > > >
> > >
> >
>
Re: Removal of table rows.
Posted by Wellington Chevreuil <we...@gmail.com>.
Yes, depending on hbade version, you would need a master restart to
effectively clean master cache.
On Thu, 17 Dec 2020, 10:12 Marc Hoppins, <ma...@eset.sk> wrote:
> OK. I had a meandering circuit to get a version of operator-tools built
> and running.
>
> After running it on the meta table
>
> sudo hbase hbck -j /tmp/hbase-hbck2-1.1.0-SNAPSHOT.jar extraRegionsInMeta
> alfa:rfilenameext --fix
>
> I have ended up with:-
>
> alfa:extgen,F,1608197544264.ba22f6113a0bb9520cee1f7b30050fa7.
> column=info:state, timestamp=1608197544904, value=OPEN
> alfa:rfilenameext column=table:state, timestamp=1604493139455,
> value=\x08\x00
> alfa:rfiles column=table:state, timestamp=1602600776355, value=\x08\x00
>
> I still have one row referring to the missing table. Do I need to restart
> hbase service to remove this or will it vanish at some time?
>
> HBASE on the master still shows the 48 regions in transition when I open
> the interface. I assume this is because the service has not been restarted.
>
> M
>
> -----Original Message-----
> From: Wellington Chevreuil <we...@gmail.com>
> Sent: Wednesday, December 16, 2020 3:39 PM
> To: Hbase-User <us...@hbase.apache.org>
> Subject: Re: Removal of table rows.
>
> EXTERNAL
>
> >
> > Do I build hbck2-tools on similar OS & java version? I have been
> > informed we have 'maven' installed on one host on a cluster, which is
> > centos and
> > (probably) a different java, and the build itself is needed for Ubuntu16.
>
> I would stick to the same java major version. OS wise, this module doesn't
> rely on any OS native call, AFAIK, so should work fine between these
> different versions.
>
> Em qua., 16 de dez. de 2020 às 13:30, Marc Hoppins <ma...@eset.sk>
> escreveu:
>
> > Thanks. Once again, a newbie in this regard but,
> >
> > Do I build hbck2-tools on similar OS & java version? I have been
> > informed we have 'maven' installed on one host on a cluster, which is
> > centos and
> > (probably) a different java, and the build itself is needed for Ubuntu16.
> >
> > -----Original Message-----
> > From: Wellington Chevreuil <we...@gmail.com>
> > Sent: Wednesday, December 16, 2020 10:08 AM
> > To: Hbase-User <us...@hbase.apache.org>
> > Subject: Re: Removal of table rows.
> >
> > EXTERNAL
> >
> > >
> > > Hbase:meta has info for one table region which is NOT on a master.
> > > Is that correct? I would have expected all meta info to be stored on
> > > a
> > master.
> > >
> > meta table is an "hbase system table" that has info about which
> > regions are assigned to which region servers in your cluster. As any
> > user table, it also has a region, and this region must be hosted on a
> > region server in the cluster. Master is just responsible for the
> > coordination of regions and some other housekeeping actions.
> >
> > As I state below, a simple method to remove everything connected to
> > > alfa:rfilenameext would seem the easiest of tasks as the
> > > namespace:name consistently appears on every relevant row.
> >
> > Looks like hbck2 *extraRegionsinMeta *is what you need here, and it
> > should work with hbase 2.1. It's not available on hbck2 1.0 release,
> > but you can build a new hbck2 jar out of current master branch, and
> > that would give you an hbck2 with extraRegionsinMeta. You can do it by
> > cloning the repo below, then do a *mvn install* from the main module:
> >
> > https://github.com/apache/hbase-operator-tools.git
> >
> > Em qua., 16 de dez. de 2020 às 08:02, Marc Hoppins
> > <ma...@eset.sk>
> > escreveu:
> >
> > > Hbase is 2.1.0 (via Cloudera CDH 6.3.2)
> > >
> > > The UI page for the master shows NO report links/tabs.
> > >
> > > Hbase:meta has info for one table region which is NOT on a master.
> > > Is that correct? I would have expected all meta info to be stored on
> > > a
> > master.
> > >
> > > Name Region Server
> > > Read Requests Write Requests
> > > Num.Storefiles MemSize Locality
> > > hbase:meta,,1.1588230740 ba-wtmp05.asgardalfa.hq.com:16030
> > > 219,168,603 35,419 1 MB 4 0 B
> > > 1.0
> > >
> > > -----Original Message-----
> > > From: Wellington Chevreuil <we...@gmail.com>
> > > Sent: Tuesday, December 15, 2020 7:10 PM
> > > To: Hbase-User <us...@hbase.apache.org>
> > > Subject: Re: Removal of table rows.
> > >
> > > EXTERNAL
> > >
> > > >
> > > > I am an HBASE newbie so I apologise if I am being repetitious.
> > > >
> > > > Apologies also if this is not the right group. Am not sure if this
> > > > may be more suited to 'dev' list.
> > > >
> > > Welcome, this is the right channel for this kind of questions.
> > >
> > >
> > > > The solution offered by hbase-operator-tools - extraRegionsinMeta
> > > > - offered hope. Once again, however, another problem has surfaced:
> > > > this tools command for extra regions is incompatible with the
> > > > hbase version we are running.
> > > >
> > > This command does not rely on any Master/RegionServer interface, so
> > > it should not have any incompatibility issues. It only uses public
> > > client API to cleanup meta table, so maybe you just need to build
> > > latest hbck2 master branch version? It would work for any hbase 2,
> > > in theory, it could even work with hbase 1, but that was never tested.
> > > Luckly, you got Stack's attention, so yeah, if you confirm on his
> > > previous questions we might be able to help with further directions.
> > >
> > > Em ter., 15 de dez. de 2020 às 17:23, Stack <st...@duboce.net>
> escreveu:
> > >
> > > > On Tue, Dec 15, 2020 at 7:46 AM Marc Hoppins
> > > > <ma...@eset.sk>
> > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I am an HBASE newbie so I apologise if I am being repetitious.
> > > > >
> > > > > Apologies also if this is not the right group. Am not sure if
> > > > > this may be more suited to 'dev' list. However,
> > > > >
> > > > > A problem question and a technical/wishlist question.
> > > > >
> > > > > Problem:
> > > > >
> > > > > I have inherited a problem with an HBASE table. The original
> > > > > issue may have erupted due to a network outage. A table has 48
> > > > > region in
> > > > transition
> > > > > operations, stuck that way for several weeks. A previous
> > > > > attempt to fix things with hbck failed. An attempt to DISABLE
> > > > > then DROP the table also failed. The four or five attempts to
> > > > > work the table ALSO now had stuck procedures. Subsequent DFS
> > > > > and ZOO operations left the situation where there was no data
> > > > > and no real table: just a 6K file in an empty
> > > > structure.
> > > > >
> > > > >
> > > > Which version of hbase, do you know Marc? (Look at the base of the
> > > > master UI. It'll tell you. Sounds like it is an hbase-2.x).
> > > >
> > > > What is the name of the 6k file? (I am trying to understand what
> > > > the file you are referencing is).
> > > >
> > > > The table directory was removed from hdfs and zookeeper?
> > > >
> > > Yes. The data had already been removed from the previous tech
> attempts
> > > to fix the issue.
> > >
> > > Thus, the HDFS 6K file structure contained no actual information,
> > > just empty files and directories. Zkshell had been used to remove
> znodes.
> > >
> > >
> > > >
> > > > > When I got to the problem my knowledge of HBASE was nil. It is
> > > > > little better than that now but anyway...
> > > > >
> > > > > Fortunately for me this is a testing/dev cluster. The 'owner'
> > > > > was content that the table can be removed - and appeared to have
> > > > > already been
> > > > done...of
> > > > > a kind.
> > > > >
> > > > > Reading and reading of others' similar issues lead me to the
> > > > > point I also was going to clean the HDFS data and ZK data for
> > > > > this table. I shut down HBASE, cleaned HDFS and ZK node data,
> > > > > deleted the masterprocwals and restarted HBASE.
> > > > >
> > > > > When all came up I was happy to see that the affected table
> > > > > appeared nowhere and that the procedures had all disappeared.
> > > > >
> > > > > However, when I hopped to hbase master, even though no table of
> > > > > that name existed, 48 regions were still in transition. Further
> > > > > research steered
> > > > me
> > > > > toward hbase:meta and sure enough, the references to the RITs
> > > > > lived
> > > > happily
> > > > > among other data for other tables.
> > > > >
> > > > >
> > > > Do you have a 'Procedures and Locks' tab on your master home page?
> > > > If so, what does it report? Is there an "HBCK Report" tab? If so,
> > > > what does it show?
> > > >
> > >
> > > When I got to the problem procedures & locks showed one item for
> > > each problem on that table. After removal of masterprocwals they all
> > > disappeared. The only entry now is one normal-looking WAL log file
> > > entry whose timestamp changes according to operation time (currently
> > > 8.25am today).
> > >
> > > >
> > > > > The solution offered by hbase-operator-tools -
> > > > > extraRegionsinMeta
> > > > > - offered hope. Once again, however, another problem has surfaced:
> > > > > this tools command for extra regions is incompatible with the
> > > > > hbase version we are running.
> > > > >
> > > > > So...
> > > > >
> > > > > How can I remove the references to namespace:kaput_table from
> > > hbase:meta?
> > > > >
> > > > >
> > > > > Sounds like an hbase-2.1.x or a 2.0.x.
> > > >
> > > > Will wait on your answers to the above... It might be a crass
> > > > delete of each row from the hbase:meta table then restart (even
> > > > then, if procedures in the procedure store, you may have to clear
> > > > it again as you did above before the restart to purge the
> > > > procedures as you don't have tooling to do it from cmdline... do
> > > > you have an 'hbck2 bypass
> > > --override'?).
> > > >
> > > A crass delete of each row would work for me if I could Identify how
> > > to remove the rows. Sample data:-
> > >
> > > alfa:rfilenameext column=table:state, timestamp=1604493139455,
> > > value=\x08\x00
> > > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > > column=info:regioninfo, timestamp=1604388258225, value={ENCODED =>
> > > 35925292c25898671e5a894ce387e167, NAME =>
> > > 'alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.'
> > > ,
> > > STARTKEY => '', ENDKEY => '0'}
> > > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > > column=info:seqnumDuringOpen, timestamp=1601269814633,
> > > value=\x00\x00\x00\x00\x00\x00\x008
> > > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > > column=info:server, timestamp=1601269814633, value=
> > > ba-wtmp04.asgardalfa.hq.com:16020
> > > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > > column=info:serverstartcode, timestamp=1601269814633,
> > > value=1601061167123
> > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > > column=info:sn, timestamp=1604388258050,
> > > value=ba-wtmp04.asgardalfa.hq.com
> > > ,16020,1601061167123
> > > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > > column=info:state, timestamp=1604388258225, value=CLOSED
> > > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > column=info:regioninfo, timestamp=1600969938610, value={ENCODED =>
> > > 787d1455b84f2d846ce6089392f01fd2, NAME =>
> > > 'alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > ',
> > > STARTKEY => '0', ENDKEY => '1'}
> > > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > column=info:seqnumDuringOpen, timestamp=1600780187722,
> > > value=\x00\x00\x00\x00\x00\x02^\xBB
> > > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > column=info:server, timestamp=1600780187722, value=
> > > ba-wtmp08.asgardalfa.hq.com:16020
> > > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > column=info:serverstartcode, timestamp=1600780187722,
> > > value=1600780162556
> > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > column=info:sn, timestamp=1600969938610,
> > > value=ba-wtmp07.asgardalfa.hq.com
> > > ,16020,1600936054386
> > > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > > column=info:state, timestamp=1600969938610, value=OPENING
> > > alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > > column=info:regioninfo, timestamp=1601060563980, value={ENCODED =>
> > > aa9d89b40a9def31a080fdd1776acb4e, NAME =>
> > > 'alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > > ',
> > > STARTKEY => '1', ENDKEY => '2'}
> > > alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > > column=info:seqnumDuringOpen, timestamp=1600780186976,
> > > value=\x00\x00\x00\x00\x00\x02^\xA9
> > > alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > > column=info:server, timestamp=1600780186976, value=
> > > dr1-wtmp02.asgardalfa.hq.com:16020
> > > alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > > column=info:serverstartcode, timestamp=1600780186976,
> > > value=1600780163021
> > alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > > column=info:sn, timestamp=1601060563980,
> > > value=ba-wtmp05.asgardalfa.hq.com
> > > ,16020,1601049145467
> > >
> > >
> > > As I state below, a simple method to remove everything connected to
> > > alfa:rfilenameext would seem the easiest of tasks as the
> > > namespace:name consistently appears on every relevant row.
> > >
> > > >
> > > >
> > > > > Technical:
> > > > >
> > > > > Is there to be any implementation of such a fix within HBASe
> > > > > itself where table manipulation can be performed by Eg.,
> > > > >
> > > > > delete hbase:meta namespace
> > > > > delete hbase:meta namespace:table
> > > > >
> > > > > or even
> > > > >
> > > > > scan hbase:meta filter = namespace:table | deleterow
> > > > >
> > > > > or some such?
> > > > >
> > > > >
> > > > Scan doesn't return result (unfortunately) so you can assign to a
> > > > shell variable; it dumps scan output2 on stdout/stderr.
> > > >
> > > > I'm sure there is a better way but something like below (don't laugh!
> > > > and I've not really tried it so be careful):
> > > >
> > > > # Get rows for the BAD_TABLENAME from hbase:meta table $ echo
> > > > 'scan "hbase:meta", {ROWPREFIXFILTER => "BAD_TABLENAME"}'| hbase
> > > > shell > /tmp/out.txt # Need to get the row only. Above got rows and
> columns.
> > > > We don't want to filter on column # because we do not know which
> > > > columns are still in hbase:meta. Need to filter on table # again
> > > > to cut the shell preamble and footer out. Unique the rows.
> > > > $ cat /tmp/out.txt |awk '{print $1}'|grep BAD_TABLENAME | sort -u
> > > > > /tmp/unique_rows.txt # Now you have rows to delete. For each in
> > > > shell do... 'deleteall "ROW_X"'... Could # edit
> > > > /tmp/unique_rows.txt and per for add .... deleteall " prefix and a "
> > > > suffix.... to make
> > > > # a file with lines of deleteall "ROW1", etc. then pass it to the
> > shell:
> > > > hbase shell /tmp/delete_rows.txt.
> > > >
> > > > But before messing w/ the above, lets get the answers to above
> > questions.
> > > > Thanks,
> > > > S
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > > Thanks
> > > > >
> > > > > M
> > > > >
> > > >
> > >
> >
>
RE: Removal of table rows.
Posted by Marc Hoppins <ma...@eset.sk>.
OK. I had a meandering circuit to get a version of operator-tools built and running.
After running it on the meta table
sudo hbase hbck -j /tmp/hbase-hbck2-1.1.0-SNAPSHOT.jar extraRegionsInMeta alfa:rfilenameext --fix
I have ended up with:-
alfa:extgen,F,1608197544264.ba22f6113a0bb9520cee1f7b30050fa7. column=info:state, timestamp=1608197544904, value=OPEN
alfa:rfilenameext column=table:state, timestamp=1604493139455, value=\x08\x00
alfa:rfiles column=table:state, timestamp=1602600776355, value=\x08\x00
I still have one row referring to the missing table. Do I need to restart hbase service to remove this or will it vanish at some time?
HBASE on the master still shows the 48 regions in transition when I open the interface. I assume this is because the service has not been restarted.
M
-----Original Message-----
From: Wellington Chevreuil <we...@gmail.com>
Sent: Wednesday, December 16, 2020 3:39 PM
To: Hbase-User <us...@hbase.apache.org>
Subject: Re: Removal of table rows.
EXTERNAL
>
> Do I build hbck2-tools on similar OS & java version? I have been
> informed we have 'maven' installed on one host on a cluster, which is
> centos and
> (probably) a different java, and the build itself is needed for Ubuntu16.
I would stick to the same java major version. OS wise, this module doesn't rely on any OS native call, AFAIK, so should work fine between these different versions.
Em qua., 16 de dez. de 2020 às 13:30, Marc Hoppins <ma...@eset.sk>
escreveu:
> Thanks. Once again, a newbie in this regard but,
>
> Do I build hbck2-tools on similar OS & java version? I have been
> informed we have 'maven' installed on one host on a cluster, which is
> centos and
> (probably) a different java, and the build itself is needed for Ubuntu16.
>
> -----Original Message-----
> From: Wellington Chevreuil <we...@gmail.com>
> Sent: Wednesday, December 16, 2020 10:08 AM
> To: Hbase-User <us...@hbase.apache.org>
> Subject: Re: Removal of table rows.
>
> EXTERNAL
>
> >
> > Hbase:meta has info for one table region which is NOT on a master.
> > Is that correct? I would have expected all meta info to be stored on
> > a
> master.
> >
> meta table is an "hbase system table" that has info about which
> regions are assigned to which region servers in your cluster. As any
> user table, it also has a region, and this region must be hosted on a
> region server in the cluster. Master is just responsible for the
> coordination of regions and some other housekeeping actions.
>
> As I state below, a simple method to remove everything connected to
> > alfa:rfilenameext would seem the easiest of tasks as the
> > namespace:name consistently appears on every relevant row.
>
> Looks like hbck2 *extraRegionsinMeta *is what you need here, and it
> should work with hbase 2.1. It's not available on hbck2 1.0 release,
> but you can build a new hbck2 jar out of current master branch, and
> that would give you an hbck2 with extraRegionsinMeta. You can do it by
> cloning the repo below, then do a *mvn install* from the main module:
>
> https://github.com/apache/hbase-operator-tools.git
>
> Em qua., 16 de dez. de 2020 às 08:02, Marc Hoppins
> <ma...@eset.sk>
> escreveu:
>
> > Hbase is 2.1.0 (via Cloudera CDH 6.3.2)
> >
> > The UI page for the master shows NO report links/tabs.
> >
> > Hbase:meta has info for one table region which is NOT on a master.
> > Is that correct? I would have expected all meta info to be stored on
> > a
> master.
> >
> > Name Region Server
> > Read Requests Write Requests
> > Num.Storefiles MemSize Locality
> > hbase:meta,,1.1588230740 ba-wtmp05.asgardalfa.hq.com:16030
> > 219,168,603 35,419 1 MB 4 0 B
> > 1.0
> >
> > -----Original Message-----
> > From: Wellington Chevreuil <we...@gmail.com>
> > Sent: Tuesday, December 15, 2020 7:10 PM
> > To: Hbase-User <us...@hbase.apache.org>
> > Subject: Re: Removal of table rows.
> >
> > EXTERNAL
> >
> > >
> > > I am an HBASE newbie so I apologise if I am being repetitious.
> > >
> > > Apologies also if this is not the right group. Am not sure if this
> > > may be more suited to 'dev' list.
> > >
> > Welcome, this is the right channel for this kind of questions.
> >
> >
> > > The solution offered by hbase-operator-tools - extraRegionsinMeta
> > > - offered hope. Once again, however, another problem has surfaced:
> > > this tools command for extra regions is incompatible with the
> > > hbase version we are running.
> > >
> > This command does not rely on any Master/RegionServer interface, so
> > it should not have any incompatibility issues. It only uses public
> > client API to cleanup meta table, so maybe you just need to build
> > latest hbck2 master branch version? It would work for any hbase 2,
> > in theory, it could even work with hbase 1, but that was never tested.
> > Luckly, you got Stack's attention, so yeah, if you confirm on his
> > previous questions we might be able to help with further directions.
> >
> > Em ter., 15 de dez. de 2020 às 17:23, Stack <st...@duboce.net> escreveu:
> >
> > > On Tue, Dec 15, 2020 at 7:46 AM Marc Hoppins
> > > <ma...@eset.sk>
> > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I am an HBASE newbie so I apologise if I am being repetitious.
> > > >
> > > > Apologies also if this is not the right group. Am not sure if
> > > > this may be more suited to 'dev' list. However,
> > > >
> > > > A problem question and a technical/wishlist question.
> > > >
> > > > Problem:
> > > >
> > > > I have inherited a problem with an HBASE table. The original
> > > > issue may have erupted due to a network outage. A table has 48
> > > > region in
> > > transition
> > > > operations, stuck that way for several weeks. A previous
> > > > attempt to fix things with hbck failed. An attempt to DISABLE
> > > > then DROP the table also failed. The four or five attempts to
> > > > work the table ALSO now had stuck procedures. Subsequent DFS
> > > > and ZOO operations left the situation where there was no data
> > > > and no real table: just a 6K file in an empty
> > > structure.
> > > >
> > > >
> > > Which version of hbase, do you know Marc? (Look at the base of the
> > > master UI. It'll tell you. Sounds like it is an hbase-2.x).
> > >
> > > What is the name of the 6k file? (I am trying to understand what
> > > the file you are referencing is).
> > >
> > > The table directory was removed from hdfs and zookeeper?
> > >
> > Yes. The data had already been removed from the previous tech attempts
> > to fix the issue.
> >
> > Thus, the HDFS 6K file structure contained no actual information,
> > just empty files and directories. Zkshell had been used to remove znodes.
> >
> >
> > >
> > > > When I got to the problem my knowledge of HBASE was nil. It is
> > > > little better than that now but anyway...
> > > >
> > > > Fortunately for me this is a testing/dev cluster. The 'owner'
> > > > was content that the table can be removed - and appeared to have
> > > > already been
> > > done...of
> > > > a kind.
> > > >
> > > > Reading and reading of others' similar issues lead me to the
> > > > point I also was going to clean the HDFS data and ZK data for
> > > > this table. I shut down HBASE, cleaned HDFS and ZK node data,
> > > > deleted the masterprocwals and restarted HBASE.
> > > >
> > > > When all came up I was happy to see that the affected table
> > > > appeared nowhere and that the procedures had all disappeared.
> > > >
> > > > However, when I hopped to hbase master, even though no table of
> > > > that name existed, 48 regions were still in transition. Further
> > > > research steered
> > > me
> > > > toward hbase:meta and sure enough, the references to the RITs
> > > > lived
> > > happily
> > > > among other data for other tables.
> > > >
> > > >
> > > Do you have a 'Procedures and Locks' tab on your master home page?
> > > If so, what does it report? Is there an "HBCK Report" tab? If so,
> > > what does it show?
> > >
> >
> > When I got to the problem procedures & locks showed one item for
> > each problem on that table. After removal of masterprocwals they all
> > disappeared. The only entry now is one normal-looking WAL log file
> > entry whose timestamp changes according to operation time (currently
> > 8.25am today).
> >
> > >
> > > > The solution offered by hbase-operator-tools -
> > > > extraRegionsinMeta
> > > > - offered hope. Once again, however, another problem has surfaced:
> > > > this tools command for extra regions is incompatible with the
> > > > hbase version we are running.
> > > >
> > > > So...
> > > >
> > > > How can I remove the references to namespace:kaput_table from
> > hbase:meta?
> > > >
> > > >
> > > > Sounds like an hbase-2.1.x or a 2.0.x.
> > >
> > > Will wait on your answers to the above... It might be a crass
> > > delete of each row from the hbase:meta table then restart (even
> > > then, if procedures in the procedure store, you may have to clear
> > > it again as you did above before the restart to purge the
> > > procedures as you don't have tooling to do it from cmdline... do
> > > you have an 'hbck2 bypass
> > --override'?).
> > >
> > A crass delete of each row would work for me if I could Identify how
> > to remove the rows. Sample data:-
> >
> > alfa:rfilenameext column=table:state, timestamp=1604493139455,
> > value=\x08\x00
> > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > column=info:regioninfo, timestamp=1604388258225, value={ENCODED =>
> > 35925292c25898671e5a894ce387e167, NAME =>
> > 'alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.'
> > ,
> > STARTKEY => '', ENDKEY => '0'}
> > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > column=info:seqnumDuringOpen, timestamp=1601269814633,
> > value=\x00\x00\x00\x00\x00\x00\x008
> > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > column=info:server, timestamp=1601269814633, value=
> > ba-wtmp04.asgardalfa.hq.com:16020
> > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > column=info:serverstartcode, timestamp=1601269814633,
> > value=1601061167123
> alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > column=info:sn, timestamp=1604388258050,
> > value=ba-wtmp04.asgardalfa.hq.com
> > ,16020,1601061167123
> > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > column=info:state, timestamp=1604388258225, value=CLOSED
> > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > column=info:regioninfo, timestamp=1600969938610, value={ENCODED =>
> > 787d1455b84f2d846ce6089392f01fd2, NAME =>
> > 'alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > ',
> > STARTKEY => '0', ENDKEY => '1'}
> > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > column=info:seqnumDuringOpen, timestamp=1600780187722,
> > value=\x00\x00\x00\x00\x00\x02^\xBB
> > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > column=info:server, timestamp=1600780187722, value=
> > ba-wtmp08.asgardalfa.hq.com:16020
> > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > column=info:serverstartcode, timestamp=1600780187722,
> > value=1600780162556
> alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > column=info:sn, timestamp=1600969938610,
> > value=ba-wtmp07.asgardalfa.hq.com
> > ,16020,1600936054386
> > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > column=info:state, timestamp=1600969938610, value=OPENING
> > alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > column=info:regioninfo, timestamp=1601060563980, value={ENCODED =>
> > aa9d89b40a9def31a080fdd1776acb4e, NAME =>
> > 'alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > ',
> > STARTKEY => '1', ENDKEY => '2'}
> > alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > column=info:seqnumDuringOpen, timestamp=1600780186976,
> > value=\x00\x00\x00\x00\x00\x02^\xA9
> > alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > column=info:server, timestamp=1600780186976, value=
> > dr1-wtmp02.asgardalfa.hq.com:16020
> > alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > column=info:serverstartcode, timestamp=1600780186976,
> > value=1600780163021
> alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > column=info:sn, timestamp=1601060563980,
> > value=ba-wtmp05.asgardalfa.hq.com
> > ,16020,1601049145467
> >
> >
> > As I state below, a simple method to remove everything connected to
> > alfa:rfilenameext would seem the easiest of tasks as the
> > namespace:name consistently appears on every relevant row.
> >
> > >
> > >
> > > > Technical:
> > > >
> > > > Is there to be any implementation of such a fix within HBASe
> > > > itself where table manipulation can be performed by Eg.,
> > > >
> > > > delete hbase:meta namespace
> > > > delete hbase:meta namespace:table
> > > >
> > > > or even
> > > >
> > > > scan hbase:meta filter = namespace:table | deleterow
> > > >
> > > > or some such?
> > > >
> > > >
> > > Scan doesn't return result (unfortunately) so you can assign to a
> > > shell variable; it dumps scan output2 on stdout/stderr.
> > >
> > > I'm sure there is a better way but something like below (don't laugh!
> > > and I've not really tried it so be careful):
> > >
> > > # Get rows for the BAD_TABLENAME from hbase:meta table $ echo
> > > 'scan "hbase:meta", {ROWPREFIXFILTER => "BAD_TABLENAME"}'| hbase
> > > shell > /tmp/out.txt # Need to get the row only. Above got rows and columns.
> > > We don't want to filter on column # because we do not know which
> > > columns are still in hbase:meta. Need to filter on table # again
> > > to cut the shell preamble and footer out. Unique the rows.
> > > $ cat /tmp/out.txt |awk '{print $1}'|grep BAD_TABLENAME | sort -u
> > > > /tmp/unique_rows.txt # Now you have rows to delete. For each in
> > > shell do... 'deleteall "ROW_X"'... Could # edit
> > > /tmp/unique_rows.txt and per for add .... deleteall " prefix and a "
> > > suffix.... to make
> > > # a file with lines of deleteall "ROW1", etc. then pass it to the
> shell:
> > > hbase shell /tmp/delete_rows.txt.
> > >
> > > But before messing w/ the above, lets get the answers to above
> questions.
> > > Thanks,
> > > S
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > > Thanks
> > > >
> > > > M
> > > >
> > >
> >
>
Re: Removal of table rows.
Posted by Wellington Chevreuil <we...@gmail.com>.
>
> Do I build hbck2-tools on similar OS & java version? I have been informed
> we have 'maven' installed on one host on a cluster, which is centos and
> (probably) a different java, and the build itself is needed for Ubuntu16.
I would stick to the same java major version. OS wise, this module doesn't
rely on any OS native call, AFAIK, so should work fine between these
different versions.
Em qua., 16 de dez. de 2020 às 13:30, Marc Hoppins <ma...@eset.sk>
escreveu:
> Thanks. Once again, a newbie in this regard but,
>
> Do I build hbck2-tools on similar OS & java version? I have been informed
> we have 'maven' installed on one host on a cluster, which is centos and
> (probably) a different java, and the build itself is needed for Ubuntu16.
>
> -----Original Message-----
> From: Wellington Chevreuil <we...@gmail.com>
> Sent: Wednesday, December 16, 2020 10:08 AM
> To: Hbase-User <us...@hbase.apache.org>
> Subject: Re: Removal of table rows.
>
> EXTERNAL
>
> >
> > Hbase:meta has info for one table region which is NOT on a master. Is
> > that correct? I would have expected all meta info to be stored on a
> master.
> >
> meta table is an "hbase system table" that has info about which regions
> are assigned to which region servers in your cluster. As any user table, it
> also has a region, and this region must be hosted on a region server in the
> cluster. Master is just responsible for the coordination of regions and
> some other housekeeping actions.
>
> As I state below, a simple method to remove everything connected to
> > alfa:rfilenameext would seem the easiest of tasks as the
> > namespace:name consistently appears on every relevant row.
>
> Looks like hbck2 *extraRegionsinMeta *is what you need here, and it
> should work with hbase 2.1. It's not available on hbck2 1.0 release, but
> you can build a new hbck2 jar out of current master branch, and that would
> give you an hbck2 with extraRegionsinMeta. You can do it by cloning the
> repo below, then do a *mvn install* from the main module:
>
> https://github.com/apache/hbase-operator-tools.git
>
> Em qua., 16 de dez. de 2020 às 08:02, Marc Hoppins <ma...@eset.sk>
> escreveu:
>
> > Hbase is 2.1.0 (via Cloudera CDH 6.3.2)
> >
> > The UI page for the master shows NO report links/tabs.
> >
> > Hbase:meta has info for one table region which is NOT on a master. Is
> > that correct? I would have expected all meta info to be stored on a
> master.
> >
> > Name Region Server
> > Read Requests Write Requests
> > Num.Storefiles MemSize Locality
> > hbase:meta,,1.1588230740 ba-wtmp05.asgardalfa.hq.com:16030
> > 219,168,603 35,419 1 MB 4 0 B
> > 1.0
> >
> > -----Original Message-----
> > From: Wellington Chevreuil <we...@gmail.com>
> > Sent: Tuesday, December 15, 2020 7:10 PM
> > To: Hbase-User <us...@hbase.apache.org>
> > Subject: Re: Removal of table rows.
> >
> > EXTERNAL
> >
> > >
> > > I am an HBASE newbie so I apologise if I am being repetitious.
> > >
> > > Apologies also if this is not the right group. Am not sure if this
> > > may be more suited to 'dev' list.
> > >
> > Welcome, this is the right channel for this kind of questions.
> >
> >
> > > The solution offered by hbase-operator-tools - extraRegionsinMeta -
> > > offered hope. Once again, however, another problem has surfaced:
> > > this tools command for extra regions is incompatible with the hbase
> > > version we are running.
> > >
> > This command does not rely on any Master/RegionServer interface, so
> > it should not have any incompatibility issues. It only uses public
> > client API to cleanup meta table, so maybe you just need to build
> > latest hbck2 master branch version? It would work for any hbase 2, in
> > theory, it could even work with hbase 1, but that was never tested.
> > Luckly, you got Stack's attention, so yeah, if you confirm on his
> > previous questions we might be able to help with further directions.
> >
> > Em ter., 15 de dez. de 2020 às 17:23, Stack <st...@duboce.net> escreveu:
> >
> > > On Tue, Dec 15, 2020 at 7:46 AM Marc Hoppins <ma...@eset.sk>
> > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I am an HBASE newbie so I apologise if I am being repetitious.
> > > >
> > > > Apologies also if this is not the right group. Am not sure if this
> > > > may be more suited to 'dev' list. However,
> > > >
> > > > A problem question and a technical/wishlist question.
> > > >
> > > > Problem:
> > > >
> > > > I have inherited a problem with an HBASE table. The original issue
> > > > may have erupted due to a network outage. A table has 48 region
> > > > in
> > > transition
> > > > operations, stuck that way for several weeks. A previous attempt
> > > > to fix things with hbck failed. An attempt to DISABLE then DROP
> > > > the table also failed. The four or five attempts to work the table
> > > > ALSO now had stuck procedures. Subsequent DFS and ZOO operations
> > > > left the situation where there was no data and no real table: just
> > > > a 6K file in an empty
> > > structure.
> > > >
> > > >
> > > Which version of hbase, do you know Marc? (Look at the base of the
> > > master UI. It'll tell you. Sounds like it is an hbase-2.x).
> > >
> > > What is the name of the 6k file? (I am trying to understand what the
> > > file you are referencing is).
> > >
> > > The table directory was removed from hdfs and zookeeper?
> > >
> > Yes. The data had already been removed from the previous tech attempts
> > to fix the issue.
> >
> > Thus, the HDFS 6K file structure contained no actual information, just
> > empty files and directories. Zkshell had been used to remove znodes.
> >
> >
> > >
> > > > When I got to the problem my knowledge of HBASE was nil. It is
> > > > little better than that now but anyway...
> > > >
> > > > Fortunately for me this is a testing/dev cluster. The 'owner' was
> > > > content that the table can be removed - and appeared to have
> > > > already been
> > > done...of
> > > > a kind.
> > > >
> > > > Reading and reading of others' similar issues lead me to the point
> > > > I also was going to clean the HDFS data and ZK data for this
> > > > table. I shut down HBASE, cleaned HDFS and ZK node data, deleted
> > > > the masterprocwals and restarted HBASE.
> > > >
> > > > When all came up I was happy to see that the affected table
> > > > appeared nowhere and that the procedures had all disappeared.
> > > >
> > > > However, when I hopped to hbase master, even though no table of
> > > > that name existed, 48 regions were still in transition. Further
> > > > research steered
> > > me
> > > > toward hbase:meta and sure enough, the references to the RITs
> > > > lived
> > > happily
> > > > among other data for other tables.
> > > >
> > > >
> > > Do you have a 'Procedures and Locks' tab on your master home page?
> > > If so, what does it report? Is there an "HBCK Report" tab? If so,
> > > what does it show?
> > >
> >
> > When I got to the problem procedures & locks showed one item for each
> > problem on that table. After removal of masterprocwals they all
> > disappeared. The only entry now is one normal-looking WAL log file
> > entry whose timestamp changes according to operation time (currently
> > 8.25am today).
> >
> > >
> > > > The solution offered by hbase-operator-tools - extraRegionsinMeta
> > > > - offered hope. Once again, however, another problem has surfaced:
> > > > this tools command for extra regions is incompatible with the
> > > > hbase version we are running.
> > > >
> > > > So...
> > > >
> > > > How can I remove the references to namespace:kaput_table from
> > hbase:meta?
> > > >
> > > >
> > > > Sounds like an hbase-2.1.x or a 2.0.x.
> > >
> > > Will wait on your answers to the above... It might be a crass delete
> > > of each row from the hbase:meta table then restart (even then, if
> > > procedures in the procedure store, you may have to clear it again as
> > > you did above before the restart to purge the procedures as you
> > > don't have tooling to do it from cmdline... do you have an 'hbck2
> > > bypass
> > --override'?).
> > >
> > A crass delete of each row would work for me if I could Identify how
> > to remove the rows. Sample data:-
> >
> > alfa:rfilenameext column=table:state, timestamp=1604493139455,
> > value=\x08\x00
> > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > column=info:regioninfo, timestamp=1604388258225, value={ENCODED =>
> > 35925292c25898671e5a894ce387e167, NAME =>
> > 'alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.',
> > STARTKEY => '', ENDKEY => '0'}
> > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > column=info:seqnumDuringOpen, timestamp=1601269814633,
> > value=\x00\x00\x00\x00\x00\x00\x008
> > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > column=info:server, timestamp=1601269814633, value=
> > ba-wtmp04.asgardalfa.hq.com:16020
> > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > column=info:serverstartcode, timestamp=1601269814633,
> > value=1601061167123
> alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > column=info:sn, timestamp=1604388258050,
> > value=ba-wtmp04.asgardalfa.hq.com
> > ,16020,1601061167123
> > alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> > column=info:state, timestamp=1604388258225, value=CLOSED
> > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > column=info:regioninfo, timestamp=1600969938610, value={ENCODED =>
> > 787d1455b84f2d846ce6089392f01fd2, NAME =>
> > 'alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.',
> > STARTKEY => '0', ENDKEY => '1'}
> > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > column=info:seqnumDuringOpen, timestamp=1600780187722,
> > value=\x00\x00\x00\x00\x00\x02^\xBB
> > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > column=info:server, timestamp=1600780187722, value=
> > ba-wtmp08.asgardalfa.hq.com:16020
> > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > column=info:serverstartcode, timestamp=1600780187722,
> > value=1600780162556
> alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > column=info:sn, timestamp=1600969938610,
> > value=ba-wtmp07.asgardalfa.hq.com
> > ,16020,1600936054386
> > alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> > column=info:state, timestamp=1600969938610, value=OPENING
> > alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > column=info:regioninfo, timestamp=1601060563980, value={ENCODED =>
> > aa9d89b40a9def31a080fdd1776acb4e, NAME =>
> > 'alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.',
> > STARTKEY => '1', ENDKEY => '2'}
> > alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > column=info:seqnumDuringOpen, timestamp=1600780186976,
> > value=\x00\x00\x00\x00\x00\x02^\xA9
> > alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > column=info:server, timestamp=1600780186976, value=
> > dr1-wtmp02.asgardalfa.hq.com:16020
> > alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > column=info:serverstartcode, timestamp=1600780186976,
> > value=1600780163021
> alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> > column=info:sn, timestamp=1601060563980,
> > value=ba-wtmp05.asgardalfa.hq.com
> > ,16020,1601049145467
> >
> >
> > As I state below, a simple method to remove everything connected to
> > alfa:rfilenameext would seem the easiest of tasks as the
> > namespace:name consistently appears on every relevant row.
> >
> > >
> > >
> > > > Technical:
> > > >
> > > > Is there to be any implementation of such a fix within HBASe
> > > > itself where table manipulation can be performed by Eg.,
> > > >
> > > > delete hbase:meta namespace
> > > > delete hbase:meta namespace:table
> > > >
> > > > or even
> > > >
> > > > scan hbase:meta filter = namespace:table | deleterow
> > > >
> > > > or some such?
> > > >
> > > >
> > > Scan doesn't return result (unfortunately) so you can assign to a
> > > shell variable; it dumps scan output2 on stdout/stderr.
> > >
> > > I'm sure there is a better way but something like below (don't laugh!
> > > and I've not really tried it so be careful):
> > >
> > > # Get rows for the BAD_TABLENAME from hbase:meta table $ echo 'scan
> > > "hbase:meta", {ROWPREFIXFILTER => "BAD_TABLENAME"}'| hbase shell >
> > > /tmp/out.txt # Need to get the row only. Above got rows and columns.
> > > We don't want to filter on column # because we do not know which
> > > columns are still in hbase:meta. Need to filter on table # again to
> > > cut the shell preamble and footer out. Unique the rows.
> > > $ cat /tmp/out.txt |awk '{print $1}'|grep BAD_TABLENAME | sort -u >
> > > /tmp/unique_rows.txt # Now you have rows to delete. For each in
> > > shell do... 'deleteall "ROW_X"'... Could # edit /tmp/unique_rows.txt
> > > and per for add .... deleteall " prefix and a "
> > > suffix.... to make
> > > # a file with lines of deleteall "ROW1", etc. then pass it to the
> shell:
> > > hbase shell /tmp/delete_rows.txt.
> > >
> > > But before messing w/ the above, lets get the answers to above
> questions.
> > > Thanks,
> > > S
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > > Thanks
> > > >
> > > > M
> > > >
> > >
> >
>
RE: Removal of table rows.
Posted by Marc Hoppins <ma...@eset.sk>.
Thanks. Once again, a newbie in this regard but,
Do I build hbck2-tools on similar OS & java version? I have been informed we have 'maven' installed on one host on a cluster, which is centos and (probably) a different java, and the build itself is needed for Ubuntu16.
-----Original Message-----
From: Wellington Chevreuil <we...@gmail.com>
Sent: Wednesday, December 16, 2020 10:08 AM
To: Hbase-User <us...@hbase.apache.org>
Subject: Re: Removal of table rows.
EXTERNAL
>
> Hbase:meta has info for one table region which is NOT on a master. Is
> that correct? I would have expected all meta info to be stored on a master.
>
meta table is an "hbase system table" that has info about which regions are assigned to which region servers in your cluster. As any user table, it also has a region, and this region must be hosted on a region server in the cluster. Master is just responsible for the coordination of regions and some other housekeeping actions.
As I state below, a simple method to remove everything connected to
> alfa:rfilenameext would seem the easiest of tasks as the
> namespace:name consistently appears on every relevant row.
Looks like hbck2 *extraRegionsinMeta *is what you need here, and it should work with hbase 2.1. It's not available on hbck2 1.0 release, but you can build a new hbck2 jar out of current master branch, and that would give you an hbck2 with extraRegionsinMeta. You can do it by cloning the repo below, then do a *mvn install* from the main module:
https://github.com/apache/hbase-operator-tools.git
Em qua., 16 de dez. de 2020 às 08:02, Marc Hoppins <ma...@eset.sk>
escreveu:
> Hbase is 2.1.0 (via Cloudera CDH 6.3.2)
>
> The UI page for the master shows NO report links/tabs.
>
> Hbase:meta has info for one table region which is NOT on a master. Is
> that correct? I would have expected all meta info to be stored on a master.
>
> Name Region Server
> Read Requests Write Requests
> Num.Storefiles MemSize Locality
> hbase:meta,,1.1588230740 ba-wtmp05.asgardalfa.hq.com:16030
> 219,168,603 35,419 1 MB 4 0 B
> 1.0
>
> -----Original Message-----
> From: Wellington Chevreuil <we...@gmail.com>
> Sent: Tuesday, December 15, 2020 7:10 PM
> To: Hbase-User <us...@hbase.apache.org>
> Subject: Re: Removal of table rows.
>
> EXTERNAL
>
> >
> > I am an HBASE newbie so I apologise if I am being repetitious.
> >
> > Apologies also if this is not the right group. Am not sure if this
> > may be more suited to 'dev' list.
> >
> Welcome, this is the right channel for this kind of questions.
>
>
> > The solution offered by hbase-operator-tools - extraRegionsinMeta -
> > offered hope. Once again, however, another problem has surfaced:
> > this tools command for extra regions is incompatible with the hbase
> > version we are running.
> >
> This command does not rely on any Master/RegionServer interface, so
> it should not have any incompatibility issues. It only uses public
> client API to cleanup meta table, so maybe you just need to build
> latest hbck2 master branch version? It would work for any hbase 2, in
> theory, it could even work with hbase 1, but that was never tested.
> Luckly, you got Stack's attention, so yeah, if you confirm on his
> previous questions we might be able to help with further directions.
>
> Em ter., 15 de dez. de 2020 às 17:23, Stack <st...@duboce.net> escreveu:
>
> > On Tue, Dec 15, 2020 at 7:46 AM Marc Hoppins <ma...@eset.sk>
> wrote:
> >
> > > Hi all,
> > >
> > > I am an HBASE newbie so I apologise if I am being repetitious.
> > >
> > > Apologies also if this is not the right group. Am not sure if this
> > > may be more suited to 'dev' list. However,
> > >
> > > A problem question and a technical/wishlist question.
> > >
> > > Problem:
> > >
> > > I have inherited a problem with an HBASE table. The original issue
> > > may have erupted due to a network outage. A table has 48 region
> > > in
> > transition
> > > operations, stuck that way for several weeks. A previous attempt
> > > to fix things with hbck failed. An attempt to DISABLE then DROP
> > > the table also failed. The four or five attempts to work the table
> > > ALSO now had stuck procedures. Subsequent DFS and ZOO operations
> > > left the situation where there was no data and no real table: just
> > > a 6K file in an empty
> > structure.
> > >
> > >
> > Which version of hbase, do you know Marc? (Look at the base of the
> > master UI. It'll tell you. Sounds like it is an hbase-2.x).
> >
> > What is the name of the 6k file? (I am trying to understand what the
> > file you are referencing is).
> >
> > The table directory was removed from hdfs and zookeeper?
> >
> Yes. The data had already been removed from the previous tech attempts
> to fix the issue.
>
> Thus, the HDFS 6K file structure contained no actual information, just
> empty files and directories. Zkshell had been used to remove znodes.
>
>
> >
> > > When I got to the problem my knowledge of HBASE was nil. It is
> > > little better than that now but anyway...
> > >
> > > Fortunately for me this is a testing/dev cluster. The 'owner' was
> > > content that the table can be removed - and appeared to have
> > > already been
> > done...of
> > > a kind.
> > >
> > > Reading and reading of others' similar issues lead me to the point
> > > I also was going to clean the HDFS data and ZK data for this
> > > table. I shut down HBASE, cleaned HDFS and ZK node data, deleted
> > > the masterprocwals and restarted HBASE.
> > >
> > > When all came up I was happy to see that the affected table
> > > appeared nowhere and that the procedures had all disappeared.
> > >
> > > However, when I hopped to hbase master, even though no table of
> > > that name existed, 48 regions were still in transition. Further
> > > research steered
> > me
> > > toward hbase:meta and sure enough, the references to the RITs
> > > lived
> > happily
> > > among other data for other tables.
> > >
> > >
> > Do you have a 'Procedures and Locks' tab on your master home page?
> > If so, what does it report? Is there an "HBCK Report" tab? If so,
> > what does it show?
> >
>
> When I got to the problem procedures & locks showed one item for each
> problem on that table. After removal of masterprocwals they all
> disappeared. The only entry now is one normal-looking WAL log file
> entry whose timestamp changes according to operation time (currently
> 8.25am today).
>
> >
> > > The solution offered by hbase-operator-tools - extraRegionsinMeta
> > > - offered hope. Once again, however, another problem has surfaced:
> > > this tools command for extra regions is incompatible with the
> > > hbase version we are running.
> > >
> > > So...
> > >
> > > How can I remove the references to namespace:kaput_table from
> hbase:meta?
> > >
> > >
> > > Sounds like an hbase-2.1.x or a 2.0.x.
> >
> > Will wait on your answers to the above... It might be a crass delete
> > of each row from the hbase:meta table then restart (even then, if
> > procedures in the procedure store, you may have to clear it again as
> > you did above before the restart to purge the procedures as you
> > don't have tooling to do it from cmdline... do you have an 'hbck2
> > bypass
> --override'?).
> >
> A crass delete of each row would work for me if I could Identify how
> to remove the rows. Sample data:-
>
> alfa:rfilenameext column=table:state, timestamp=1604493139455,
> value=\x08\x00
> alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> column=info:regioninfo, timestamp=1604388258225, value={ENCODED =>
> 35925292c25898671e5a894ce387e167, NAME =>
> 'alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.',
> STARTKEY => '', ENDKEY => '0'}
> alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> column=info:seqnumDuringOpen, timestamp=1601269814633,
> value=\x00\x00\x00\x00\x00\x00\x008
> alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> column=info:server, timestamp=1601269814633, value=
> ba-wtmp04.asgardalfa.hq.com:16020
> alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> column=info:serverstartcode, timestamp=1601269814633,
> value=1601061167123 alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> column=info:sn, timestamp=1604388258050,
> value=ba-wtmp04.asgardalfa.hq.com
> ,16020,1601061167123
> alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> column=info:state, timestamp=1604388258225, value=CLOSED
> alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> column=info:regioninfo, timestamp=1600969938610, value={ENCODED =>
> 787d1455b84f2d846ce6089392f01fd2, NAME =>
> 'alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.',
> STARTKEY => '0', ENDKEY => '1'}
> alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> column=info:seqnumDuringOpen, timestamp=1600780187722,
> value=\x00\x00\x00\x00\x00\x02^\xBB
> alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> column=info:server, timestamp=1600780187722, value=
> ba-wtmp08.asgardalfa.hq.com:16020
> alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> column=info:serverstartcode, timestamp=1600780187722,
> value=1600780162556 alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> column=info:sn, timestamp=1600969938610,
> value=ba-wtmp07.asgardalfa.hq.com
> ,16020,1600936054386
> alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> column=info:state, timestamp=1600969938610, value=OPENING
> alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> column=info:regioninfo, timestamp=1601060563980, value={ENCODED =>
> aa9d89b40a9def31a080fdd1776acb4e, NAME =>
> 'alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.',
> STARTKEY => '1', ENDKEY => '2'}
> alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> column=info:seqnumDuringOpen, timestamp=1600780186976,
> value=\x00\x00\x00\x00\x00\x02^\xA9
> alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> column=info:server, timestamp=1600780186976, value=
> dr1-wtmp02.asgardalfa.hq.com:16020
> alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> column=info:serverstartcode, timestamp=1600780186976,
> value=1600780163021 alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> column=info:sn, timestamp=1601060563980,
> value=ba-wtmp05.asgardalfa.hq.com
> ,16020,1601049145467
>
>
> As I state below, a simple method to remove everything connected to
> alfa:rfilenameext would seem the easiest of tasks as the
> namespace:name consistently appears on every relevant row.
>
> >
> >
> > > Technical:
> > >
> > > Is there to be any implementation of such a fix within HBASe
> > > itself where table manipulation can be performed by Eg.,
> > >
> > > delete hbase:meta namespace
> > > delete hbase:meta namespace:table
> > >
> > > or even
> > >
> > > scan hbase:meta filter = namespace:table | deleterow
> > >
> > > or some such?
> > >
> > >
> > Scan doesn't return result (unfortunately) so you can assign to a
> > shell variable; it dumps scan output2 on stdout/stderr.
> >
> > I'm sure there is a better way but something like below (don't laugh!
> > and I've not really tried it so be careful):
> >
> > # Get rows for the BAD_TABLENAME from hbase:meta table $ echo 'scan
> > "hbase:meta", {ROWPREFIXFILTER => "BAD_TABLENAME"}'| hbase shell >
> > /tmp/out.txt # Need to get the row only. Above got rows and columns.
> > We don't want to filter on column # because we do not know which
> > columns are still in hbase:meta. Need to filter on table # again to
> > cut the shell preamble and footer out. Unique the rows.
> > $ cat /tmp/out.txt |awk '{print $1}'|grep BAD_TABLENAME | sort -u >
> > /tmp/unique_rows.txt # Now you have rows to delete. For each in
> > shell do... 'deleteall "ROW_X"'... Could # edit /tmp/unique_rows.txt
> > and per for add .... deleteall " prefix and a "
> > suffix.... to make
> > # a file with lines of deleteall "ROW1", etc. then pass it to the shell:
> > hbase shell /tmp/delete_rows.txt.
> >
> > But before messing w/ the above, lets get the answers to above questions.
> > Thanks,
> > S
> >
> >
> >
> >
> >
> >
> >
> >
> > > Thanks
> > >
> > > M
> > >
> >
>
Re: Removal of table rows.
Posted by Wellington Chevreuil <we...@gmail.com>.
>
> Hbase:meta has info for one table region which is NOT on a master. Is that
> correct? I would have expected all meta info to be stored on a master.
>
meta table is an "hbase system table" that has info about which regions are
assigned to which region servers in your cluster. As any user table, it
also has a region, and this region must be hosted on a region server in the
cluster. Master is just responsible for the coordination of regions and
some other housekeeping actions.
As I state below, a simple method to remove everything connected to
> alfa:rfilenameext would seem the easiest of tasks as the namespace:name
> consistently appears on every relevant row.
Looks like hbck2 *extraRegionsinMeta *is what you need here, and it should
work with hbase 2.1. It's not available on hbck2 1.0 release, but you can
build a new hbck2 jar out of current master branch, and that would give you
an hbck2 with extraRegionsinMeta. You can do it by cloning the repo below,
then do a *mvn install* from the main module:
https://github.com/apache/hbase-operator-tools.git
Em qua., 16 de dez. de 2020 às 08:02, Marc Hoppins <ma...@eset.sk>
escreveu:
> Hbase is 2.1.0 (via Cloudera CDH 6.3.2)
>
> The UI page for the master shows NO report links/tabs.
>
> Hbase:meta has info for one table region which is NOT on a master. Is that
> correct? I would have expected all meta info to be stored on a master.
>
> Name Region Server
> Read Requests Write Requests
> Num.Storefiles MemSize Locality
> hbase:meta,,1.1588230740 ba-wtmp05.asgardalfa.hq.com:16030
> 219,168,603 35,419 1 MB 4 0 B
> 1.0
>
> -----Original Message-----
> From: Wellington Chevreuil <we...@gmail.com>
> Sent: Tuesday, December 15, 2020 7:10 PM
> To: Hbase-User <us...@hbase.apache.org>
> Subject: Re: Removal of table rows.
>
> EXTERNAL
>
> >
> > I am an HBASE newbie so I apologise if I am being repetitious.
> >
> > Apologies also if this is not the right group. Am not sure if this may
> > be more suited to 'dev' list.
> >
> Welcome, this is the right channel for this kind of questions.
>
>
> > The solution offered by hbase-operator-tools - extraRegionsinMeta -
> > offered hope. Once again, however, another problem has surfaced: this
> > tools command for extra regions is incompatible with the hbase version
> > we are running.
> >
> This command does not rely on any Master/RegionServer interface, so it
> should not have any incompatibility issues. It only uses public client API
> to cleanup meta table, so maybe you just need to build latest hbck2 master
> branch version? It would work for any hbase 2, in theory, it could even
> work with hbase 1, but that was never tested. Luckly, you got Stack's
> attention, so yeah, if you confirm on his previous questions we might be
> able to help with further directions.
>
> Em ter., 15 de dez. de 2020 às 17:23, Stack <st...@duboce.net> escreveu:
>
> > On Tue, Dec 15, 2020 at 7:46 AM Marc Hoppins <ma...@eset.sk>
> wrote:
> >
> > > Hi all,
> > >
> > > I am an HBASE newbie so I apologise if I am being repetitious.
> > >
> > > Apologies also if this is not the right group. Am not sure if this
> > > may be more suited to 'dev' list. However,
> > >
> > > A problem question and a technical/wishlist question.
> > >
> > > Problem:
> > >
> > > I have inherited a problem with an HBASE table. The original issue
> > > may have erupted due to a network outage. A table has 48 region in
> > transition
> > > operations, stuck that way for several weeks. A previous attempt to
> > > fix things with hbck failed. An attempt to DISABLE then DROP the
> > > table also failed. The four or five attempts to work the table ALSO
> > > now had stuck procedures. Subsequent DFS and ZOO operations left
> > > the situation where there was no data and no real table: just a 6K
> > > file in an empty
> > structure.
> > >
> > >
> > Which version of hbase, do you know Marc? (Look at the base of the
> > master UI. It'll tell you. Sounds like it is an hbase-2.x).
> >
> > What is the name of the 6k file? (I am trying to understand what the
> > file you are referencing is).
> >
> > The table directory was removed from hdfs and zookeeper?
> >
> Yes. The data had already been removed from the previous tech attempts
> to fix the issue.
>
> Thus, the HDFS 6K file structure contained no actual information, just
> empty files and directories. Zkshell had been used to remove znodes.
>
>
> >
> > > When I got to the problem my knowledge of HBASE was nil. It is
> > > little better than that now but anyway...
> > >
> > > Fortunately for me this is a testing/dev cluster. The 'owner' was
> > > content that the table can be removed - and appeared to have already
> > > been
> > done...of
> > > a kind.
> > >
> > > Reading and reading of others' similar issues lead me to the point I
> > > also was going to clean the HDFS data and ZK data for this table. I
> > > shut down HBASE, cleaned HDFS and ZK node data, deleted the
> > > masterprocwals and restarted HBASE.
> > >
> > > When all came up I was happy to see that the affected table appeared
> > > nowhere and that the procedures had all disappeared.
> > >
> > > However, when I hopped to hbase master, even though no table of that
> > > name existed, 48 regions were still in transition. Further research
> > > steered
> > me
> > > toward hbase:meta and sure enough, the references to the RITs lived
> > happily
> > > among other data for other tables.
> > >
> > >
> > Do you have a 'Procedures and Locks' tab on your master home page? If
> > so, what does it report? Is there an "HBCK Report" tab? If so, what
> > does it show?
> >
>
> When I got to the problem procedures & locks showed one item for each
> problem on that table. After removal of masterprocwals they all
> disappeared. The only entry now is one normal-looking WAL log file entry
> whose timestamp changes according to operation time (currently 8.25am
> today).
>
> >
> > > The solution offered by hbase-operator-tools - extraRegionsinMeta -
> > > offered hope. Once again, however, another problem has surfaced:
> > > this tools command for extra regions is incompatible with the hbase
> > > version we are running.
> > >
> > > So...
> > >
> > > How can I remove the references to namespace:kaput_table from
> hbase:meta?
> > >
> > >
> > > Sounds like an hbase-2.1.x or a 2.0.x.
> >
> > Will wait on your answers to the above... It might be a crass delete
> > of each row from the hbase:meta table then restart (even then, if
> > procedures in the procedure store, you may have to clear it again as
> > you did above before the restart to purge the procedures as you don't
> > have tooling to do it from cmdline... do you have an 'hbck2 bypass
> --override'?).
> >
> A crass delete of each row would work for me if I could Identify how to
> remove the rows. Sample data:-
>
> alfa:rfilenameext column=table:state, timestamp=1604493139455,
> value=\x08\x00
> alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> column=info:regioninfo, timestamp=1604388258225, value={ENCODED =>
> 35925292c25898671e5a894ce387e167, NAME =>
> 'alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.',
> STARTKEY => '', ENDKEY => '0'}
> alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> column=info:seqnumDuringOpen, timestamp=1601269814633,
> value=\x00\x00\x00\x00\x00\x00\x008
> alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> column=info:server, timestamp=1601269814633, value=
> ba-wtmp04.asgardalfa.hq.com:16020
> alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> column=info:serverstartcode, timestamp=1601269814633, value=1601061167123
> alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> column=info:sn, timestamp=1604388258050, value=ba-wtmp04.asgardalfa.hq.com
> ,16020,1601061167123
> alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.
> column=info:state, timestamp=1604388258225, value=CLOSED
> alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> column=info:regioninfo, timestamp=1600969938610, value={ENCODED =>
> 787d1455b84f2d846ce6089392f01fd2, NAME =>
> 'alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.',
> STARTKEY => '0', ENDKEY => '1'}
> alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> column=info:seqnumDuringOpen, timestamp=1600780187722,
> value=\x00\x00\x00\x00\x00\x02^\xBB
> alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> column=info:server, timestamp=1600780187722, value=
> ba-wtmp08.asgardalfa.hq.com:16020
> alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> column=info:serverstartcode, timestamp=1600780187722, value=1600780162556
> alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> column=info:sn, timestamp=1600969938610, value=ba-wtmp07.asgardalfa.hq.com
> ,16020,1600936054386
> alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.
> column=info:state, timestamp=1600969938610, value=OPENING
> alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> column=info:regioninfo, timestamp=1601060563980, value={ENCODED =>
> aa9d89b40a9def31a080fdd1776acb4e, NAME =>
> 'alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.',
> STARTKEY => '1', ENDKEY => '2'}
> alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> column=info:seqnumDuringOpen, timestamp=1600780186976,
> value=\x00\x00\x00\x00\x00\x02^\xA9
> alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> column=info:server, timestamp=1600780186976, value=
> dr1-wtmp02.asgardalfa.hq.com:16020
> alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> column=info:serverstartcode, timestamp=1600780186976, value=1600780163021
> alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.
> column=info:sn, timestamp=1601060563980, value=ba-wtmp05.asgardalfa.hq.com
> ,16020,1601049145467
>
>
> As I state below, a simple method to remove everything connected to
> alfa:rfilenameext would seem the easiest of tasks as the namespace:name
> consistently appears on every relevant row.
>
> >
> >
> > > Technical:
> > >
> > > Is there to be any implementation of such a fix within HBASe itself
> > > where table manipulation can be performed by Eg.,
> > >
> > > delete hbase:meta namespace
> > > delete hbase:meta namespace:table
> > >
> > > or even
> > >
> > > scan hbase:meta filter = namespace:table | deleterow
> > >
> > > or some such?
> > >
> > >
> > Scan doesn't return result (unfortunately) so you can assign to a
> > shell variable; it dumps scan output2 on stdout/stderr.
> >
> > I'm sure there is a better way but something like below (don't laugh!
> > and I've not really tried it so be careful):
> >
> > # Get rows for the BAD_TABLENAME from hbase:meta table $ echo 'scan
> > "hbase:meta", {ROWPREFIXFILTER => "BAD_TABLENAME"}'| hbase shell >
> > /tmp/out.txt # Need to get the row only. Above got rows and columns.
> > We don't want to filter on column # because we do not know which
> > columns are still in hbase:meta. Need to filter on table # again to
> > cut the shell preamble and footer out. Unique the rows.
> > $ cat /tmp/out.txt |awk '{print $1}'|grep BAD_TABLENAME | sort -u >
> > /tmp/unique_rows.txt # Now you have rows to delete. For each in shell
> > do... 'deleteall "ROW_X"'... Could # edit /tmp/unique_rows.txt and per
> > for add .... deleteall " prefix and a "
> > suffix.... to make
> > # a file with lines of deleteall "ROW1", etc. then pass it to the shell:
> > hbase shell /tmp/delete_rows.txt.
> >
> > But before messing w/ the above, lets get the answers to above questions.
> > Thanks,
> > S
> >
> >
> >
> >
> >
> >
> >
> >
> > > Thanks
> > >
> > > M
> > >
> >
>
RE: Removal of table rows.
Posted by Marc Hoppins <ma...@eset.sk>.
Hbase is 2.1.0 (via Cloudera CDH 6.3.2)
The UI page for the master shows NO report links/tabs.
Hbase:meta has info for one table region which is NOT on a master. Is that correct? I would have expected all meta info to be stored on a master.
Name Region Server Read Requests Write Requests Num.Storefiles MemSize Locality
hbase:meta,,1.1588230740 ba-wtmp05.asgardalfa.hq.com:16030 219,168,603 35,419 1 MB 4 0 B 1.0
-----Original Message-----
From: Wellington Chevreuil <we...@gmail.com>
Sent: Tuesday, December 15, 2020 7:10 PM
To: Hbase-User <us...@hbase.apache.org>
Subject: Re: Removal of table rows.
EXTERNAL
>
> I am an HBASE newbie so I apologise if I am being repetitious.
>
> Apologies also if this is not the right group. Am not sure if this may
> be more suited to 'dev' list.
>
Welcome, this is the right channel for this kind of questions.
> The solution offered by hbase-operator-tools - extraRegionsinMeta -
> offered hope. Once again, however, another problem has surfaced: this
> tools command for extra regions is incompatible with the hbase version
> we are running.
>
This command does not rely on any Master/RegionServer interface, so it should not have any incompatibility issues. It only uses public client API to cleanup meta table, so maybe you just need to build latest hbck2 master branch version? It would work for any hbase 2, in theory, it could even work with hbase 1, but that was never tested. Luckly, you got Stack's attention, so yeah, if you confirm on his previous questions we might be able to help with further directions.
Em ter., 15 de dez. de 2020 às 17:23, Stack <st...@duboce.net> escreveu:
> On Tue, Dec 15, 2020 at 7:46 AM Marc Hoppins <ma...@eset.sk> wrote:
>
> > Hi all,
> >
> > I am an HBASE newbie so I apologise if I am being repetitious.
> >
> > Apologies also if this is not the right group. Am not sure if this
> > may be more suited to 'dev' list. However,
> >
> > A problem question and a technical/wishlist question.
> >
> > Problem:
> >
> > I have inherited a problem with an HBASE table. The original issue
> > may have erupted due to a network outage. A table has 48 region in
> transition
> > operations, stuck that way for several weeks. A previous attempt to
> > fix things with hbck failed. An attempt to DISABLE then DROP the
> > table also failed. The four or five attempts to work the table ALSO
> > now had stuck procedures. Subsequent DFS and ZOO operations left
> > the situation where there was no data and no real table: just a 6K
> > file in an empty
> structure.
> >
> >
> Which version of hbase, do you know Marc? (Look at the base of the
> master UI. It'll tell you. Sounds like it is an hbase-2.x).
>
> What is the name of the 6k file? (I am trying to understand what the
> file you are referencing is).
>
> The table directory was removed from hdfs and zookeeper?
>
Yes. The data had already been removed from the previous tech attempts to fix the issue.
Thus, the HDFS 6K file structure contained no actual information, just empty files and directories. Zkshell had been used to remove znodes.
>
> > When I got to the problem my knowledge of HBASE was nil. It is
> > little better than that now but anyway...
> >
> > Fortunately for me this is a testing/dev cluster. The 'owner' was
> > content that the table can be removed - and appeared to have already
> > been
> done...of
> > a kind.
> >
> > Reading and reading of others' similar issues lead me to the point I
> > also was going to clean the HDFS data and ZK data for this table. I
> > shut down HBASE, cleaned HDFS and ZK node data, deleted the
> > masterprocwals and restarted HBASE.
> >
> > When all came up I was happy to see that the affected table appeared
> > nowhere and that the procedures had all disappeared.
> >
> > However, when I hopped to hbase master, even though no table of that
> > name existed, 48 regions were still in transition. Further research
> > steered
> me
> > toward hbase:meta and sure enough, the references to the RITs lived
> happily
> > among other data for other tables.
> >
> >
> Do you have a 'Procedures and Locks' tab on your master home page? If
> so, what does it report? Is there an "HBCK Report" tab? If so, what
> does it show?
>
When I got to the problem procedures & locks showed one item for each problem on that table. After removal of masterprocwals they all disappeared. The only entry now is one normal-looking WAL log file entry whose timestamp changes according to operation time (currently 8.25am today).
>
> > The solution offered by hbase-operator-tools - extraRegionsinMeta -
> > offered hope. Once again, however, another problem has surfaced:
> > this tools command for extra regions is incompatible with the hbase
> > version we are running.
> >
> > So...
> >
> > How can I remove the references to namespace:kaput_table from hbase:meta?
> >
> >
> > Sounds like an hbase-2.1.x or a 2.0.x.
>
> Will wait on your answers to the above... It might be a crass delete
> of each row from the hbase:meta table then restart (even then, if
> procedures in the procedure store, you may have to clear it again as
> you did above before the restart to purge the procedures as you don't
> have tooling to do it from cmdline... do you have an 'hbck2 bypass --override'?).
>
A crass delete of each row would work for me if I could Identify how to remove the rows. Sample data:-
alfa:rfilenameext column=table:state, timestamp=1604493139455, value=\x08\x00
alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167. column=info:regioninfo, timestamp=1604388258225, value={ENCODED => 35925292c25898671e5a894ce387e167, NAME => 'alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167.', STARTKEY => '', ENDKEY => '0'}
alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167. column=info:seqnumDuringOpen, timestamp=1601269814633, value=\x00\x00\x00\x00\x00\x00\x008
alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167. column=info:server, timestamp=1601269814633, value=ba-wtmp04.asgardalfa.hq.com:16020
alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167. column=info:serverstartcode, timestamp=1601269814633, value=1601061167123
alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167. column=info:sn, timestamp=1604388258050, value=ba-wtmp04.asgardalfa.hq.com,16020,1601061167123
alfa:rfilenameext,,1557760164826.35925292c25898671e5a894ce387e167. column=info:state, timestamp=1604388258225, value=CLOSED
alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2. column=info:regioninfo, timestamp=1600969938610, value={ENCODED => 787d1455b84f2d846ce6089392f01fd2, NAME => 'alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2.', STARTKEY => '0', ENDKEY => '1'}
alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2. column=info:seqnumDuringOpen, timestamp=1600780187722, value=\x00\x00\x00\x00\x00\x02^\xBB
alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2. column=info:server, timestamp=1600780187722, value=ba-wtmp08.asgardalfa.hq.com:16020
alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2. column=info:serverstartcode, timestamp=1600780187722, value=1600780162556
alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2. column=info:sn, timestamp=1600969938610, value=ba-wtmp07.asgardalfa.hq.com,16020,1600936054386
alfa:rfilenameext,0,1557760164826.787d1455b84f2d846ce6089392f01fd2. column=info:state, timestamp=1600969938610, value=OPENING
alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e. column=info:regioninfo, timestamp=1601060563980, value={ENCODED => aa9d89b40a9def31a080fdd1776acb4e, NAME => 'alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e.', STARTKEY => '1', ENDKEY => '2'}
alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e. column=info:seqnumDuringOpen, timestamp=1600780186976, value=\x00\x00\x00\x00\x00\x02^\xA9
alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e. column=info:server, timestamp=1600780186976, value=dr1-wtmp02.asgardalfa.hq.com:16020
alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e. column=info:serverstartcode, timestamp=1600780186976, value=1600780163021
alfa:rfilenameext,1,1557760164826.aa9d89b40a9def31a080fdd1776acb4e. column=info:sn, timestamp=1601060563980, value=ba-wtmp05.asgardalfa.hq.com,16020,1601049145467
As I state below, a simple method to remove everything connected to alfa:rfilenameext would seem the easiest of tasks as the namespace:name consistently appears on every relevant row.
>
>
> > Technical:
> >
> > Is there to be any implementation of such a fix within HBASe itself
> > where table manipulation can be performed by Eg.,
> >
> > delete hbase:meta namespace
> > delete hbase:meta namespace:table
> >
> > or even
> >
> > scan hbase:meta filter = namespace:table | deleterow
> >
> > or some such?
> >
> >
> Scan doesn't return result (unfortunately) so you can assign to a
> shell variable; it dumps scan output2 on stdout/stderr.
>
> I'm sure there is a better way but something like below (don't laugh!
> and I've not really tried it so be careful):
>
> # Get rows for the BAD_TABLENAME from hbase:meta table $ echo 'scan
> "hbase:meta", {ROWPREFIXFILTER => "BAD_TABLENAME"}'| hbase shell >
> /tmp/out.txt # Need to get the row only. Above got rows and columns.
> We don't want to filter on column # because we do not know which
> columns are still in hbase:meta. Need to filter on table # again to
> cut the shell preamble and footer out. Unique the rows.
> $ cat /tmp/out.txt |awk '{print $1}'|grep BAD_TABLENAME | sort -u >
> /tmp/unique_rows.txt # Now you have rows to delete. For each in shell
> do... 'deleteall "ROW_X"'... Could # edit /tmp/unique_rows.txt and per
> for add .... deleteall " prefix and a "
> suffix.... to make
> # a file with lines of deleteall "ROW1", etc. then pass it to the shell:
> hbase shell /tmp/delete_rows.txt.
>
> But before messing w/ the above, lets get the answers to above questions.
> Thanks,
> S
>
>
>
>
>
>
>
>
> > Thanks
> >
> > M
> >
>
Re: Removal of table rows.
Posted by Wellington Chevreuil <we...@gmail.com>.
>
> I am an HBASE newbie so I apologise if I am being repetitious.
>
> Apologies also if this is not the right group. Am not sure if this may be
> more suited to 'dev' list.
>
Welcome, this is the right channel for this kind of questions.
> The solution offered by hbase-operator-tools - extraRegionsinMeta -
> offered hope. Once again, however, another problem has surfaced: this
> tools command for extra regions is incompatible with the hbase version we
> are running.
>
This command does not rely on any Master/RegionServer interface, so it
should not have any incompatibility issues. It only uses public client API
to cleanup meta table, so maybe you just need to build latest hbck2 master
branch version? It would work for any hbase 2, in theory, it could even
work with hbase 1, but that was never tested. Luckly, you got Stack's
attention, so yeah, if you confirm on his previous questions we might be
able to help with further directions.
Em ter., 15 de dez. de 2020 às 17:23, Stack <st...@duboce.net> escreveu:
> On Tue, Dec 15, 2020 at 7:46 AM Marc Hoppins <ma...@eset.sk> wrote:
>
> > Hi all,
> >
> > I am an HBASE newbie so I apologise if I am being repetitious.
> >
> > Apologies also if this is not the right group. Am not sure if this may be
> > more suited to 'dev' list. However,
> >
> > A problem question and a technical/wishlist question.
> >
> > Problem:
> >
> > I have inherited a problem with an HBASE table. The original issue may
> > have erupted due to a network outage. A table has 48 region in
> transition
> > operations, stuck that way for several weeks. A previous attempt to fix
> > things with hbck failed. An attempt to DISABLE then DROP the table also
> > failed. The four or five attempts to work the table ALSO now had stuck
> > procedures. Subsequent DFS and ZOO operations left the situation where
> > there was no data and no real table: just a 6K file in an empty
> structure.
> >
> >
> Which version of hbase, do you know Marc? (Look at the base of the master
> UI. It'll tell you. Sounds like it is an hbase-2.x).
>
> What is the name of the 6k file? (I am trying to understand what the file
> you are referencing is).
>
> The table directory was removed from hdfs and zookeeper?
>
>
>
> > When I got to the problem my knowledge of HBASE was nil. It is little
> > better than that now but anyway...
> >
> > Fortunately for me this is a testing/dev cluster. The 'owner' was content
> > that the table can be removed - and appeared to have already been
> done...of
> > a kind.
> >
> > Reading and reading of others' similar issues lead me to the point I also
> > was going to clean the HDFS data and ZK data for this table. I shut down
> > HBASE, cleaned HDFS and ZK node data, deleted the masterprocwals and
> > restarted HBASE.
> >
> > When all came up I was happy to see that the affected table appeared
> > nowhere and that the procedures had all disappeared.
> >
> > However, when I hopped to hbase master, even though no table of that name
> > existed, 48 regions were still in transition. Further research steered
> me
> > toward hbase:meta and sure enough, the references to the RITs lived
> happily
> > among other data for other tables.
> >
> >
> Do you have a 'Procedures and Locks' tab on your master home page? If so,
> what does it report? Is there an "HBCK Report" tab? If so, what does it
> show?
>
>
>
> > The solution offered by hbase-operator-tools - extraRegionsinMeta -
> > offered hope. Once again, however, another problem has surfaced: this
> > tools command for extra regions is incompatible with the hbase version we
> > are running.
> >
> > So...
> >
> > How can I remove the references to namespace:kaput_table from hbase:meta?
> >
> >
> > Sounds like an hbase-2.1.x or a 2.0.x.
>
> Will wait on your answers to the above... It might be a crass delete of
> each row from the hbase:meta table then restart (even then, if procedures
> in the procedure store, you may have to clear it again as you did above
> before the restart to purge the procedures as you don't have tooling to do
> it from cmdline... do you have an 'hbck2 bypass --override'?).
>
>
>
>
> > Technical:
> >
> > Is there to be any implementation of such a fix within HBASe itself where
> > table manipulation can be performed by Eg.,
> >
> > delete hbase:meta namespace
> > delete hbase:meta namespace:table
> >
> > or even
> >
> > scan hbase:meta filter = namespace:table | deleterow
> >
> > or some such?
> >
> >
> Scan doesn't return result (unfortunately) so you can assign to a shell
> variable; it dumps scan output2 on stdout/stderr.
>
> I'm sure there is a better way but something like below (don't laugh! and
> I've not really tried it so be careful):
>
> # Get rows for the BAD_TABLENAME from hbase:meta table
> $ echo 'scan "hbase:meta", {ROWPREFIXFILTER => "BAD_TABLENAME"}'| hbase
> shell > /tmp/out.txt
> # Need to get the row only. Above got rows and columns. We don't want to
> filter on column
> # because we do not know which columns are still in hbase:meta. Need to
> filter on table
> # again to cut the shell preamble and footer out. Unique the rows.
> $ cat /tmp/out.txt |awk '{print $1}'|grep BAD_TABLENAME | sort -u >
> /tmp/unique_rows.txt
> # Now you have rows to delete. For each in shell do... 'deleteall
> "ROW_X"'... Could
> # edit /tmp/unique_rows.txt and per for add .... deleteall " prefix and a "
> suffix.... to make
> # a file with lines of deleteall "ROW1", etc. then pass it to the shell:
> hbase shell /tmp/delete_rows.txt.
>
> But before messing w/ the above, lets get the answers to above questions.
> Thanks,
> S
>
>
>
>
>
>
>
>
> > Thanks
> >
> > M
> >
>
Re: Removal of table rows.
Posted by Stack <st...@duboce.net>.
On Tue, Dec 15, 2020 at 7:46 AM Marc Hoppins <ma...@eset.sk> wrote:
> Hi all,
>
> I am an HBASE newbie so I apologise if I am being repetitious.
>
> Apologies also if this is not the right group. Am not sure if this may be
> more suited to 'dev' list. However,
>
> A problem question and a technical/wishlist question.
>
> Problem:
>
> I have inherited a problem with an HBASE table. The original issue may
> have erupted due to a network outage. A table has 48 region in transition
> operations, stuck that way for several weeks. A previous attempt to fix
> things with hbck failed. An attempt to DISABLE then DROP the table also
> failed. The four or five attempts to work the table ALSO now had stuck
> procedures. Subsequent DFS and ZOO operations left the situation where
> there was no data and no real table: just a 6K file in an empty structure.
>
>
Which version of hbase, do you know Marc? (Look at the base of the master
UI. It'll tell you. Sounds like it is an hbase-2.x).
What is the name of the 6k file? (I am trying to understand what the file
you are referencing is).
The table directory was removed from hdfs and zookeeper?
> When I got to the problem my knowledge of HBASE was nil. It is little
> better than that now but anyway...
>
> Fortunately for me this is a testing/dev cluster. The 'owner' was content
> that the table can be removed - and appeared to have already been done...of
> a kind.
>
> Reading and reading of others' similar issues lead me to the point I also
> was going to clean the HDFS data and ZK data for this table. I shut down
> HBASE, cleaned HDFS and ZK node data, deleted the masterprocwals and
> restarted HBASE.
>
> When all came up I was happy to see that the affected table appeared
> nowhere and that the procedures had all disappeared.
>
> However, when I hopped to hbase master, even though no table of that name
> existed, 48 regions were still in transition. Further research steered me
> toward hbase:meta and sure enough, the references to the RITs lived happily
> among other data for other tables.
>
>
Do you have a 'Procedures and Locks' tab on your master home page? If so,
what does it report? Is there an "HBCK Report" tab? If so, what does it
show?
> The solution offered by hbase-operator-tools - extraRegionsinMeta -
> offered hope. Once again, however, another problem has surfaced: this
> tools command for extra regions is incompatible with the hbase version we
> are running.
>
> So...
>
> How can I remove the references to namespace:kaput_table from hbase:meta?
>
>
> Sounds like an hbase-2.1.x or a 2.0.x.
Will wait on your answers to the above... It might be a crass delete of
each row from the hbase:meta table then restart (even then, if procedures
in the procedure store, you may have to clear it again as you did above
before the restart to purge the procedures as you don't have tooling to do
it from cmdline... do you have an 'hbck2 bypass --override'?).
> Technical:
>
> Is there to be any implementation of such a fix within HBASe itself where
> table manipulation can be performed by Eg.,
>
> delete hbase:meta namespace
> delete hbase:meta namespace:table
>
> or even
>
> scan hbase:meta filter = namespace:table | deleterow
>
> or some such?
>
>
Scan doesn't return result (unfortunately) so you can assign to a shell
variable; it dumps scan output2 on stdout/stderr.
I'm sure there is a better way but something like below (don't laugh! and
I've not really tried it so be careful):
# Get rows for the BAD_TABLENAME from hbase:meta table
$ echo 'scan "hbase:meta", {ROWPREFIXFILTER => "BAD_TABLENAME"}'| hbase
shell > /tmp/out.txt
# Need to get the row only. Above got rows and columns. We don't want to
filter on column
# because we do not know which columns are still in hbase:meta. Need to
filter on table
# again to cut the shell preamble and footer out. Unique the rows.
$ cat /tmp/out.txt |awk '{print $1}'|grep BAD_TABLENAME | sort -u >
/tmp/unique_rows.txt
# Now you have rows to delete. For each in shell do... 'deleteall
"ROW_X"'... Could
# edit /tmp/unique_rows.txt and per for add .... deleteall " prefix and a "
suffix.... to make
# a file with lines of deleteall "ROW1", etc. then pass it to the shell:
hbase shell /tmp/delete_rows.txt.
But before messing w/ the above, lets get the answers to above questions.
Thanks,
S
> Thanks
>
> M
>