You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Tom Chan <tc...@prediction.io> on 2015/06/03 03:37:18 UTC
Re: No serialized HRegionInfo
Thanks for the hint. I found from the log that regions assigned to a
particular regionserver (out of 3 machines) cause warning. I decommissioned
the regionserver in question and restarted it, fixed the inconsistencies
with -fixEmptyMetaCells and now the warning is gone.
Thanks,
Tom
On Sat, May 30, 2015 at 7:08 AM, Ted Yu <yu...@gmail.com> wrote:
> The pastebin of master log snippet doesn't provide enough clue.
>
> Can you trace, say, 1042331bd51614fabf01034113804665, back in master log to
> see whether it went through splits ?
> Please pastebin related portion of master log.
>
> Cheers
>
> On Fri, May 29, 2015 at 12:29 PM, Tom Chan <tc...@prediction.io> wrote:
>
> > On Thu, May 28, 2015 at 7:53 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > What release of hbase are you using ?
> > >
> > I'm using 0.98.9. I should have mentioned it before anything else.
> >
> > >
> > > If you trace 5fe111d3302cefb2e96d541a489ff026 in master log, do you
> find
> > > some clue ?
> > >
> > From the timestamp of the logs, all these warnings are given out when I
> do
> > "pio status" which tests if hbase is up by writing to it a table
> > "predictionio_eventdata:events_0" then deleting it.
> >
> > >
> > > Pastebin related log snippet if possible.
> > >
> >
> > http://pastebin.com/WDV01WQ7. There is only one line of "WARN No
> > HRegionInfo" instead of thousands of lines since I issued the "hbase hbck
> > -fixEmptyMetaCells" just now (but it keeps accumulating and that causes
> > problem).
> >
> > Also one thing I notice is that before fixing the meta table
> > (-fixEmptyMetaCells), in hbase shell, counting the non-existing table
> > 'predictionio_eventdata:events_0' takes much longer (a minute or two)
> than
> > some other non-existing tables. It spits out this line (or more) before
> > prompting that the table does not exist:
> >
> > ERROR: HRegionInfo was null or empty in hbase:meta,
> >
> >
> row=keyvalues={predictionio_eventdata:events_0,,1432926325587.9bb21fe3a575503da1bae02a8d22a8c0./info:seqnumDuringOpen/1432926325684/Put/vlen=8/mvcc=0,
> >
> >
> predictionio_eventdata:events_0,,1432926325587.9bb21fe3a575503da1bae02a8d22a8c0./info:server/1432926325684/Put/vlen=46/mvcc=0,
> >
> >
> predictionio_eventdata:events_0,,1432926325587.9bb21fe3a575503da1bae02a8d22a8c0./info:serverstartcode/1432926325684/Put/vlen=8/mvcc=0}
> >
> > Tom
> >
> >
> > >
> > > Thanks
> > >
> > > On Thu, May 28, 2015 at 6:48 PM, Tom Chan <tc...@prediction.io> wrote:
> > >
> > > > Hi,
> > > >
> > > > I kept running into this whenever I remove tables
> > > >
> > > > WARN catalog.MetaReader: No serialized HRegionInfo in
> > > >
> > > >
> > >
> >
> keyvalues={predictionio_eventdata:events_0,,1430338711772.915bc4b384b2fa24cc6113319f830100./info:seqnumDuringOpen/1430338711901/Put/vlen=8/mvcc=0,
> > > >
> > > >
> > >
> >
> predictionio_eventdata:events_0,,1430338711772.915bc4b384b2fa24cc6113319f830100./info:server/1430338711901/Put/vlen=46/mvcc=0,
> > > >
> > > >
> > >
> >
> predictionio_eventdata:events_0,,1430338711772.915bc4b384b2fa24cc6113319f830100./info:serverstartcode/1430338711901/Put/vlen=8/mvcc=0}
> > > >
> > > > create tables
> > > >
> > > > WARN client.HBaseAdmin: No serialized HRegionInfo in
> > > >
> > > >
> > >
> >
> keyvalues={predictionio_eventdata:events_0,,1427845220889.5fe111d3302cefb2e96d541a489ff026./info:seqnumDuringOpen/1427845220983/Put/vlen=8/mvcc=0,
> > > >
> > > >
> > >
> >
> predictionio_eventdata:events_0,,1427845220889.5fe111d3302cefb2e96d541a489ff026./info:server/1427845220983/Put/vlen=46/mvcc=0,
> > > >
> > > >
> > >
> >
> predictionio_eventdata:events_0,,1427845220889.5fe111d3302cefb2e96d541a489ff026./info:serverstartcode/1427845220983/Put/vlen=8/mvcc=0}
> > > >
> > > >
> > > > When doing hbck, I found some 1800 inconsistencies of the form
> > > >
> > > > ERROR: Empty REGIONINFO_QUALIFIER found in hbase:meta
> > > >
> > > >
> > > >
> > >
> >
> keyvalues={predictionio_eventdata:events_0,,1431758974706.765ee245540bd03dcd1ef66071e301d0./info:seqnumDuringOpen/1431758974883/Put/vlen=8/mvcc=0,
> > > >
> > > >
> > >
> >
> predictionio_eventdata:events_0,,1431758974706.765ee245540bd03dcd1ef66071e301d0./info:server/1431758974883/Put/vlen=46/mvcc=0,
> > > >
> > > >
> > >
> >
> predictionio_eventdata:events_0,,1431758974706.765ee245540bd03dcd1ef66071e301d0./info:serverstartcode/1431758974883/Put/vlen=8/mvcc=0}
> > > >
> > > > while all the tables are okay.
> > > >
> > > > I ran "hbase hbck -fixEmptyMetaCells" according to the suggestion on
> > > >
> > > >
> > >
> >
> http://mail-archives.apache.org/mod_mbox/hbase-dev/201312.mbox/%3CCAPQV63VrpJ-_ffOo92m=RyaZa+Z3LrpPhiNMi5Szx5uar=u0rw@mail.gmail.com%3E
> > > > which
> > > > cleared the inconsistencies and those warning messages, but the "No
> > > > serialized HRegionInfo" and HBase inconsistencies continued to
> > accumulate
> > > > so some config is still not quite right. Any help is appreciated.
> > > >
> > > > Tom
> > > >
> > >
> > >
> >
>
Re: No serialized HRegionInfo
Posted by Ted Yu <yu...@gmail.com>.
It would be interesting to see what caused the regionserver to malfunction.
Simplify restarting it may be good for now but without finding root cause,
your cluster may experience this problem again.
Cheers
On Tue, Jun 2, 2015 at 6:37 PM, Tom Chan <tc...@prediction.io> wrote:
> Thanks for the hint. I found from the log that regions assigned to a
> particular regionserver (out of 3 machines) cause warning. I decommissioned
> the regionserver in question and restarted it, fixed the inconsistencies
> with -fixEmptyMetaCells and now the warning is gone.
>
> Thanks,
> Tom
>
> On Sat, May 30, 2015 at 7:08 AM, Ted Yu <yu...@gmail.com> wrote:
>
> > The pastebin of master log snippet doesn't provide enough clue.
> >
> > Can you trace, say, 1042331bd51614fabf01034113804665, back in master log
> to
> > see whether it went through splits ?
> > Please pastebin related portion of master log.
> >
> > Cheers
> >
> > On Fri, May 29, 2015 at 12:29 PM, Tom Chan <tc...@prediction.io> wrote:
> >
> > > On Thu, May 28, 2015 at 7:53 PM, Ted Yu <yu...@gmail.com> wrote:
> > >
> > > > What release of hbase are you using ?
> > > >
> > > I'm using 0.98.9. I should have mentioned it before anything else.
> > >
> > > >
> > > > If you trace 5fe111d3302cefb2e96d541a489ff026 in master log, do you
> > find
> > > > some clue ?
> > > >
> > > From the timestamp of the logs, all these warnings are given out when I
> > do
> > > "pio status" which tests if hbase is up by writing to it a table
> > > "predictionio_eventdata:events_0" then deleting it.
> > >
> > > >
> > > > Pastebin related log snippet if possible.
> > > >
> > >
> > > http://pastebin.com/WDV01WQ7. There is only one line of "WARN No
> > > HRegionInfo" instead of thousands of lines since I issued the "hbase
> hbck
> > > -fixEmptyMetaCells" just now (but it keeps accumulating and that causes
> > > problem).
> > >
> > > Also one thing I notice is that before fixing the meta table
> > > (-fixEmptyMetaCells), in hbase shell, counting the non-existing table
> > > 'predictionio_eventdata:events_0' takes much longer (a minute or two)
> > than
> > > some other non-existing tables. It spits out this line (or more) before
> > > prompting that the table does not exist:
> > >
> > > ERROR: HRegionInfo was null or empty in hbase:meta,
> > >
> > >
> >
> row=keyvalues={predictionio_eventdata:events_0,,1432926325587.9bb21fe3a575503da1bae02a8d22a8c0./info:seqnumDuringOpen/1432926325684/Put/vlen=8/mvcc=0,
> > >
> > >
> >
> predictionio_eventdata:events_0,,1432926325587.9bb21fe3a575503da1bae02a8d22a8c0./info:server/1432926325684/Put/vlen=46/mvcc=0,
> > >
> > >
> >
> predictionio_eventdata:events_0,,1432926325587.9bb21fe3a575503da1bae02a8d22a8c0./info:serverstartcode/1432926325684/Put/vlen=8/mvcc=0}
> > >
> > > Tom
> > >
> > >
> > > >
> > > > Thanks
> > > >
> > > > On Thu, May 28, 2015 at 6:48 PM, Tom Chan <tc...@prediction.io>
> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I kept running into this whenever I remove tables
> > > > >
> > > > > WARN catalog.MetaReader: No serialized HRegionInfo in
> > > > >
> > > > >
> > > >
> > >
> >
> keyvalues={predictionio_eventdata:events_0,,1430338711772.915bc4b384b2fa24cc6113319f830100./info:seqnumDuringOpen/1430338711901/Put/vlen=8/mvcc=0,
> > > > >
> > > > >
> > > >
> > >
> >
> predictionio_eventdata:events_0,,1430338711772.915bc4b384b2fa24cc6113319f830100./info:server/1430338711901/Put/vlen=46/mvcc=0,
> > > > >
> > > > >
> > > >
> > >
> >
> predictionio_eventdata:events_0,,1430338711772.915bc4b384b2fa24cc6113319f830100./info:serverstartcode/1430338711901/Put/vlen=8/mvcc=0}
> > > > >
> > > > > create tables
> > > > >
> > > > > WARN client.HBaseAdmin: No serialized HRegionInfo in
> > > > >
> > > > >
> > > >
> > >
> >
> keyvalues={predictionio_eventdata:events_0,,1427845220889.5fe111d3302cefb2e96d541a489ff026./info:seqnumDuringOpen/1427845220983/Put/vlen=8/mvcc=0,
> > > > >
> > > > >
> > > >
> > >
> >
> predictionio_eventdata:events_0,,1427845220889.5fe111d3302cefb2e96d541a489ff026./info:server/1427845220983/Put/vlen=46/mvcc=0,
> > > > >
> > > > >
> > > >
> > >
> >
> predictionio_eventdata:events_0,,1427845220889.5fe111d3302cefb2e96d541a489ff026./info:serverstartcode/1427845220983/Put/vlen=8/mvcc=0}
> > > > >
> > > > >
> > > > > When doing hbck, I found some 1800 inconsistencies of the form
> > > > >
> > > > > ERROR: Empty REGIONINFO_QUALIFIER found in hbase:meta
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> keyvalues={predictionio_eventdata:events_0,,1431758974706.765ee245540bd03dcd1ef66071e301d0./info:seqnumDuringOpen/1431758974883/Put/vlen=8/mvcc=0,
> > > > >
> > > > >
> > > >
> > >
> >
> predictionio_eventdata:events_0,,1431758974706.765ee245540bd03dcd1ef66071e301d0./info:server/1431758974883/Put/vlen=46/mvcc=0,
> > > > >
> > > > >
> > > >
> > >
> >
> predictionio_eventdata:events_0,,1431758974706.765ee245540bd03dcd1ef66071e301d0./info:serverstartcode/1431758974883/Put/vlen=8/mvcc=0}
> > > > >
> > > > > while all the tables are okay.
> > > > >
> > > > > I ran "hbase hbck -fixEmptyMetaCells" according to the suggestion
> on
> > > > >
> > > > >
> > > >
> > >
> >
> http://mail-archives.apache.org/mod_mbox/hbase-dev/201312.mbox/%3CCAPQV63VrpJ-_ffOo92m=RyaZa+Z3LrpPhiNMi5Szx5uar=u0rw@mail.gmail.com%3E
> > > > > which
> > > > > cleared the inconsistencies and those warning messages, but the "No
> > > > > serialized HRegionInfo" and HBase inconsistencies continued to
> > > accumulate
> > > > > so some config is still not quite right. Any help is appreciated.
> > > > >
> > > > > Tom
> > > > >
> > > >
> > > >
> > >
> >
>