You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Ted Yu <yu...@gmail.com> on 2015/07/08 05:37:04 UTC
Re: [DISCUSSION] Merge of the hbase-11339 mob branch into master.
Looks like all MOB-related JIRAs have been resolved.
Should the voting process be resumed ?
Cheers
On Thu, May 28, 2015 at 5:48 PM, Anoop John <an...@gmail.com> wrote:
> Yes Andy. The sweep tool is completely optional now. We have a chore doing
> the compaction, like we trigger auto major compaction. We can configure
> the interval. Auto can be turned off and user can explicitly call also. We
> have shell and API support.
>
> Anoop
>
> On Friday, May 29, 2015, Andrew Purtell <ap...@apache.org> wrote:
> > MOB references in cells won't find their value if the MOB hfile has been
> > corrupted. Dealing with that would be like any other corrupted HFile,
> > understood. The dangling references make thinking about (partial)
> recovery
> > and repair interesting.
> >
> >
> > On Thursday, May 28, 2015, Jingcheng Du <ji...@intel.com> wrote:
> >
> >> Andrew Purtell wrote
> >> > HBCK can check and sideline dangling reference files. I think of MOB
> >> files
> >> > as "core enough" auxiliary files that need some support. I suppose
> unlike
> >> > reference files their presence or absence won't produce a region open
> >> > failure, we would see dangling pointers later when tying to service
> >> > queries. (Yes?) Will that abort the RS? Pardon the ignorant question,
> >> > normally I could check the code but I'm at the airport on a phone.
> >>
> >> Thanks for comments!
> >> In mob, usually the reference cells are committed after the mob files
> are
> >> done. I think it hardly happens that a reference cell cannot find its
> mob
> >> file.
> >> Even if there's a dangling reference cell, the RS won't be aborted, a
> empty
> >> cell is returned instead.
> >>
> >>
> >> Andrew Purtell wrote
> >> > On that subject, I should file follow up issues for more check and
> repair
> >> > options for HFiles. We should be able to detect missing or corrupt
> files
> >> > of
> >> > all variety: HFile, reference, MOB. This may require an expensive scan
> >> > over
> >> > lots of files, but this is like fsck full disk surface scans and those
> >> > have
> >> > similar costs. Providing MR based tools is fine but we should have
> >> > multithreaded tools that can stand in if a MR runtime is not
> available.
> >> > Import, Export, VerifyReplication...all of these tools are in a
> >> different,
> >> > lesser, class than integrity and repair tools, in my opinion. Since
> MOB
> >> > will likely be merged into trunk by then I'll be sure to include it. I
> >> > agree it's not fair to ask more of MOB then what we have now for
> HFile.
> >>
> >> To detect the corrupt files, some code are needed in the file checker to
> >> check mob files after knowing it's a mob-enabled column.
> >> To detect the missing or dangling reference cells, I think we have to do
> a
> >> full-table scan, like what is done now in HFile.main.
> >> We can do that.
> >>
> >>
> >>
> >> --
> >> View this message in context:
> >>
>
> http://apache-hbase.679495.n3.nabble.com/DISCUSSION-Merge-of-the-hbase-11339-mob-branch-into-master-tp4071644p4071911.html
> >> Sent from the HBase Developer mailing list archive at Nabble.com.
> >>
> >
> >
> > --
> > Best regards,
> >
> > - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>
Re: [DISCUSSION] Merge of the hbase-11339 mob branch into master.
Posted by Jingcheng Du <ji...@intel.com>.
Thanks Ted. The VOTE mail was posted a while ago.
Regards
Jingcheng
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/DISCUSSION-Merge-of-the-hbase-11339-mob-branch-into-master-tp4071644p4073309.html
Sent from the HBase Developer mailing list archive at Nabble.com.
Re: [DISCUSSION] Merge of the hbase-11339 mob branch into master.
Posted by Ted Yu <yu...@gmail.com>.
Jingcheng:
Let us know if you need anything for composition of the vote thread.
On Tue, Jul 14, 2015 at 3:58 PM, Andrew Purtell <ap...@apache.org> wrote:
> That's great, and I also appreciate the time spent discussing on this
> thread. I plan to vote yes.
>
> On Tue, Jul 14, 2015 at 3:57 PM, Jonathan Hsieh <jo...@cloudera.com> wrote:
>
> > Since Jingcheng did the majority of the work, I'd like him to stat the
> > thread vote thread. It should happen in the next few days.
> >
> > Jon
> >
> > On Tue, Jul 14, 2015 at 3:53 PM, Andrew Purtell <ap...@apache.org>
> > wrote:
> >
> > > Start a vote thread? This says DISCUSSION
> > >
> > > On Tue, Jul 14, 2015 at 2:05 PM, Ted Yu <yu...@gmail.com> wrote:
> > >
> > > > There have been several iterations since the first mega patch was
> > posted.
> > > >
> > > > Currently QA run is green.
> > > >
> > > > IntegrationTestIngestWithMOB has been run which passes.
> > > >
> > > > I want to give +1 for merging to master branch.
> > > >
> > > > On Tue, Jul 7, 2015 at 9:37 PM, Anoop John <an...@gmail.com>
> > > wrote:
> > > >
> > > > > +1
> > > > >
> > > > > We will work on making it a Vote thread.
> > > > >
> > > > > -Anoop-
> > > > >
> > > > > On Wed, Jul 8, 2015 at 9:46 AM, ramkrishna vasudevan <
> > > > > ramkrishna.s.vasudevan@gmail.com> wrote:
> > > > >
> > > > > > +1 to start the voting again.
> > > > > >
> > > > > > On Wed, Jul 8, 2015 at 9:07 AM, Ted Yu <yu...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > Looks like all MOB-related JIRAs have been resolved.
> > > > > > >
> > > > > > > Should the voting process be resumed ?
> > > > > > >
> > > > > > > Cheers
> > > > > > >
> > > > > > > On Thu, May 28, 2015 at 5:48 PM, Anoop John <
> > anoop.hbase@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Yes Andy. The sweep tool is completely optional now. We have
> a
> > > > chore
> > > > > > > doing
> > > > > > > > the compaction, like we trigger auto major compaction. We
> can
> > > > > > configure
> > > > > > > > the interval. Auto can be turned off and user can explicitly
> > call
> > > > > also.
> > > > > > > We
> > > > > > > > have shell and API support.
> > > > > > > >
> > > > > > > > Anoop
> > > > > > > >
> > > > > > > > On Friday, May 29, 2015, Andrew Purtell <apurtell@apache.org
> >
> > > > wrote:
> > > > > > > > > MOB references in cells won't find their value if the MOB
> > hfile
> > > > has
> > > > > > > been
> > > > > > > > > corrupted. Dealing with that would be like any other
> > corrupted
> > > > > HFile,
> > > > > > > > > understood. The dangling references make thinking about
> > > (partial)
> > > > > > > > recovery
> > > > > > > > > and repair interesting.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thursday, May 28, 2015, Jingcheng Du <
> > > jingcheng.du@intel.com>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Andrew Purtell wrote
> > > > > > > > >> > HBCK can check and sideline dangling reference files. I
> > > think
> > > > of
> > > > > > MOB
> > > > > > > > >> files
> > > > > > > > >> > as "core enough" auxiliary files that need some
> support. I
> > > > > suppose
> > > > > > > > unlike
> > > > > > > > >> > reference files their presence or absence won't produce
> a
> > > > region
> > > > > > > open
> > > > > > > > >> > failure, we would see dangling pointers later when tying
> > to
> > > > > > service
> > > > > > > > >> > queries. (Yes?) Will that abort the RS? Pardon the
> > ignorant
> > > > > > > question,
> > > > > > > > >> > normally I could check the code but I'm at the airport
> on
> > a
> > > > > phone.
> > > > > > > > >>
> > > > > > > > >> Thanks for comments!
> > > > > > > > >> In mob, usually the reference cells are committed after
> the
> > > mob
> > > > > > files
> > > > > > > > are
> > > > > > > > >> done. I think it hardly happens that a reference cell
> cannot
> > > > find
> > > > > > its
> > > > > > > > mob
> > > > > > > > >> file.
> > > > > > > > >> Even if there's a dangling reference cell, the RS won't be
> > > > > aborted,
> > > > > > a
> > > > > > > > empty
> > > > > > > > >> cell is returned instead.
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> Andrew Purtell wrote
> > > > > > > > >> > On that subject, I should file follow up issues for more
> > > check
> > > > > and
> > > > > > > > repair
> > > > > > > > >> > options for HFiles. We should be able to detect missing
> or
> > > > > corrupt
> > > > > > > > files
> > > > > > > > >> > of
> > > > > > > > >> > all variety: HFile, reference, MOB. This may require an
> > > > > expensive
> > > > > > > scan
> > > > > > > > >> > over
> > > > > > > > >> > lots of files, but this is like fsck full disk surface
> > scans
> > > > and
> > > > > > > those
> > > > > > > > >> > have
> > > > > > > > >> > similar costs. Providing MR based tools is fine but we
> > > should
> > > > > have
> > > > > > > > >> > multithreaded tools that can stand in if a MR runtime is
> > not
> > > > > > > > available.
> > > > > > > > >> > Import, Export, VerifyReplication...all of these tools
> are
> > > in
> > > > a
> > > > > > > > >> different,
> > > > > > > > >> > lesser, class than integrity and repair tools, in my
> > > opinion.
> > > > > > Since
> > > > > > > > MOB
> > > > > > > > >> > will likely be merged into trunk by then I'll be sure to
> > > > include
> > > > > > > it. I
> > > > > > > > >> > agree it's not fair to ask more of MOB then what we have
> > now
> > > > for
> > > > > > > > HFile.
> > > > > > > > >>
> > > > > > > > >> To detect the corrupt files, some code are needed in the
> > file
> > > > > > checker
> > > > > > > to
> > > > > > > > >> check mob files after knowing it's a mob-enabled column.
> > > > > > > > >> To detect the missing or dangling reference cells, I think
> > we
> > > > have
> > > > > > to
> > > > > > > do
> > > > > > > > a
> > > > > > > > >> full-table scan, like what is done now in HFile.main.
> > > > > > > > >> We can do that.
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> --
> > > > > > > > >> View this message in context:
> > > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://apache-hbase.679495.n3.nabble.com/DISCUSSION-Merge-of-the-hbase-11339-mob-branch-into-master-tp4071644p4071911.html
> > > > > > > > >> Sent from the HBase Developer mailing list archive at
> > > > Nabble.com.
> > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Best regards,
> > > > > > > > >
> > > > > > > > > - Andy
> > > > > > > > >
> > > > > > > > > Problems worthy of attack prove their worth by hitting
> back.
> > -
> > > > Piet
> > > > > > > Hein
> > > > > > > > > (via Tom White)
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Best regards,
> > >
> > > - Andy
> > >
> > > Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> > > (via Tom White)
> > >
> >
> >
> >
> > --
> > // Jonathan Hsieh (shay)
> > // HBase Tech Lead, Software Engineer, Cloudera
> > // jon@cloudera.com // @jmhsieh
> >
>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
Re: [DISCUSSION] Merge of the hbase-11339 mob branch into master.
Posted by Andrew Purtell <ap...@apache.org>.
That's great, and I also appreciate the time spent discussing on this
thread. I plan to vote yes.
On Tue, Jul 14, 2015 at 3:57 PM, Jonathan Hsieh <jo...@cloudera.com> wrote:
> Since Jingcheng did the majority of the work, I'd like him to stat the
> thread vote thread. It should happen in the next few days.
>
> Jon
>
> On Tue, Jul 14, 2015 at 3:53 PM, Andrew Purtell <ap...@apache.org>
> wrote:
>
> > Start a vote thread? This says DISCUSSION
> >
> > On Tue, Jul 14, 2015 at 2:05 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > There have been several iterations since the first mega patch was
> posted.
> > >
> > > Currently QA run is green.
> > >
> > > IntegrationTestIngestWithMOB has been run which passes.
> > >
> > > I want to give +1 for merging to master branch.
> > >
> > > On Tue, Jul 7, 2015 at 9:37 PM, Anoop John <an...@gmail.com>
> > wrote:
> > >
> > > > +1
> > > >
> > > > We will work on making it a Vote thread.
> > > >
> > > > -Anoop-
> > > >
> > > > On Wed, Jul 8, 2015 at 9:46 AM, ramkrishna vasudevan <
> > > > ramkrishna.s.vasudevan@gmail.com> wrote:
> > > >
> > > > > +1 to start the voting again.
> > > > >
> > > > > On Wed, Jul 8, 2015 at 9:07 AM, Ted Yu <yu...@gmail.com>
> wrote:
> > > > >
> > > > > > Looks like all MOB-related JIRAs have been resolved.
> > > > > >
> > > > > > Should the voting process be resumed ?
> > > > > >
> > > > > > Cheers
> > > > > >
> > > > > > On Thu, May 28, 2015 at 5:48 PM, Anoop John <
> anoop.hbase@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Yes Andy. The sweep tool is completely optional now. We have a
> > > chore
> > > > > > doing
> > > > > > > the compaction, like we trigger auto major compaction. We can
> > > > > configure
> > > > > > > the interval. Auto can be turned off and user can explicitly
> call
> > > > also.
> > > > > > We
> > > > > > > have shell and API support.
> > > > > > >
> > > > > > > Anoop
> > > > > > >
> > > > > > > On Friday, May 29, 2015, Andrew Purtell <ap...@apache.org>
> > > wrote:
> > > > > > > > MOB references in cells won't find their value if the MOB
> hfile
> > > has
> > > > > > been
> > > > > > > > corrupted. Dealing with that would be like any other
> corrupted
> > > > HFile,
> > > > > > > > understood. The dangling references make thinking about
> > (partial)
> > > > > > > recovery
> > > > > > > > and repair interesting.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thursday, May 28, 2015, Jingcheng Du <
> > jingcheng.du@intel.com>
> > > > > > wrote:
> > > > > > > >
> > > > > > > >> Andrew Purtell wrote
> > > > > > > >> > HBCK can check and sideline dangling reference files. I
> > think
> > > of
> > > > > MOB
> > > > > > > >> files
> > > > > > > >> > as "core enough" auxiliary files that need some support. I
> > > > suppose
> > > > > > > unlike
> > > > > > > >> > reference files their presence or absence won't produce a
> > > region
> > > > > > open
> > > > > > > >> > failure, we would see dangling pointers later when tying
> to
> > > > > service
> > > > > > > >> > queries. (Yes?) Will that abort the RS? Pardon the
> ignorant
> > > > > > question,
> > > > > > > >> > normally I could check the code but I'm at the airport on
> a
> > > > phone.
> > > > > > > >>
> > > > > > > >> Thanks for comments!
> > > > > > > >> In mob, usually the reference cells are committed after the
> > mob
> > > > > files
> > > > > > > are
> > > > > > > >> done. I think it hardly happens that a reference cell cannot
> > > find
> > > > > its
> > > > > > > mob
> > > > > > > >> file.
> > > > > > > >> Even if there's a dangling reference cell, the RS won't be
> > > > aborted,
> > > > > a
> > > > > > > empty
> > > > > > > >> cell is returned instead.
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> Andrew Purtell wrote
> > > > > > > >> > On that subject, I should file follow up issues for more
> > check
> > > > and
> > > > > > > repair
> > > > > > > >> > options for HFiles. We should be able to detect missing or
> > > > corrupt
> > > > > > > files
> > > > > > > >> > of
> > > > > > > >> > all variety: HFile, reference, MOB. This may require an
> > > > expensive
> > > > > > scan
> > > > > > > >> > over
> > > > > > > >> > lots of files, but this is like fsck full disk surface
> scans
> > > and
> > > > > > those
> > > > > > > >> > have
> > > > > > > >> > similar costs. Providing MR based tools is fine but we
> > should
> > > > have
> > > > > > > >> > multithreaded tools that can stand in if a MR runtime is
> not
> > > > > > > available.
> > > > > > > >> > Import, Export, VerifyReplication...all of these tools are
> > in
> > > a
> > > > > > > >> different,
> > > > > > > >> > lesser, class than integrity and repair tools, in my
> > opinion.
> > > > > Since
> > > > > > > MOB
> > > > > > > >> > will likely be merged into trunk by then I'll be sure to
> > > include
> > > > > > it. I
> > > > > > > >> > agree it's not fair to ask more of MOB then what we have
> now
> > > for
> > > > > > > HFile.
> > > > > > > >>
> > > > > > > >> To detect the corrupt files, some code are needed in the
> file
> > > > > checker
> > > > > > to
> > > > > > > >> check mob files after knowing it's a mob-enabled column.
> > > > > > > >> To detect the missing or dangling reference cells, I think
> we
> > > have
> > > > > to
> > > > > > do
> > > > > > > a
> > > > > > > >> full-table scan, like what is done now in HFile.main.
> > > > > > > >> We can do that.
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> --
> > > > > > > >> View this message in context:
> > > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://apache-hbase.679495.n3.nabble.com/DISCUSSION-Merge-of-the-hbase-11339-mob-branch-into-master-tp4071644p4071911.html
> > > > > > > >> Sent from the HBase Developer mailing list archive at
> > > Nabble.com.
> > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Best regards,
> > > > > > > >
> > > > > > > > - Andy
> > > > > > > >
> > > > > > > > Problems worthy of attack prove their worth by hitting back.
> -
> > > Piet
> > > > > > Hein
> > > > > > > > (via Tom White)
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Best regards,
> >
> > - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>
>
>
> --
> // Jonathan Hsieh (shay)
> // HBase Tech Lead, Software Engineer, Cloudera
> // jon@cloudera.com // @jmhsieh
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: [DISCUSSION] Merge of the hbase-11339 mob branch into master.
Posted by Jonathan Hsieh <jo...@cloudera.com>.
Since Jingcheng did the majority of the work, I'd like him to stat the
thread vote thread. It should happen in the next few days.
Jon
On Tue, Jul 14, 2015 at 3:53 PM, Andrew Purtell <ap...@apache.org> wrote:
> Start a vote thread? This says DISCUSSION
>
> On Tue, Jul 14, 2015 at 2:05 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > There have been several iterations since the first mega patch was posted.
> >
> > Currently QA run is green.
> >
> > IntegrationTestIngestWithMOB has been run which passes.
> >
> > I want to give +1 for merging to master branch.
> >
> > On Tue, Jul 7, 2015 at 9:37 PM, Anoop John <an...@gmail.com>
> wrote:
> >
> > > +1
> > >
> > > We will work on making it a Vote thread.
> > >
> > > -Anoop-
> > >
> > > On Wed, Jul 8, 2015 at 9:46 AM, ramkrishna vasudevan <
> > > ramkrishna.s.vasudevan@gmail.com> wrote:
> > >
> > > > +1 to start the voting again.
> > > >
> > > > On Wed, Jul 8, 2015 at 9:07 AM, Ted Yu <yu...@gmail.com> wrote:
> > > >
> > > > > Looks like all MOB-related JIRAs have been resolved.
> > > > >
> > > > > Should the voting process be resumed ?
> > > > >
> > > > > Cheers
> > > > >
> > > > > On Thu, May 28, 2015 at 5:48 PM, Anoop John <anoop.hbase@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > > Yes Andy. The sweep tool is completely optional now. We have a
> > chore
> > > > > doing
> > > > > > the compaction, like we trigger auto major compaction. We can
> > > > configure
> > > > > > the interval. Auto can be turned off and user can explicitly call
> > > also.
> > > > > We
> > > > > > have shell and API support.
> > > > > >
> > > > > > Anoop
> > > > > >
> > > > > > On Friday, May 29, 2015, Andrew Purtell <ap...@apache.org>
> > wrote:
> > > > > > > MOB references in cells won't find their value if the MOB hfile
> > has
> > > > > been
> > > > > > > corrupted. Dealing with that would be like any other corrupted
> > > HFile,
> > > > > > > understood. The dangling references make thinking about
> (partial)
> > > > > > recovery
> > > > > > > and repair interesting.
> > > > > > >
> > > > > > >
> > > > > > > On Thursday, May 28, 2015, Jingcheng Du <
> jingcheng.du@intel.com>
> > > > > wrote:
> > > > > > >
> > > > > > >> Andrew Purtell wrote
> > > > > > >> > HBCK can check and sideline dangling reference files. I
> think
> > of
> > > > MOB
> > > > > > >> files
> > > > > > >> > as "core enough" auxiliary files that need some support. I
> > > suppose
> > > > > > unlike
> > > > > > >> > reference files their presence or absence won't produce a
> > region
> > > > > open
> > > > > > >> > failure, we would see dangling pointers later when tying to
> > > > service
> > > > > > >> > queries. (Yes?) Will that abort the RS? Pardon the ignorant
> > > > > question,
> > > > > > >> > normally I could check the code but I'm at the airport on a
> > > phone.
> > > > > > >>
> > > > > > >> Thanks for comments!
> > > > > > >> In mob, usually the reference cells are committed after the
> mob
> > > > files
> > > > > > are
> > > > > > >> done. I think it hardly happens that a reference cell cannot
> > find
> > > > its
> > > > > > mob
> > > > > > >> file.
> > > > > > >> Even if there's a dangling reference cell, the RS won't be
> > > aborted,
> > > > a
> > > > > > empty
> > > > > > >> cell is returned instead.
> > > > > > >>
> > > > > > >>
> > > > > > >> Andrew Purtell wrote
> > > > > > >> > On that subject, I should file follow up issues for more
> check
> > > and
> > > > > > repair
> > > > > > >> > options for HFiles. We should be able to detect missing or
> > > corrupt
> > > > > > files
> > > > > > >> > of
> > > > > > >> > all variety: HFile, reference, MOB. This may require an
> > > expensive
> > > > > scan
> > > > > > >> > over
> > > > > > >> > lots of files, but this is like fsck full disk surface scans
> > and
> > > > > those
> > > > > > >> > have
> > > > > > >> > similar costs. Providing MR based tools is fine but we
> should
> > > have
> > > > > > >> > multithreaded tools that can stand in if a MR runtime is not
> > > > > > available.
> > > > > > >> > Import, Export, VerifyReplication...all of these tools are
> in
> > a
> > > > > > >> different,
> > > > > > >> > lesser, class than integrity and repair tools, in my
> opinion.
> > > > Since
> > > > > > MOB
> > > > > > >> > will likely be merged into trunk by then I'll be sure to
> > include
> > > > > it. I
> > > > > > >> > agree it's not fair to ask more of MOB then what we have now
> > for
> > > > > > HFile.
> > > > > > >>
> > > > > > >> To detect the corrupt files, some code are needed in the file
> > > > checker
> > > > > to
> > > > > > >> check mob files after knowing it's a mob-enabled column.
> > > > > > >> To detect the missing or dangling reference cells, I think we
> > have
> > > > to
> > > > > do
> > > > > > a
> > > > > > >> full-table scan, like what is done now in HFile.main.
> > > > > > >> We can do that.
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> --
> > > > > > >> View this message in context:
> > > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://apache-hbase.679495.n3.nabble.com/DISCUSSION-Merge-of-the-hbase-11339-mob-branch-into-master-tp4071644p4071911.html
> > > > > > >> Sent from the HBase Developer mailing list archive at
> > Nabble.com.
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Best regards,
> > > > > > >
> > > > > > > - Andy
> > > > > > >
> > > > > > > Problems worthy of attack prove their worth by hitting back. -
> > Piet
> > > > > Hein
> > > > > > > (via Tom White)
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
--
// Jonathan Hsieh (shay)
// HBase Tech Lead, Software Engineer, Cloudera
// jon@cloudera.com // @jmhsieh
Re: [DISCUSSION] Merge of the hbase-11339 mob branch into master.
Posted by Andrew Purtell <ap...@apache.org>.
Start a vote thread? This says DISCUSSION
On Tue, Jul 14, 2015 at 2:05 PM, Ted Yu <yu...@gmail.com> wrote:
> There have been several iterations since the first mega patch was posted.
>
> Currently QA run is green.
>
> IntegrationTestIngestWithMOB has been run which passes.
>
> I want to give +1 for merging to master branch.
>
> On Tue, Jul 7, 2015 at 9:37 PM, Anoop John <an...@gmail.com> wrote:
>
> > +1
> >
> > We will work on making it a Vote thread.
> >
> > -Anoop-
> >
> > On Wed, Jul 8, 2015 at 9:46 AM, ramkrishna vasudevan <
> > ramkrishna.s.vasudevan@gmail.com> wrote:
> >
> > > +1 to start the voting again.
> > >
> > > On Wed, Jul 8, 2015 at 9:07 AM, Ted Yu <yu...@gmail.com> wrote:
> > >
> > > > Looks like all MOB-related JIRAs have been resolved.
> > > >
> > > > Should the voting process be resumed ?
> > > >
> > > > Cheers
> > > >
> > > > On Thu, May 28, 2015 at 5:48 PM, Anoop John <an...@gmail.com>
> > > wrote:
> > > >
> > > > > Yes Andy. The sweep tool is completely optional now. We have a
> chore
> > > > doing
> > > > > the compaction, like we trigger auto major compaction. We can
> > > configure
> > > > > the interval. Auto can be turned off and user can explicitly call
> > also.
> > > > We
> > > > > have shell and API support.
> > > > >
> > > > > Anoop
> > > > >
> > > > > On Friday, May 29, 2015, Andrew Purtell <ap...@apache.org>
> wrote:
> > > > > > MOB references in cells won't find their value if the MOB hfile
> has
> > > > been
> > > > > > corrupted. Dealing with that would be like any other corrupted
> > HFile,
> > > > > > understood. The dangling references make thinking about (partial)
> > > > > recovery
> > > > > > and repair interesting.
> > > > > >
> > > > > >
> > > > > > On Thursday, May 28, 2015, Jingcheng Du <ji...@intel.com>
> > > > wrote:
> > > > > >
> > > > > >> Andrew Purtell wrote
> > > > > >> > HBCK can check and sideline dangling reference files. I think
> of
> > > MOB
> > > > > >> files
> > > > > >> > as "core enough" auxiliary files that need some support. I
> > suppose
> > > > > unlike
> > > > > >> > reference files their presence or absence won't produce a
> region
> > > > open
> > > > > >> > failure, we would see dangling pointers later when tying to
> > > service
> > > > > >> > queries. (Yes?) Will that abort the RS? Pardon the ignorant
> > > > question,
> > > > > >> > normally I could check the code but I'm at the airport on a
> > phone.
> > > > > >>
> > > > > >> Thanks for comments!
> > > > > >> In mob, usually the reference cells are committed after the mob
> > > files
> > > > > are
> > > > > >> done. I think it hardly happens that a reference cell cannot
> find
> > > its
> > > > > mob
> > > > > >> file.
> > > > > >> Even if there's a dangling reference cell, the RS won't be
> > aborted,
> > > a
> > > > > empty
> > > > > >> cell is returned instead.
> > > > > >>
> > > > > >>
> > > > > >> Andrew Purtell wrote
> > > > > >> > On that subject, I should file follow up issues for more check
> > and
> > > > > repair
> > > > > >> > options for HFiles. We should be able to detect missing or
> > corrupt
> > > > > files
> > > > > >> > of
> > > > > >> > all variety: HFile, reference, MOB. This may require an
> > expensive
> > > > scan
> > > > > >> > over
> > > > > >> > lots of files, but this is like fsck full disk surface scans
> and
> > > > those
> > > > > >> > have
> > > > > >> > similar costs. Providing MR based tools is fine but we should
> > have
> > > > > >> > multithreaded tools that can stand in if a MR runtime is not
> > > > > available.
> > > > > >> > Import, Export, VerifyReplication...all of these tools are in
> a
> > > > > >> different,
> > > > > >> > lesser, class than integrity and repair tools, in my opinion.
> > > Since
> > > > > MOB
> > > > > >> > will likely be merged into trunk by then I'll be sure to
> include
> > > > it. I
> > > > > >> > agree it's not fair to ask more of MOB then what we have now
> for
> > > > > HFile.
> > > > > >>
> > > > > >> To detect the corrupt files, some code are needed in the file
> > > checker
> > > > to
> > > > > >> check mob files after knowing it's a mob-enabled column.
> > > > > >> To detect the missing or dangling reference cells, I think we
> have
> > > to
> > > > do
> > > > > a
> > > > > >> full-table scan, like what is done now in HFile.main.
> > > > > >> We can do that.
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >> View this message in context:
> > > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
> http://apache-hbase.679495.n3.nabble.com/DISCUSSION-Merge-of-the-hbase-11339-mob-branch-into-master-tp4071644p4071911.html
> > > > > >> Sent from the HBase Developer mailing list archive at
> Nabble.com.
> > > > > >>
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Best regards,
> > > > > >
> > > > > > - Andy
> > > > > >
> > > > > > Problems worthy of attack prove their worth by hitting back. -
> Piet
> > > > Hein
> > > > > > (via Tom White)
> > > > > >
> > > > >
> > > >
> > >
> >
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: [DISCUSSION] Merge of the hbase-11339 mob branch into master.
Posted by Ted Yu <yu...@gmail.com>.
There have been several iterations since the first mega patch was posted.
Currently QA run is green.
IntegrationTestIngestWithMOB has been run which passes.
I want to give +1 for merging to master branch.
On Tue, Jul 7, 2015 at 9:37 PM, Anoop John <an...@gmail.com> wrote:
> +1
>
> We will work on making it a Vote thread.
>
> -Anoop-
>
> On Wed, Jul 8, 2015 at 9:46 AM, ramkrishna vasudevan <
> ramkrishna.s.vasudevan@gmail.com> wrote:
>
> > +1 to start the voting again.
> >
> > On Wed, Jul 8, 2015 at 9:07 AM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > Looks like all MOB-related JIRAs have been resolved.
> > >
> > > Should the voting process be resumed ?
> > >
> > > Cheers
> > >
> > > On Thu, May 28, 2015 at 5:48 PM, Anoop John <an...@gmail.com>
> > wrote:
> > >
> > > > Yes Andy. The sweep tool is completely optional now. We have a chore
> > > doing
> > > > the compaction, like we trigger auto major compaction. We can
> > configure
> > > > the interval. Auto can be turned off and user can explicitly call
> also.
> > > We
> > > > have shell and API support.
> > > >
> > > > Anoop
> > > >
> > > > On Friday, May 29, 2015, Andrew Purtell <ap...@apache.org> wrote:
> > > > > MOB references in cells won't find their value if the MOB hfile has
> > > been
> > > > > corrupted. Dealing with that would be like any other corrupted
> HFile,
> > > > > understood. The dangling references make thinking about (partial)
> > > > recovery
> > > > > and repair interesting.
> > > > >
> > > > >
> > > > > On Thursday, May 28, 2015, Jingcheng Du <ji...@intel.com>
> > > wrote:
> > > > >
> > > > >> Andrew Purtell wrote
> > > > >> > HBCK can check and sideline dangling reference files. I think of
> > MOB
> > > > >> files
> > > > >> > as "core enough" auxiliary files that need some support. I
> suppose
> > > > unlike
> > > > >> > reference files their presence or absence won't produce a region
> > > open
> > > > >> > failure, we would see dangling pointers later when tying to
> > service
> > > > >> > queries. (Yes?) Will that abort the RS? Pardon the ignorant
> > > question,
> > > > >> > normally I could check the code but I'm at the airport on a
> phone.
> > > > >>
> > > > >> Thanks for comments!
> > > > >> In mob, usually the reference cells are committed after the mob
> > files
> > > > are
> > > > >> done. I think it hardly happens that a reference cell cannot find
> > its
> > > > mob
> > > > >> file.
> > > > >> Even if there's a dangling reference cell, the RS won't be
> aborted,
> > a
> > > > empty
> > > > >> cell is returned instead.
> > > > >>
> > > > >>
> > > > >> Andrew Purtell wrote
> > > > >> > On that subject, I should file follow up issues for more check
> and
> > > > repair
> > > > >> > options for HFiles. We should be able to detect missing or
> corrupt
> > > > files
> > > > >> > of
> > > > >> > all variety: HFile, reference, MOB. This may require an
> expensive
> > > scan
> > > > >> > over
> > > > >> > lots of files, but this is like fsck full disk surface scans and
> > > those
> > > > >> > have
> > > > >> > similar costs. Providing MR based tools is fine but we should
> have
> > > > >> > multithreaded tools that can stand in if a MR runtime is not
> > > > available.
> > > > >> > Import, Export, VerifyReplication...all of these tools are in a
> > > > >> different,
> > > > >> > lesser, class than integrity and repair tools, in my opinion.
> > Since
> > > > MOB
> > > > >> > will likely be merged into trunk by then I'll be sure to include
> > > it. I
> > > > >> > agree it's not fair to ask more of MOB then what we have now for
> > > > HFile.
> > > > >>
> > > > >> To detect the corrupt files, some code are needed in the file
> > checker
> > > to
> > > > >> check mob files after knowing it's a mob-enabled column.
> > > > >> To detect the missing or dangling reference cells, I think we have
> > to
> > > do
> > > > a
> > > > >> full-table scan, like what is done now in HFile.main.
> > > > >> We can do that.
> > > > >>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> View this message in context:
> > > > >>
> > > >
> > > >
> > >
> >
> http://apache-hbase.679495.n3.nabble.com/DISCUSSION-Merge-of-the-hbase-11339-mob-branch-into-master-tp4071644p4071911.html
> > > > >> Sent from the HBase Developer mailing list archive at Nabble.com.
> > > > >>
> > > > >
> > > > >
> > > > > --
> > > > > Best regards,
> > > > >
> > > > > - Andy
> > > > >
> > > > > Problems worthy of attack prove their worth by hitting back. - Piet
> > > Hein
> > > > > (via Tom White)
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSSION] Merge of the hbase-11339 mob branch into master.
Posted by Anoop John <an...@gmail.com>.
+1
We will work on making it a Vote thread.
-Anoop-
On Wed, Jul 8, 2015 at 9:46 AM, ramkrishna vasudevan <
ramkrishna.s.vasudevan@gmail.com> wrote:
> +1 to start the voting again.
>
> On Wed, Jul 8, 2015 at 9:07 AM, Ted Yu <yu...@gmail.com> wrote:
>
> > Looks like all MOB-related JIRAs have been resolved.
> >
> > Should the voting process be resumed ?
> >
> > Cheers
> >
> > On Thu, May 28, 2015 at 5:48 PM, Anoop John <an...@gmail.com>
> wrote:
> >
> > > Yes Andy. The sweep tool is completely optional now. We have a chore
> > doing
> > > the compaction, like we trigger auto major compaction. We can
> configure
> > > the interval. Auto can be turned off and user can explicitly call also.
> > We
> > > have shell and API support.
> > >
> > > Anoop
> > >
> > > On Friday, May 29, 2015, Andrew Purtell <ap...@apache.org> wrote:
> > > > MOB references in cells won't find their value if the MOB hfile has
> > been
> > > > corrupted. Dealing with that would be like any other corrupted HFile,
> > > > understood. The dangling references make thinking about (partial)
> > > recovery
> > > > and repair interesting.
> > > >
> > > >
> > > > On Thursday, May 28, 2015, Jingcheng Du <ji...@intel.com>
> > wrote:
> > > >
> > > >> Andrew Purtell wrote
> > > >> > HBCK can check and sideline dangling reference files. I think of
> MOB
> > > >> files
> > > >> > as "core enough" auxiliary files that need some support. I suppose
> > > unlike
> > > >> > reference files their presence or absence won't produce a region
> > open
> > > >> > failure, we would see dangling pointers later when tying to
> service
> > > >> > queries. (Yes?) Will that abort the RS? Pardon the ignorant
> > question,
> > > >> > normally I could check the code but I'm at the airport on a phone.
> > > >>
> > > >> Thanks for comments!
> > > >> In mob, usually the reference cells are committed after the mob
> files
> > > are
> > > >> done. I think it hardly happens that a reference cell cannot find
> its
> > > mob
> > > >> file.
> > > >> Even if there's a dangling reference cell, the RS won't be aborted,
> a
> > > empty
> > > >> cell is returned instead.
> > > >>
> > > >>
> > > >> Andrew Purtell wrote
> > > >> > On that subject, I should file follow up issues for more check and
> > > repair
> > > >> > options for HFiles. We should be able to detect missing or corrupt
> > > files
> > > >> > of
> > > >> > all variety: HFile, reference, MOB. This may require an expensive
> > scan
> > > >> > over
> > > >> > lots of files, but this is like fsck full disk surface scans and
> > those
> > > >> > have
> > > >> > similar costs. Providing MR based tools is fine but we should have
> > > >> > multithreaded tools that can stand in if a MR runtime is not
> > > available.
> > > >> > Import, Export, VerifyReplication...all of these tools are in a
> > > >> different,
> > > >> > lesser, class than integrity and repair tools, in my opinion.
> Since
> > > MOB
> > > >> > will likely be merged into trunk by then I'll be sure to include
> > it. I
> > > >> > agree it's not fair to ask more of MOB then what we have now for
> > > HFile.
> > > >>
> > > >> To detect the corrupt files, some code are needed in the file
> checker
> > to
> > > >> check mob files after knowing it's a mob-enabled column.
> > > >> To detect the missing or dangling reference cells, I think we have
> to
> > do
> > > a
> > > >> full-table scan, like what is done now in HFile.main.
> > > >> We can do that.
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> View this message in context:
> > > >>
> > >
> > >
> >
> http://apache-hbase.679495.n3.nabble.com/DISCUSSION-Merge-of-the-hbase-11339-mob-branch-into-master-tp4071644p4071911.html
> > > >> Sent from the HBase Developer mailing list archive at Nabble.com.
> > > >>
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > >
> > > > - Andy
> > > >
> > > > Problems worthy of attack prove their worth by hitting back. - Piet
> > Hein
> > > > (via Tom White)
> > > >
> > >
> >
>
Re: [DISCUSSION] Merge of the hbase-11339 mob branch into master.
Posted by ramkrishna vasudevan <ra...@gmail.com>.
+1 to start the voting again.
On Wed, Jul 8, 2015 at 9:07 AM, Ted Yu <yu...@gmail.com> wrote:
> Looks like all MOB-related JIRAs have been resolved.
>
> Should the voting process be resumed ?
>
> Cheers
>
> On Thu, May 28, 2015 at 5:48 PM, Anoop John <an...@gmail.com> wrote:
>
> > Yes Andy. The sweep tool is completely optional now. We have a chore
> doing
> > the compaction, like we trigger auto major compaction. We can configure
> > the interval. Auto can be turned off and user can explicitly call also.
> We
> > have shell and API support.
> >
> > Anoop
> >
> > On Friday, May 29, 2015, Andrew Purtell <ap...@apache.org> wrote:
> > > MOB references in cells won't find their value if the MOB hfile has
> been
> > > corrupted. Dealing with that would be like any other corrupted HFile,
> > > understood. The dangling references make thinking about (partial)
> > recovery
> > > and repair interesting.
> > >
> > >
> > > On Thursday, May 28, 2015, Jingcheng Du <ji...@intel.com>
> wrote:
> > >
> > >> Andrew Purtell wrote
> > >> > HBCK can check and sideline dangling reference files. I think of MOB
> > >> files
> > >> > as "core enough" auxiliary files that need some support. I suppose
> > unlike
> > >> > reference files their presence or absence won't produce a region
> open
> > >> > failure, we would see dangling pointers later when tying to service
> > >> > queries. (Yes?) Will that abort the RS? Pardon the ignorant
> question,
> > >> > normally I could check the code but I'm at the airport on a phone.
> > >>
> > >> Thanks for comments!
> > >> In mob, usually the reference cells are committed after the mob files
> > are
> > >> done. I think it hardly happens that a reference cell cannot find its
> > mob
> > >> file.
> > >> Even if there's a dangling reference cell, the RS won't be aborted, a
> > empty
> > >> cell is returned instead.
> > >>
> > >>
> > >> Andrew Purtell wrote
> > >> > On that subject, I should file follow up issues for more check and
> > repair
> > >> > options for HFiles. We should be able to detect missing or corrupt
> > files
> > >> > of
> > >> > all variety: HFile, reference, MOB. This may require an expensive
> scan
> > >> > over
> > >> > lots of files, but this is like fsck full disk surface scans and
> those
> > >> > have
> > >> > similar costs. Providing MR based tools is fine but we should have
> > >> > multithreaded tools that can stand in if a MR runtime is not
> > available.
> > >> > Import, Export, VerifyReplication...all of these tools are in a
> > >> different,
> > >> > lesser, class than integrity and repair tools, in my opinion. Since
> > MOB
> > >> > will likely be merged into trunk by then I'll be sure to include
> it. I
> > >> > agree it's not fair to ask more of MOB then what we have now for
> > HFile.
> > >>
> > >> To detect the corrupt files, some code are needed in the file checker
> to
> > >> check mob files after knowing it's a mob-enabled column.
> > >> To detect the missing or dangling reference cells, I think we have to
> do
> > a
> > >> full-table scan, like what is done now in HFile.main.
> > >> We can do that.
> > >>
> > >>
> > >>
> > >> --
> > >> View this message in context:
> > >>
> >
> >
> http://apache-hbase.679495.n3.nabble.com/DISCUSSION-Merge-of-the-hbase-11339-mob-branch-into-master-tp4071644p4071911.html
> > >> Sent from the HBase Developer mailing list archive at Nabble.com.
> > >>
> > >
> > >
> > > --
> > > Best regards,
> > >
> > > - Andy
> > >
> > > Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> > > (via Tom White)
> > >
> >
>