You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Jonathan Hsieh <jo...@cloudera.com> on 2015/05/20 18:21:20 UTC
[DISCUSSION] Merge of the hbase-11339 mob branch into master.
Hi folks,
The Medium Object (MOB) Storage feature (HBASE-11339[1]) is modified I/O
and compaction path that allows individual moderately sized values
(10k-10MB) to be stored so that write amplification is reduced when
compared to the normal I/O path. At a high level, it provides alternate
flush and compaction mechanisms that segregates large cells into a separate
area where they are not subject to potentially frequent compaction and
splits that can be encountered in the normal I/O path. A more detailed
design doc can be found on the hbase-11339 jira.
Jingcheng Du has been working on the mob feature for a while and Anoop, Ram
and I have been shepherding him through the design revisions and
implementation of the feature in the hbase-11339 branch.[2]
The branch we are proposing to merge into master is compatible with HBase's
core functionality including snapshots, replication, shell support, behaves
well with table alters, bulk loads and does not require external MR
processes. It has been documented, and subject to many integration test
runs (ITBLL, ITAcidGuarantees, ITIngest) including fault injection.
Performance testing of the feature shows what can be a 2x-3x throughput
improvement for workloads that contain mobs. These results can be seen on
the hbase 2.0 panel discussion slides from hbasecon (once published).
Recently there have been some hfile encryption related shortcomings that we
could address in branch or in master.
Earlier iterations of the feature has been tested in production by users
that Jingcheng has been responsible for. A version has also been deployed
at users I have been responsible for. Some of the folks from Huawei
(ashutosh) have also been submitting the recent encryption bug reports
against the hbase-11339 branch so there is some evidence of usage by them.
The four of us (Jingcheng, Ram, Anoop and I) are satisfied with the
feature and feel it is a good time to call a merge vote. Ive posted a
megapatch version for folks who want to peruse the code. [3]
What do you all think?
Thanks,
Jingcheng, Jon, Ram, and Anoop.
[1] https://issues.apache.org/jira/browse/HBASE-11339
[2] https://github.com/apache/hbase/tree/hbase-11339
[3] https://reviews.apache.org/r/34475/
--
// Jonathan Hsieh (shay)
// HBase Tech Lead, Software Engineer, Cloudera
// jon@cloudera.com // @jmhsieh
Re: [DISCUSSION] Merge of the hbase-11339 mob branch into master.
Posted by Jonathan Hsieh <jo...@cloudera.com>.
Not that I know of.
Here's a quick high level comparison based on my read of the ozone doc are
api differences:
- ozone is described an s3 clone while hbase mob is similar to a rdbms's
blob feature.
- ozone names - 3-64 bytes, hbase mob names - hbase qual limit -(64k i
think)
- ozone object sizes - 100k's to 100M's; hbase mob - 10k's-10'sM (tested
at 50M)
- ozone - hash partitioning, hbase mob - range partitioning
It isn't clear to me how they handle updates and deletes, hdfs snapshots
and other hdfs backup mechanisms.
Jon.
On Wed, May 20, 2015 at 8:10 PM, 张铎 <pa...@gmail.com> wrote:
> Is there any comparison between HBASE-11339 and HDFS-7240?
> Is their 'Object' a super set of our 'Medium Object'?
>
> 2015-05-21 10:38 GMT+08:00 Ted Yu <yu...@gmail.com>:
>
> > This is a useful feature, Jon.
> >
> > I went over the mega-patch and left some comments on review board.
> >
> > I noticed that hbck was not included in the patch. Neither did I find a
> > sub-task of HBASE-11339 that covers hbck.
> >
> > Do you or Jingcheng plan to add MOB-aware capability for hbck ?
> >
> > Cheers
> >
> > On Wed, May 20, 2015 at 9:21 AM, Jonathan Hsieh <jo...@cloudera.com>
> wrote:
> >
> > > Hi folks,
> > >
> > > The Medium Object (MOB) Storage feature (HBASE-11339[1]) is modified
> I/O
> > > and compaction path that allows individual moderately sized values
> > > (10k-10MB) to be stored so that write amplification is reduced when
> > > compared to the normal I/O path. At a high level, it provides
> alternate
> > > flush and compaction mechanisms that segregates large cells into a
> > separate
> > > area where they are not subject to potentially frequent compaction and
> > > splits that can be encountered in the normal I/O path. A more detailed
> > > design doc can be found on the hbase-11339 jira.
> > >
> > > Jingcheng Du has been working on the mob feature for a while and Anoop,
> > Ram
> > > and I have been shepherding him through the design revisions and
> > > implementation of the feature in the hbase-11339 branch.[2]
> > >
> > > The branch we are proposing to merge into master is compatible with
> > HBase's
> > > core functionality including snapshots, replication, shell support,
> > behaves
> > > well with table alters, bulk loads and does not require external MR
> > > processes. It has been documented, and subject to many integration test
> > > runs (ITBLL, ITAcidGuarantees, ITIngest) including fault injection.
> > > Performance testing of the feature shows what can be a 2x-3x throughput
> > > improvement for workloads that contain mobs. These results can be seen
> on
> > > the hbase 2.0 panel discussion slides from hbasecon (once published).
> > >
> > > Recently there have been some hfile encryption related shortcomings
> that
> > we
> > > could address in branch or in master.
> > >
> > > Earlier iterations of the feature has been tested in production by
> users
> > > that Jingcheng has been responsible for. A version has also been
> > deployed
> > > at users I have been responsible for. Some of the folks from Huawei
> > > (ashutosh) have also been submitting the recent encryption bug reports
> > > against the hbase-11339 branch so there is some evidence of usage by
> > them.
> > >
> > > The four of us (Jingcheng, Ram, Anoop and I) are satisfied with the
> > > feature and feel it is a good time to call a merge vote. Ive posted a
> > > megapatch version for folks who want to peruse the code. [3]
> > >
> > > What do you all think?
> > >
> > > Thanks,
> > > Jingcheng, Jon, Ram, and Anoop.
> > >
> > > [1] https://issues.apache.org/jira/browse/HBASE-11339
> > > [2] https://github.com/apache/hbase/tree/hbase-11339
> > > [3] https://reviews.apache.org/r/34475/
> > > --
> > > // Jonathan Hsieh (shay)
> > > // HBase Tech Lead, Software Engineer, Cloudera
> > > // jon@cloudera.com // @jmhsieh
> > >
> >
>
--
// Jonathan Hsieh (shay)
// HBase Tech Lead, Software Engineer, Cloudera
// jon@cloudera.com // @jmhsieh
Re: [DISCUSSION] Merge of the hbase-11339 mob branch into master.
Posted by 张铎 <pa...@gmail.com>.
Is there any comparison between HBASE-11339 and HDFS-7240?
Is their 'Object' a super set of our 'Medium Object'?
2015-05-21 10:38 GMT+08:00 Ted Yu <yu...@gmail.com>:
> This is a useful feature, Jon.
>
> I went over the mega-patch and left some comments on review board.
>
> I noticed that hbck was not included in the patch. Neither did I find a
> sub-task of HBASE-11339 that covers hbck.
>
> Do you or Jingcheng plan to add MOB-aware capability for hbck ?
>
> Cheers
>
> On Wed, May 20, 2015 at 9:21 AM, Jonathan Hsieh <jo...@cloudera.com> wrote:
>
> > Hi folks,
> >
> > The Medium Object (MOB) Storage feature (HBASE-11339[1]) is modified I/O
> > and compaction path that allows individual moderately sized values
> > (10k-10MB) to be stored so that write amplification is reduced when
> > compared to the normal I/O path. At a high level, it provides alternate
> > flush and compaction mechanisms that segregates large cells into a
> separate
> > area where they are not subject to potentially frequent compaction and
> > splits that can be encountered in the normal I/O path. A more detailed
> > design doc can be found on the hbase-11339 jira.
> >
> > Jingcheng Du has been working on the mob feature for a while and Anoop,
> Ram
> > and I have been shepherding him through the design revisions and
> > implementation of the feature in the hbase-11339 branch.[2]
> >
> > The branch we are proposing to merge into master is compatible with
> HBase's
> > core functionality including snapshots, replication, shell support,
> behaves
> > well with table alters, bulk loads and does not require external MR
> > processes. It has been documented, and subject to many integration test
> > runs (ITBLL, ITAcidGuarantees, ITIngest) including fault injection.
> > Performance testing of the feature shows what can be a 2x-3x throughput
> > improvement for workloads that contain mobs. These results can be seen on
> > the hbase 2.0 panel discussion slides from hbasecon (once published).
> >
> > Recently there have been some hfile encryption related shortcomings that
> we
> > could address in branch or in master.
> >
> > Earlier iterations of the feature has been tested in production by users
> > that Jingcheng has been responsible for. A version has also been
> deployed
> > at users I have been responsible for. Some of the folks from Huawei
> > (ashutosh) have also been submitting the recent encryption bug reports
> > against the hbase-11339 branch so there is some evidence of usage by
> them.
> >
> > The four of us (Jingcheng, Ram, Anoop and I) are satisfied with the
> > feature and feel it is a good time to call a merge vote. Ive posted a
> > megapatch version for folks who want to peruse the code. [3]
> >
> > What do you all think?
> >
> > Thanks,
> > Jingcheng, Jon, Ram, and Anoop.
> >
> > [1] https://issues.apache.org/jira/browse/HBASE-11339
> > [2] https://github.com/apache/hbase/tree/hbase-11339
> > [3] https://reviews.apache.org/r/34475/
> > --
> > // Jonathan Hsieh (shay)
> > // HBase Tech Lead, Software Engineer, Cloudera
> > // jon@cloudera.com // @jmhsieh
> >
>
Re: [DISCUSSION] Merge of the hbase-11339 mob branch into master.
Posted by Andrew Purtell <ap...@apache.org>.
So the sweep tool is completely optional? A deploy won't degrade if the
sweep tool is never run? Then that sounds good.
On Thursday, May 28, 2015, ramkrishna vasudevan <
ramkrishna.s.vasudevan@gmail.com> wrote:
> bq.So is a MR runtime required for MOB or not? I read maybe, then no, then
> here maybe again. What happens if one does not have a MR runtime and
> therefore can never run the sweeper tool?
> Just to make it clear, now MOB does not have MR dependency. The V1 version
> had a sweeper tool that was dependent on MR. The tool exists even now and
> that still depends on MR. Its like an add on.
>
> The compaction of MOB is now embedded as part of the core feature of
> compaction without having to use MR.
>
> Regards
> Ram
>
> On Thu, May 28, 2015 at 10:20 AM, Andrew Purtell <andrew.purtell@gmail.com
> <javascript:;>>
> wrote:
>
> > I have no concerns about MOB in trunk. Go for it.
> >
> > I do have concerns about a subsequent proposal to put it in 1.2. Those
> > concerns center around stability and performance impacts, and a possible
> > dependency on a MR runtime for what I would consider core function.
> >
> > > Regarding the tools and integrity checks,
> > MOB has a tool based on MR basically for sweeping and compaction apart
> from
> > the compactor that runs in the core (without MR dependency).
> >
> > So is a MR runtime required for MOB or not? I read maybe, then no, then
> > here maybe again. What happens if one does not have a MR runtime and
> > therefore can never run the sweeper tool? An incomplete feature on trunk
> > isn't a problem. Later commits can fill in the gaps and then the sum of
> MOB
> > commits can go back to branch-1. (Experimental != incomplete, IMHO.)
> >
> > If as you say stability and performance testing have already be done and
> > both look great, then that means *when* this is done again for a branch-1
> > merge candidate, the results will likely also be good. I'd like to help
> out
> > with this. You won't need to prove it, I will do the legwork for my own
> > concerns.
> >
> >
> > > On May 27, 2015, at 8:59 PM, ramkrishna vasudevan <
> > ramkrishna.s.vasudevan@gmail.com <javascript:;>> wrote:
> > >
> > > Chiming late here,
> > >
> > > As Matt suggested earlier, utmost care had been taken to ensure that
> the
> > > MOB code does not interfere with the normal flow and ensured that
> things
> > > work normally when MOB is not enabled on a family.
> > >
> > > So the entire flow for MOB can be treated as an experimental feature,
> if
> > > need be. Take the latest case of guys from Huawei, since they have
> some
> > > interest in this feature they are trying the branch hbase-11339 and
> > trying
> > > to see how MOB works.
> > >
> > > If we move this to trunk, then chances of even more people looking into
> > it
> > > and by the time it comes to 1.3 or1.4 we are stable enough.
> > >
> > > Regarding the tools and integrity checks,
> > > MOB has a tool based on MR basically for sweeping and compaction apart
> > from
> > > the compactor that runs in the core (without MR dependency). We could
> > > always add feature to the existing tool to do integrity checks like Jon
> > > suggests.
> > >
> > > .Also for an experimental feature we could always come up with such a
> > tool,
> > > but in case of MOB the inter dependency on the MOB and actual HFile
> data
> > is
> > > more so just a stand alone too to check integrity on the Hfile may not
> be
> > > easy without having to do some sort of scan on the Hfiles and MOB
> files.
> > > (Not thought on that fully).
> > >
> > > I would still think that having this feature as experimental in 1.2
> makes
> > > sense. Just my thoughts on this also after being part on the dev
> process
> > > for this feature where we tried not to touch the core areas affecting
> non
> > > MOB cases.
> > >
> > > Some of the perf results performed by Jingcheng's team and Cloudera
> folks
> > > substantiates the gain this feature provides.
> > >
> > > Regards
> > > Ram
> > >
> > >
> > >
> > >
> > >> On Thu, May 28, 2015 at 9:04 AM, Andrew Purtell <apurtell@apache.org
> <javascript:;>>
> > wrote:
> > >>
> > >> Inline
> > >>
> > >>> On Wednesday, May 27, 2015, Jonathan Hsieh <jon@cloudera.com
> <javascript:;>> wrote:
> > >>>
> > >>> On Sat, May 23, 2015 at 9:40 AM, Andrew Purtell <apurtell@apache.org
> <javascript:;>
> > >>> <javascript:;>> wrote:
> > >>>
> > >>>> Regarding performance testing: Whatever has been done on the MOB
> > branch
> > >>>> will be interesting data points, and, potentially encouraging, but
> > >>> porting
> > >>>> to branch-1 will produce a new code base. Earlier results on other
> > code
> > >>>> will not be applicable. We have to start over. Like I said
> elsewhere,
> > >> I'm
> > >>>> happy to help with (re)characterizing the perf impact and
> improvements
> > >>>> produced by the changes.
> > >>> Thank you for offer for help -- we'd appreciated it!
> > >> You bet.
> > >>
> > >>
> > >>> Although most of my it tests and perf tests results were done against
> > >>> against trunk (from sept '14 and then later feb '15 -- we've been
> doing
> > >>> them roughly every two weeks now) Jingcheng's most recent performance
> > >>> testing and fault injection testing results were actually done
> against
> > a
> > >>> version merged/rebased on to hbase 1.0.0[1]. Though not on the most
> > >> recent
> > >>> branch-1, would this be close enough and sufficient or would you
> still
> > >> want
> > >>> to redoing them?
> > >>
> > >>
> > >> Closer, yes.
> > >>
> > >> Redo on the branch-1 merge proposal would be important as a confidence
> > >> builder still I believe.
> > >>
> > >>
> > >>>
> > >>> If we want to redo them when we have a 1.x backport is ready to
> > propose,
> > >>> we'll include the augmented ltt[2] that will make it easy to exercise
> > the
> > >>> mob feature's performance.
> > >>>
> > >>> [1]
> https://github.com/cloudera/hbase/commits/cdh5-1.0.0_5.4.0?page=2
> > >>> (this is cdh5.4.0's hbase 1.0.0-based hbase)
> > >>> [2] https://issues.apache.org/jira/browse/HBASE-13277
> > >>>
> > >>>
> > >>> What coverage do we have for verifying the integrity of MOB
> references?
> > >>>> Will the sweep tool detect, alert on, and optionally repair dangling
> > >>>> references? (I could answer this for myself by looking at MOB
> branch,
> > >> but
> > >>>> hopefully someone here has an answer at the ready.) I assume we
> > >> calculate
> > >>>> and store checksums for MOB data itself so we know if values are
> > >> corrupt.
> > >>>> Does the sweep tool detect MOB value corruption? Can it be repaired?
> > Do
> > >>> we
> > >>>> have a good ops story for why HBCK is no longer sufficient on its
> own,
> > >>>> there's a separate tool with a whole new set of options - and a
> > >>> requirement
> > >>>> for a MR runtime! - for checking MOB data? That last one is a
> > >> rhetorical
> > >>>> question (smile), the ops story is... unsatisfying. It's like we've
> > >>> taken a
> > >>>> self sufficient HBase and bolted in parts of Hive, so now we need
> MR.
> > >>>>
> > >>>> Our internal compaction detects and alerts at warn level if there
> is a
> > >>> missing link [3], and then returns a empty value [4]
> > >>
> > >>
> > >> Ok, thanks
> > >>
> > >>
> > >>> Mobs are stored in hfiles so we have the same checksumming all other
> > >> hfiles
> > >>> have.
> > >>
> > >>
> > >> Ok, thanks
> > >>
> > >>
> > >>>
> > >>> In the other response, I answered about hbck and how something like
> > >>> Hfile.main() could be a more appropriate checking tool to address
> this
> > >>> situation.
> > >>
> > >>
> > >> Ok. Replied there.
> > >>
> > >>
> > >>>
> > >>> I'm afraid then much of our complete operational story is
> > "unsatisfying"
> > >>
> > >> even without mob because it still requires MR -- e.g. copytable,
> export,
> > >>> import, walplayer, or verifyreplicaion mr jobs. While I'll agree that
> > >>> having an external system is undesirable and unacceptable for what
> are
> > >>> mandatory internal operations like compactions, I think requiring mr
> > for
> > >> a
> > >>> verifiymob mr job would as acceptable as the verfiyreplication job.
> > >>
> > >>
> > >> I think integrity checks are a different class of tool than all others
> > and
> > >> we shouldn't mandate the presence of a MR runtime to execute those.
> > OTOH,
> > >> it's reasonable to provide a standalone tool (if multithreaded) but
> > >> then also a recommended MR version that can achieve better
> parallelism.
> > >>
> > >>
> > >>>
> > >>> [3]
> > >>
> >
> https://github.com/apache/hbase/blob/hbase-11339/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HMobStore.java#L400
> > >>> [4]
> > >>
> >
> https://github.com/apache/hbase/blob/hbase-11339/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobCompactor.java#L224
> > >>>
> > >>>>
> > >>>>> On Fri, May 22, 2015 at 1:45 PM, Jonathan Hsieh <jon@cloudera.com
> <javascript:;>
> > >>>> <javascript:;>> wrote:
> > >>>>
> > >>>>> In another thread andrew purtell brought up some concerns about the
> > >> mob
> > >>>>> feature:
> > >>>>>
> > >>>>> On Fri, May 22, 2015 at 12:40 PM, Andrew Purtell <
> > >> apurtell@apache.org <javascript:;>
> > >>> <javascript:;>>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> Another point of clarification, sorry, I hit the send button too
> > >>> early
> > >>>> it
> > >>>>>> seems: I don't believe MOB is fully integrated yet, for example
> the
> > >>>>>> feature
> > >>>>>> is an extension to store that lacks support for encryption (this
> > >>> would
> > >>>>>> technically be a feature regression); and HBCK. I have not been
> > >>>> following
> > >>>>>> MOB too closely so could be mistaken. These issues do not preclude
> > >> a
> > >>>>> merge
> > >>>>>> of MOB into trunk, but do preclude a merge back of MOB from trunk
> > >> to
> > >>>>>> branch-1. I would veto the latter until such shortcomings in the
> > >>>>>> implementation that could be described as regressions are
> > >> addressed.
> > >>> I
> > >>>>>> would also like to see a performance analysis of a range of
> > >> workloads
> > >>>>>> before and after in as much detail as can be mustered, and would
> be
> > >>>> happy
> > >>>>>> to volunteer to help out with that.
> > >>>>>
> > >>>>> Here's info on the points brought up:
> > >>>>>
> > >>>>> Encryption support shortcoming is being addrsessed here:
> > >>>>> https://issues.apache.org/jira/browse/HBASE-13693 (closed)
> > >>>>> https://issues.apache.org/jira/browse/HBASE-13720 (in review)
> > >>>>>
> > >>>>> Hbck has been actually run against the integration test rigs while
> > >> the
> > >>>>> feature has been enabled but currently has no explicit unit test or
> > >>>> simple
> > >>>>> to run integration test. It currently doesn't report anything
> > >> special
> > >>>>> about the mob storage area. We can add unit tests that cover hbck
> > >> when
> > >>>> the
> > >>>>> mob path is exercised.
> > >>>>>
> > >>>>> Another suggestion was a tool to check that mob references had
> > >>>>> corresponding mob data. We currently include a mr-based sweeper
> job
> > >>> that
> > >>>>> could be used to perform this verification. We can add this tool
> and
> > >>>>> testing for the tool.
> > >>>>>
> > >>>>> I've done some performance testing and Jingcheng and his colleagues
> > >>> have
> > >>>>> done significant amounts of performance testing. We currently have
> a
> > >>> blog
> > >>>>> post in progress that will share the results of this performance
> > >>> testing.
> > >>>>>
> > >>>>> Jon.
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> On Wed, May 20, 2015 at 7:38 PM, Ted Yu <yuzhihong@gmail.com
> <javascript:;>
> > >>> <javascript:;>> wrote:
> > >>>>>
> > >>>>>> This is a useful feature, Jon.
> > >>>>>>
> > >>>>>> I went over the mega-patch and left some comments on review board.
> > >>>>>>
> > >>>>>> I noticed that hbck was not included in the patch. Neither did I
> > >>> find a
> > >>>>>> sub-task of HBASE-11339 that covers hbck.
> > >>>>>>
> > >>>>>> Do you or Jingcheng plan to add MOB-aware capability for hbck ?
> > >>>>>>
> > >>>>>> Cheers
> > >>>>>>
> > >>>>>> On Wed, May 20, 2015 at 9:21 AM, Jonathan Hsieh <jon@cloudera.com
> <javascript:;>
> > >>> <javascript:;>>
> > >>>>> wrote:
> > >>>>>>
> > >>>>>>> Hi folks,
> > >>>>>>>
> > >>>>>>> The Medium Object (MOB) Storage feature (HBASE-11339[1]) is
> > >>> modified
> > >>>>> I/O
> > >>>>>>> and compaction path that allows individual moderately sized
> > >> values
> > >>>>>>> (10k-10MB) to be stored so that write amplification is reduced
> > >> when
> > >>>>>>> compared to the normal I/O path. At a high level, it provides
> > >>>>> alternate
> > >>>>>>> flush and compaction mechanisms that segregates large cells into
> > >> a
> > >>>>>> separate
> > >>>>>>> area where they are not subject to potentially frequent
> > >> compaction
> > >>>> and
> > >>>>>>> splits that can be encountered in the normal I/O path. A more
> > >>>> detailed
> > >>>>>>> design doc can be found on the hbase-11339 jira.
> > >>>>>>>
> > >>>>>>> Jingcheng Du has been working on the mob feature for a while and
> > >>>> Anoop,
> > >>>>>> Ram
> > >>>>>>> and I have been shepherding him through the design revisions and
> > >>>>>>> implementation of the feature in the hbase-11339 branch.[2]
> > >>>>>>>
> > >>>>>>> The branch we are proposing to merge into master is compatible
> > >> with
> > >>>>>> HBase's
> > >>>>>>> core functionality including snapshots, replication, shell
> > >> support,
> > >>>>>> behaves
> > >>>>>>> well with table alters, bulk loads and does not require external
> > >> MR
> > >>>>>>> processes. It has been documented, and subject to many
> > >> integration
> > >>>> test
> > >>>>>>> runs (ITBLL, ITAcidGuarantees, ITIngest) including fault
> > >>> injection.
> > >>>>>>> Performance testing of the feature shows what can be a 2x-3x
> > >>>> throughput
> > >>>>>>> improvement for workloads that contain mobs. These results can be
> > >>>> seen
> > >>>>> on
> > >>>>>>> the hbase 2.0 panel discussion slides from hbasecon (once
> > >>> published).
> > >>>>>>>
> > >>>>>>> Recently there have been some hfile encryption related
> > >> shortcomings
> > >>>>> that
> > >>>>>> we
> > >>>>>>> could address in branch or in master.
> > >>>>>>>
> > >>>>>>> Earlier iterations of the feature has been tested in production
> > >> by
> > >>>>> users
> > >>>>>>> that Jingcheng has been responsible for. A version has also been
> > >>>>>> deployed
> > >>>>>>> at users I have been responsible for. Some of the folks from
> > >>> Huawei
> > >>>>>>> (ashutosh) have also been submitting the recent encryption bug
> > >>>> reports
> > >>>>>>> against the hbase-11339 branch so there is some evidence of usage
> > >>> by
> > >>>>>> them.
> > >>>>>>>
> > >>>>>>> The four of us (Jingcheng, Ram, Anoop and I) are satisfied with
> > >>> the
> > >>>>>>> feature and feel it is a good time to call a merge vote. Ive
> > >>> posted
> > >>>> a
> > >>>>>>> megapatch version for folks who want to peruse the code. [3]
> > >>>>>>>
> > >>>>>>> What do you all think?
> > >>>>>>>
> > >>>>>>> Thanks,
> > >>>>>>> Jingcheng, Jon, Ram, and Anoop.
> > >>>>>>>
> > >>>>>>> [1] https://issues.apache.org/jira/browse/HBASE-11339
> > >>>>>>> [2] https://github.com/apache/hbase/tree/hbase-11339
> > >>>>>>> [3] https://reviews.apache.org/r/34475/
> > >>>>>>> --
> > >>>>>>> // Jonathan Hsieh (shay)
> > >>>>>>> // HBase Tech Lead, Software Engineer, Cloudera
> > >>>>>>> // jon@cloudera.com <javascript:;> <javascript:;> // @jmhsieh
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> --
> > >>>>> // Jonathan Hsieh (shay)
> > >>>>> // HBase Tech Lead, Software Engineer, Cloudera
> > >>>>> // jon@cloudera.com <javascript:;> <javascript:;> // @jmhsieh
> > >>>>
> > >>>>
> > >>>>
> > >>>> --
> > >>>> Best regards,
> > >>>>
> > >>>> - Andy
> > >>>>
> > >>>> Problems worthy of attack prove their worth by hitting back. - Piet
> > >> Hein
> > >>>> (via Tom White)
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> // Jonathan Hsieh (shay)
> > >>> // HBase Tech Lead, Software Engineer, Cloudera
> > >>> // jon@cloudera.com <javascript:;> <javascript:;> // @jmhsieh
> > >>
> > >>
> > >> --
> > >> Best regards,
> > >>
> > >> - Andy
> > >>
> > >> Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> > >> (via Tom White)
> > >>
> >
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: [DISCUSSION] Merge of the hbase-11339 mob branch into master.
Posted by ramkrishna vasudevan <ra...@gmail.com>.
bq.So is a MR runtime required for MOB or not? I read maybe, then no, then
here maybe again. What happens if one does not have a MR runtime and
therefore can never run the sweeper tool?
Just to make it clear, now MOB does not have MR dependency. The V1 version
had a sweeper tool that was dependent on MR. The tool exists even now and
that still depends on MR. Its like an add on.
The compaction of MOB is now embedded as part of the core feature of
compaction without having to use MR.
Regards
Ram
On Thu, May 28, 2015 at 10:20 AM, Andrew Purtell <an...@gmail.com>
wrote:
> I have no concerns about MOB in trunk. Go for it.
>
> I do have concerns about a subsequent proposal to put it in 1.2. Those
> concerns center around stability and performance impacts, and a possible
> dependency on a MR runtime for what I would consider core function.
>
> > Regarding the tools and integrity checks,
> MOB has a tool based on MR basically for sweeping and compaction apart from
> the compactor that runs in the core (without MR dependency).
>
> So is a MR runtime required for MOB or not? I read maybe, then no, then
> here maybe again. What happens if one does not have a MR runtime and
> therefore can never run the sweeper tool? An incomplete feature on trunk
> isn't a problem. Later commits can fill in the gaps and then the sum of MOB
> commits can go back to branch-1. (Experimental != incomplete, IMHO.)
>
> If as you say stability and performance testing have already be done and
> both look great, then that means *when* this is done again for a branch-1
> merge candidate, the results will likely also be good. I'd like to help out
> with this. You won't need to prove it, I will do the legwork for my own
> concerns.
>
>
> > On May 27, 2015, at 8:59 PM, ramkrishna vasudevan <
> ramkrishna.s.vasudevan@gmail.com> wrote:
> >
> > Chiming late here,
> >
> > As Matt suggested earlier, utmost care had been taken to ensure that the
> > MOB code does not interfere with the normal flow and ensured that things
> > work normally when MOB is not enabled on a family.
> >
> > So the entire flow for MOB can be treated as an experimental feature, if
> > need be. Take the latest case of guys from Huawei, since they have some
> > interest in this feature they are trying the branch hbase-11339 and
> trying
> > to see how MOB works.
> >
> > If we move this to trunk, then chances of even more people looking into
> it
> > and by the time it comes to 1.3 or1.4 we are stable enough.
> >
> > Regarding the tools and integrity checks,
> > MOB has a tool based on MR basically for sweeping and compaction apart
> from
> > the compactor that runs in the core (without MR dependency). We could
> > always add feature to the existing tool to do integrity checks like Jon
> > suggests.
> >
> > .Also for an experimental feature we could always come up with such a
> tool,
> > but in case of MOB the inter dependency on the MOB and actual HFile data
> is
> > more so just a stand alone too to check integrity on the Hfile may not be
> > easy without having to do some sort of scan on the Hfiles and MOB files.
> > (Not thought on that fully).
> >
> > I would still think that having this feature as experimental in 1.2 makes
> > sense. Just my thoughts on this also after being part on the dev process
> > for this feature where we tried not to touch the core areas affecting non
> > MOB cases.
> >
> > Some of the perf results performed by Jingcheng's team and Cloudera folks
> > substantiates the gain this feature provides.
> >
> > Regards
> > Ram
> >
> >
> >
> >
> >> On Thu, May 28, 2015 at 9:04 AM, Andrew Purtell <ap...@apache.org>
> wrote:
> >>
> >> Inline
> >>
> >>> On Wednesday, May 27, 2015, Jonathan Hsieh <jo...@cloudera.com> wrote:
> >>>
> >>> On Sat, May 23, 2015 at 9:40 AM, Andrew Purtell <apurtell@apache.org
> >>> <javascript:;>> wrote:
> >>>
> >>>> Regarding performance testing: Whatever has been done on the MOB
> branch
> >>>> will be interesting data points, and, potentially encouraging, but
> >>> porting
> >>>> to branch-1 will produce a new code base. Earlier results on other
> code
> >>>> will not be applicable. We have to start over. Like I said elsewhere,
> >> I'm
> >>>> happy to help with (re)characterizing the perf impact and improvements
> >>>> produced by the changes.
> >>> Thank you for offer for help -- we'd appreciated it!
> >> You bet.
> >>
> >>
> >>> Although most of my it tests and perf tests results were done against
> >>> against trunk (from sept '14 and then later feb '15 -- we've been doing
> >>> them roughly every two weeks now) Jingcheng's most recent performance
> >>> testing and fault injection testing results were actually done against
> a
> >>> version merged/rebased on to hbase 1.0.0[1]. Though not on the most
> >> recent
> >>> branch-1, would this be close enough and sufficient or would you still
> >> want
> >>> to redoing them?
> >>
> >>
> >> Closer, yes.
> >>
> >> Redo on the branch-1 merge proposal would be important as a confidence
> >> builder still I believe.
> >>
> >>
> >>>
> >>> If we want to redo them when we have a 1.x backport is ready to
> propose,
> >>> we'll include the augmented ltt[2] that will make it easy to exercise
> the
> >>> mob feature's performance.
> >>>
> >>> [1] https://github.com/cloudera/hbase/commits/cdh5-1.0.0_5.4.0?page=2
> >>> (this is cdh5.4.0's hbase 1.0.0-based hbase)
> >>> [2] https://issues.apache.org/jira/browse/HBASE-13277
> >>>
> >>>
> >>> What coverage do we have for verifying the integrity of MOB references?
> >>>> Will the sweep tool detect, alert on, and optionally repair dangling
> >>>> references? (I could answer this for myself by looking at MOB branch,
> >> but
> >>>> hopefully someone here has an answer at the ready.) I assume we
> >> calculate
> >>>> and store checksums for MOB data itself so we know if values are
> >> corrupt.
> >>>> Does the sweep tool detect MOB value corruption? Can it be repaired?
> Do
> >>> we
> >>>> have a good ops story for why HBCK is no longer sufficient on its own,
> >>>> there's a separate tool with a whole new set of options - and a
> >>> requirement
> >>>> for a MR runtime! - for checking MOB data? That last one is a
> >> rhetorical
> >>>> question (smile), the ops story is... unsatisfying. It's like we've
> >>> taken a
> >>>> self sufficient HBase and bolted in parts of Hive, so now we need MR.
> >>>>
> >>>> Our internal compaction detects and alerts at warn level if there is a
> >>> missing link [3], and then returns a empty value [4]
> >>
> >>
> >> Ok, thanks
> >>
> >>
> >>> Mobs are stored in hfiles so we have the same checksumming all other
> >> hfiles
> >>> have.
> >>
> >>
> >> Ok, thanks
> >>
> >>
> >>>
> >>> In the other response, I answered about hbck and how something like
> >>> Hfile.main() could be a more appropriate checking tool to address this
> >>> situation.
> >>
> >>
> >> Ok. Replied there.
> >>
> >>
> >>>
> >>> I'm afraid then much of our complete operational story is
> "unsatisfying"
> >>
> >> even without mob because it still requires MR -- e.g. copytable, export,
> >>> import, walplayer, or verifyreplicaion mr jobs. While I'll agree that
> >>> having an external system is undesirable and unacceptable for what are
> >>> mandatory internal operations like compactions, I think requiring mr
> for
> >> a
> >>> verifiymob mr job would as acceptable as the verfiyreplication job.
> >>
> >>
> >> I think integrity checks are a different class of tool than all others
> and
> >> we shouldn't mandate the presence of a MR runtime to execute those.
> OTOH,
> >> it's reasonable to provide a standalone tool (if multithreaded) but
> >> then also a recommended MR version that can achieve better parallelism.
> >>
> >>
> >>>
> >>> [3]
> >>
> https://github.com/apache/hbase/blob/hbase-11339/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HMobStore.java#L400
> >>> [4]
> >>
> https://github.com/apache/hbase/blob/hbase-11339/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobCompactor.java#L224
> >>>
> >>>>
> >>>>> On Fri, May 22, 2015 at 1:45 PM, Jonathan Hsieh <jon@cloudera.com
> >>>> <javascript:;>> wrote:
> >>>>
> >>>>> In another thread andrew purtell brought up some concerns about the
> >> mob
> >>>>> feature:
> >>>>>
> >>>>> On Fri, May 22, 2015 at 12:40 PM, Andrew Purtell <
> >> apurtell@apache.org
> >>> <javascript:;>>
> >>>>> wrote:
> >>>>>
> >>>>>> Another point of clarification, sorry, I hit the send button too
> >>> early
> >>>> it
> >>>>>> seems: I don't believe MOB is fully integrated yet, for example the
> >>>>>> feature
> >>>>>> is an extension to store that lacks support for encryption (this
> >>> would
> >>>>>> technically be a feature regression); and HBCK. I have not been
> >>>> following
> >>>>>> MOB too closely so could be mistaken. These issues do not preclude
> >> a
> >>>>> merge
> >>>>>> of MOB into trunk, but do preclude a merge back of MOB from trunk
> >> to
> >>>>>> branch-1. I would veto the latter until such shortcomings in the
> >>>>>> implementation that could be described as regressions are
> >> addressed.
> >>> I
> >>>>>> would also like to see a performance analysis of a range of
> >> workloads
> >>>>>> before and after in as much detail as can be mustered, and would be
> >>>> happy
> >>>>>> to volunteer to help out with that.
> >>>>>
> >>>>> Here's info on the points brought up:
> >>>>>
> >>>>> Encryption support shortcoming is being addrsessed here:
> >>>>> https://issues.apache.org/jira/browse/HBASE-13693 (closed)
> >>>>> https://issues.apache.org/jira/browse/HBASE-13720 (in review)
> >>>>>
> >>>>> Hbck has been actually run against the integration test rigs while
> >> the
> >>>>> feature has been enabled but currently has no explicit unit test or
> >>>> simple
> >>>>> to run integration test. It currently doesn't report anything
> >> special
> >>>>> about the mob storage area. We can add unit tests that cover hbck
> >> when
> >>>> the
> >>>>> mob path is exercised.
> >>>>>
> >>>>> Another suggestion was a tool to check that mob references had
> >>>>> corresponding mob data. We currently include a mr-based sweeper job
> >>> that
> >>>>> could be used to perform this verification. We can add this tool and
> >>>>> testing for the tool.
> >>>>>
> >>>>> I've done some performance testing and Jingcheng and his colleagues
> >>> have
> >>>>> done significant amounts of performance testing. We currently have a
> >>> blog
> >>>>> post in progress that will share the results of this performance
> >>> testing.
> >>>>>
> >>>>> Jon.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Wed, May 20, 2015 at 7:38 PM, Ted Yu <yuzhihong@gmail.com
> >>> <javascript:;>> wrote:
> >>>>>
> >>>>>> This is a useful feature, Jon.
> >>>>>>
> >>>>>> I went over the mega-patch and left some comments on review board.
> >>>>>>
> >>>>>> I noticed that hbck was not included in the patch. Neither did I
> >>> find a
> >>>>>> sub-task of HBASE-11339 that covers hbck.
> >>>>>>
> >>>>>> Do you or Jingcheng plan to add MOB-aware capability for hbck ?
> >>>>>>
> >>>>>> Cheers
> >>>>>>
> >>>>>> On Wed, May 20, 2015 at 9:21 AM, Jonathan Hsieh <jon@cloudera.com
> >>> <javascript:;>>
> >>>>> wrote:
> >>>>>>
> >>>>>>> Hi folks,
> >>>>>>>
> >>>>>>> The Medium Object (MOB) Storage feature (HBASE-11339[1]) is
> >>> modified
> >>>>> I/O
> >>>>>>> and compaction path that allows individual moderately sized
> >> values
> >>>>>>> (10k-10MB) to be stored so that write amplification is reduced
> >> when
> >>>>>>> compared to the normal I/O path. At a high level, it provides
> >>>>> alternate
> >>>>>>> flush and compaction mechanisms that segregates large cells into
> >> a
> >>>>>> separate
> >>>>>>> area where they are not subject to potentially frequent
> >> compaction
> >>>> and
> >>>>>>> splits that can be encountered in the normal I/O path. A more
> >>>> detailed
> >>>>>>> design doc can be found on the hbase-11339 jira.
> >>>>>>>
> >>>>>>> Jingcheng Du has been working on the mob feature for a while and
> >>>> Anoop,
> >>>>>> Ram
> >>>>>>> and I have been shepherding him through the design revisions and
> >>>>>>> implementation of the feature in the hbase-11339 branch.[2]
> >>>>>>>
> >>>>>>> The branch we are proposing to merge into master is compatible
> >> with
> >>>>>> HBase's
> >>>>>>> core functionality including snapshots, replication, shell
> >> support,
> >>>>>> behaves
> >>>>>>> well with table alters, bulk loads and does not require external
> >> MR
> >>>>>>> processes. It has been documented, and subject to many
> >> integration
> >>>> test
> >>>>>>> runs (ITBLL, ITAcidGuarantees, ITIngest) including fault
> >>> injection.
> >>>>>>> Performance testing of the feature shows what can be a 2x-3x
> >>>> throughput
> >>>>>>> improvement for workloads that contain mobs. These results can be
> >>>> seen
> >>>>> on
> >>>>>>> the hbase 2.0 panel discussion slides from hbasecon (once
> >>> published).
> >>>>>>>
> >>>>>>> Recently there have been some hfile encryption related
> >> shortcomings
> >>>>> that
> >>>>>> we
> >>>>>>> could address in branch or in master.
> >>>>>>>
> >>>>>>> Earlier iterations of the feature has been tested in production
> >> by
> >>>>> users
> >>>>>>> that Jingcheng has been responsible for. A version has also been
> >>>>>> deployed
> >>>>>>> at users I have been responsible for. Some of the folks from
> >>> Huawei
> >>>>>>> (ashutosh) have also been submitting the recent encryption bug
> >>>> reports
> >>>>>>> against the hbase-11339 branch so there is some evidence of usage
> >>> by
> >>>>>> them.
> >>>>>>>
> >>>>>>> The four of us (Jingcheng, Ram, Anoop and I) are satisfied with
> >>> the
> >>>>>>> feature and feel it is a good time to call a merge vote. Ive
> >>> posted
> >>>> a
> >>>>>>> megapatch version for folks who want to peruse the code. [3]
> >>>>>>>
> >>>>>>> What do you all think?
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> Jingcheng, Jon, Ram, and Anoop.
> >>>>>>>
> >>>>>>> [1] https://issues.apache.org/jira/browse/HBASE-11339
> >>>>>>> [2] https://github.com/apache/hbase/tree/hbase-11339
> >>>>>>> [3] https://reviews.apache.org/r/34475/
> >>>>>>> --
> >>>>>>> // Jonathan Hsieh (shay)
> >>>>>>> // HBase Tech Lead, Software Engineer, Cloudera
> >>>>>>> // jon@cloudera.com <javascript:;> // @jmhsieh
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> // Jonathan Hsieh (shay)
> >>>>> // HBase Tech Lead, Software Engineer, Cloudera
> >>>>> // jon@cloudera.com <javascript:;> // @jmhsieh
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Best regards,
> >>>>
> >>>> - Andy
> >>>>
> >>>> Problems worthy of attack prove their worth by hitting back. - Piet
> >> Hein
> >>>> (via Tom White)
> >>>
> >>>
> >>>
> >>> --
> >>> // Jonathan Hsieh (shay)
> >>> // HBase Tech Lead, Software Engineer, Cloudera
> >>> // jon@cloudera.com <javascript:;> // @jmhsieh
> >>
> >>
> >> --
> >> Best regards,
> >>
> >> - Andy
> >>
> >> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> >> (via Tom White)
> >>
>
Re: [DISCUSSION] Merge of the hbase-11339 mob branch into master.
Posted by Andrew Purtell <an...@gmail.com>.
I have no concerns about MOB in trunk. Go for it.
I do have concerns about a subsequent proposal to put it in 1.2. Those concerns center around stability and performance impacts, and a possible dependency on a MR runtime for what I would consider core function.
> Regarding the tools and integrity checks,
MOB has a tool based on MR basically for sweeping and compaction apart from
the compactor that runs in the core (without MR dependency).
So is a MR runtime required for MOB or not? I read maybe, then no, then here maybe again. What happens if one does not have a MR runtime and therefore can never run the sweeper tool? An incomplete feature on trunk isn't a problem. Later commits can fill in the gaps and then the sum of MOB commits can go back to branch-1. (Experimental != incomplete, IMHO.)
If as you say stability and performance testing have already be done and both look great, then that means *when* this is done again for a branch-1 merge candidate, the results will likely also be good. I'd like to help out with this. You won't need to prove it, I will do the legwork for my own concerns.
> On May 27, 2015, at 8:59 PM, ramkrishna vasudevan <ra...@gmail.com> wrote:
>
> Chiming late here,
>
> As Matt suggested earlier, utmost care had been taken to ensure that the
> MOB code does not interfere with the normal flow and ensured that things
> work normally when MOB is not enabled on a family.
>
> So the entire flow for MOB can be treated as an experimental feature, if
> need be. Take the latest case of guys from Huawei, since they have some
> interest in this feature they are trying the branch hbase-11339 and trying
> to see how MOB works.
>
> If we move this to trunk, then chances of even more people looking into it
> and by the time it comes to 1.3 or1.4 we are stable enough.
>
> Regarding the tools and integrity checks,
> MOB has a tool based on MR basically for sweeping and compaction apart from
> the compactor that runs in the core (without MR dependency). We could
> always add feature to the existing tool to do integrity checks like Jon
> suggests.
>
> .Also for an experimental feature we could always come up with such a tool,
> but in case of MOB the inter dependency on the MOB and actual HFile data is
> more so just a stand alone too to check integrity on the Hfile may not be
> easy without having to do some sort of scan on the Hfiles and MOB files.
> (Not thought on that fully).
>
> I would still think that having this feature as experimental in 1.2 makes
> sense. Just my thoughts on this also after being part on the dev process
> for this feature where we tried not to touch the core areas affecting non
> MOB cases.
>
> Some of the perf results performed by Jingcheng's team and Cloudera folks
> substantiates the gain this feature provides.
>
> Regards
> Ram
>
>
>
>
>> On Thu, May 28, 2015 at 9:04 AM, Andrew Purtell <ap...@apache.org> wrote:
>>
>> Inline
>>
>>> On Wednesday, May 27, 2015, Jonathan Hsieh <jo...@cloudera.com> wrote:
>>>
>>> On Sat, May 23, 2015 at 9:40 AM, Andrew Purtell <apurtell@apache.org
>>> <javascript:;>> wrote:
>>>
>>>> Regarding performance testing: Whatever has been done on the MOB branch
>>>> will be interesting data points, and, potentially encouraging, but
>>> porting
>>>> to branch-1 will produce a new code base. Earlier results on other code
>>>> will not be applicable. We have to start over. Like I said elsewhere,
>> I'm
>>>> happy to help with (re)characterizing the perf impact and improvements
>>>> produced by the changes.
>>> Thank you for offer for help -- we'd appreciated it!
>> You bet.
>>
>>
>>> Although most of my it tests and perf tests results were done against
>>> against trunk (from sept '14 and then later feb '15 -- we've been doing
>>> them roughly every two weeks now) Jingcheng's most recent performance
>>> testing and fault injection testing results were actually done against a
>>> version merged/rebased on to hbase 1.0.0[1]. Though not on the most
>> recent
>>> branch-1, would this be close enough and sufficient or would you still
>> want
>>> to redoing them?
>>
>>
>> Closer, yes.
>>
>> Redo on the branch-1 merge proposal would be important as a confidence
>> builder still I believe.
>>
>>
>>>
>>> If we want to redo them when we have a 1.x backport is ready to propose,
>>> we'll include the augmented ltt[2] that will make it easy to exercise the
>>> mob feature's performance.
>>>
>>> [1] https://github.com/cloudera/hbase/commits/cdh5-1.0.0_5.4.0?page=2
>>> (this is cdh5.4.0's hbase 1.0.0-based hbase)
>>> [2] https://issues.apache.org/jira/browse/HBASE-13277
>>>
>>>
>>> What coverage do we have for verifying the integrity of MOB references?
>>>> Will the sweep tool detect, alert on, and optionally repair dangling
>>>> references? (I could answer this for myself by looking at MOB branch,
>> but
>>>> hopefully someone here has an answer at the ready.) I assume we
>> calculate
>>>> and store checksums for MOB data itself so we know if values are
>> corrupt.
>>>> Does the sweep tool detect MOB value corruption? Can it be repaired? Do
>>> we
>>>> have a good ops story for why HBCK is no longer sufficient on its own,
>>>> there's a separate tool with a whole new set of options - and a
>>> requirement
>>>> for a MR runtime! - for checking MOB data? That last one is a
>> rhetorical
>>>> question (smile), the ops story is... unsatisfying. It's like we've
>>> taken a
>>>> self sufficient HBase and bolted in parts of Hive, so now we need MR.
>>>>
>>>> Our internal compaction detects and alerts at warn level if there is a
>>> missing link [3], and then returns a empty value [4]
>>
>>
>> Ok, thanks
>>
>>
>>> Mobs are stored in hfiles so we have the same checksumming all other
>> hfiles
>>> have.
>>
>>
>> Ok, thanks
>>
>>
>>>
>>> In the other response, I answered about hbck and how something like
>>> Hfile.main() could be a more appropriate checking tool to address this
>>> situation.
>>
>>
>> Ok. Replied there.
>>
>>
>>>
>>> I'm afraid then much of our complete operational story is "unsatisfying"
>>
>> even without mob because it still requires MR -- e.g. copytable, export,
>>> import, walplayer, or verifyreplicaion mr jobs. While I'll agree that
>>> having an external system is undesirable and unacceptable for what are
>>> mandatory internal operations like compactions, I think requiring mr for
>> a
>>> verifiymob mr job would as acceptable as the verfiyreplication job.
>>
>>
>> I think integrity checks are a different class of tool than all others and
>> we shouldn't mandate the presence of a MR runtime to execute those. OTOH,
>> it's reasonable to provide a standalone tool (if multithreaded) but
>> then also a recommended MR version that can achieve better parallelism.
>>
>>
>>>
>>> [3]
>> https://github.com/apache/hbase/blob/hbase-11339/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HMobStore.java#L400
>>> [4]
>> https://github.com/apache/hbase/blob/hbase-11339/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobCompactor.java#L224
>>>
>>>>
>>>>> On Fri, May 22, 2015 at 1:45 PM, Jonathan Hsieh <jon@cloudera.com
>>>> <javascript:;>> wrote:
>>>>
>>>>> In another thread andrew purtell brought up some concerns about the
>> mob
>>>>> feature:
>>>>>
>>>>> On Fri, May 22, 2015 at 12:40 PM, Andrew Purtell <
>> apurtell@apache.org
>>> <javascript:;>>
>>>>> wrote:
>>>>>
>>>>>> Another point of clarification, sorry, I hit the send button too
>>> early
>>>> it
>>>>>> seems: I don't believe MOB is fully integrated yet, for example the
>>>>>> feature
>>>>>> is an extension to store that lacks support for encryption (this
>>> would
>>>>>> technically be a feature regression); and HBCK. I have not been
>>>> following
>>>>>> MOB too closely so could be mistaken. These issues do not preclude
>> a
>>>>> merge
>>>>>> of MOB into trunk, but do preclude a merge back of MOB from trunk
>> to
>>>>>> branch-1. I would veto the latter until such shortcomings in the
>>>>>> implementation that could be described as regressions are
>> addressed.
>>> I
>>>>>> would also like to see a performance analysis of a range of
>> workloads
>>>>>> before and after in as much detail as can be mustered, and would be
>>>> happy
>>>>>> to volunteer to help out with that.
>>>>>
>>>>> Here's info on the points brought up:
>>>>>
>>>>> Encryption support shortcoming is being addrsessed here:
>>>>> https://issues.apache.org/jira/browse/HBASE-13693 (closed)
>>>>> https://issues.apache.org/jira/browse/HBASE-13720 (in review)
>>>>>
>>>>> Hbck has been actually run against the integration test rigs while
>> the
>>>>> feature has been enabled but currently has no explicit unit test or
>>>> simple
>>>>> to run integration test. It currently doesn't report anything
>> special
>>>>> about the mob storage area. We can add unit tests that cover hbck
>> when
>>>> the
>>>>> mob path is exercised.
>>>>>
>>>>> Another suggestion was a tool to check that mob references had
>>>>> corresponding mob data. We currently include a mr-based sweeper job
>>> that
>>>>> could be used to perform this verification. We can add this tool and
>>>>> testing for the tool.
>>>>>
>>>>> I've done some performance testing and Jingcheng and his colleagues
>>> have
>>>>> done significant amounts of performance testing. We currently have a
>>> blog
>>>>> post in progress that will share the results of this performance
>>> testing.
>>>>>
>>>>> Jon.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Wed, May 20, 2015 at 7:38 PM, Ted Yu <yuzhihong@gmail.com
>>> <javascript:;>> wrote:
>>>>>
>>>>>> This is a useful feature, Jon.
>>>>>>
>>>>>> I went over the mega-patch and left some comments on review board.
>>>>>>
>>>>>> I noticed that hbck was not included in the patch. Neither did I
>>> find a
>>>>>> sub-task of HBASE-11339 that covers hbck.
>>>>>>
>>>>>> Do you or Jingcheng plan to add MOB-aware capability for hbck ?
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>> On Wed, May 20, 2015 at 9:21 AM, Jonathan Hsieh <jon@cloudera.com
>>> <javascript:;>>
>>>>> wrote:
>>>>>>
>>>>>>> Hi folks,
>>>>>>>
>>>>>>> The Medium Object (MOB) Storage feature (HBASE-11339[1]) is
>>> modified
>>>>> I/O
>>>>>>> and compaction path that allows individual moderately sized
>> values
>>>>>>> (10k-10MB) to be stored so that write amplification is reduced
>> when
>>>>>>> compared to the normal I/O path. At a high level, it provides
>>>>> alternate
>>>>>>> flush and compaction mechanisms that segregates large cells into
>> a
>>>>>> separate
>>>>>>> area where they are not subject to potentially frequent
>> compaction
>>>> and
>>>>>>> splits that can be encountered in the normal I/O path. A more
>>>> detailed
>>>>>>> design doc can be found on the hbase-11339 jira.
>>>>>>>
>>>>>>> Jingcheng Du has been working on the mob feature for a while and
>>>> Anoop,
>>>>>> Ram
>>>>>>> and I have been shepherding him through the design revisions and
>>>>>>> implementation of the feature in the hbase-11339 branch.[2]
>>>>>>>
>>>>>>> The branch we are proposing to merge into master is compatible
>> with
>>>>>> HBase's
>>>>>>> core functionality including snapshots, replication, shell
>> support,
>>>>>> behaves
>>>>>>> well with table alters, bulk loads and does not require external
>> MR
>>>>>>> processes. It has been documented, and subject to many
>> integration
>>>> test
>>>>>>> runs (ITBLL, ITAcidGuarantees, ITIngest) including fault
>>> injection.
>>>>>>> Performance testing of the feature shows what can be a 2x-3x
>>>> throughput
>>>>>>> improvement for workloads that contain mobs. These results can be
>>>> seen
>>>>> on
>>>>>>> the hbase 2.0 panel discussion slides from hbasecon (once
>>> published).
>>>>>>>
>>>>>>> Recently there have been some hfile encryption related
>> shortcomings
>>>>> that
>>>>>> we
>>>>>>> could address in branch or in master.
>>>>>>>
>>>>>>> Earlier iterations of the feature has been tested in production
>> by
>>>>> users
>>>>>>> that Jingcheng has been responsible for. A version has also been
>>>>>> deployed
>>>>>>> at users I have been responsible for. Some of the folks from
>>> Huawei
>>>>>>> (ashutosh) have also been submitting the recent encryption bug
>>>> reports
>>>>>>> against the hbase-11339 branch so there is some evidence of usage
>>> by
>>>>>> them.
>>>>>>>
>>>>>>> The four of us (Jingcheng, Ram, Anoop and I) are satisfied with
>>> the
>>>>>>> feature and feel it is a good time to call a merge vote. Ive
>>> posted
>>>> a
>>>>>>> megapatch version for folks who want to peruse the code. [3]
>>>>>>>
>>>>>>> What do you all think?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Jingcheng, Jon, Ram, and Anoop.
>>>>>>>
>>>>>>> [1] https://issues.apache.org/jira/browse/HBASE-11339
>>>>>>> [2] https://github.com/apache/hbase/tree/hbase-11339
>>>>>>> [3] https://reviews.apache.org/r/34475/
>>>>>>> --
>>>>>>> // Jonathan Hsieh (shay)
>>>>>>> // HBase Tech Lead, Software Engineer, Cloudera
>>>>>>> // jon@cloudera.com <javascript:;> // @jmhsieh
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> // Jonathan Hsieh (shay)
>>>>> // HBase Tech Lead, Software Engineer, Cloudera
>>>>> // jon@cloudera.com <javascript:;> // @jmhsieh
>>>>
>>>>
>>>>
>>>> --
>>>> Best regards,
>>>>
>>>> - Andy
>>>>
>>>> Problems worthy of attack prove their worth by hitting back. - Piet
>> Hein
>>>> (via Tom White)
>>>
>>>
>>>
>>> --
>>> // Jonathan Hsieh (shay)
>>> // HBase Tech Lead, Software Engineer, Cloudera
>>> // jon@cloudera.com <javascript:;> // @jmhsieh
>>
>>
>> --
>> Best regards,
>>
>> - Andy
>>
>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> (via Tom White)
>>
Re: [DISCUSSION] Merge of the hbase-11339 mob branch into master.
Posted by ramkrishna vasudevan <ra...@gmail.com>.
Chiming late here,
As Matt suggested earlier, utmost care had been taken to ensure that the
MOB code does not interfere with the normal flow and ensured that things
work normally when MOB is not enabled on a family.
So the entire flow for MOB can be treated as an experimental feature, if
need be. Take the latest case of guys from Huawei, since they have some
interest in this feature they are trying the branch hbase-11339 and trying
to see how MOB works.
If we move this to trunk, then chances of even more people looking into it
and by the time it comes to 1.3 or1.4 we are stable enough.
Regarding the tools and integrity checks,
MOB has a tool based on MR basically for sweeping and compaction apart from
the compactor that runs in the core (without MR dependency). We could
always add feature to the existing tool to do integrity checks like Jon
suggests.
.Also for an experimental feature we could always come up with such a tool,
but in case of MOB the inter dependency on the MOB and actual HFile data is
more so just a stand alone too to check integrity on the Hfile may not be
easy without having to do some sort of scan on the Hfiles and MOB files.
(Not thought on that fully).
I would still think that having this feature as experimental in 1.2 makes
sense. Just my thoughts on this also after being part on the dev process
for this feature where we tried not to touch the core areas affecting non
MOB cases.
Some of the perf results performed by Jingcheng's team and Cloudera folks
substantiates the gain this feature provides.
Regards
Ram
On Thu, May 28, 2015 at 9:04 AM, Andrew Purtell <ap...@apache.org> wrote:
> Inline
>
> On Wednesday, May 27, 2015, Jonathan Hsieh <jo...@cloudera.com> wrote:
>
> > On Sat, May 23, 2015 at 9:40 AM, Andrew Purtell <apurtell@apache.org
> > <javascript:;>> wrote:
> >
> > > Regarding performance testing: Whatever has been done on the MOB branch
> > > will be interesting data points, and, potentially encouraging, but
> > porting
> > > to branch-1 will produce a new code base. Earlier results on other code
> > > will not be applicable. We have to start over. Like I said elsewhere,
> I'm
> > > happy to help with (re)characterizing the perf impact and improvements
> > > produced by the changes.
> > >
> > >
> > Thank you for offer for help -- we'd appreciated it!
> >
> >
> You bet.
>
>
> > Although most of my it tests and perf tests results were done against
> > against trunk (from sept '14 and then later feb '15 -- we've been doing
> > them roughly every two weeks now) Jingcheng's most recent performance
> > testing and fault injection testing results were actually done against a
> > version merged/rebased on to hbase 1.0.0[1]. Though not on the most
> recent
> > branch-1, would this be close enough and sufficient or would you still
> want
> > to redoing them?
>
>
> Closer, yes.
>
> Redo on the branch-1 merge proposal would be important as a confidence
> builder still I believe.
>
>
> >
> > If we want to redo them when we have a 1.x backport is ready to propose,
> > we'll include the augmented ltt[2] that will make it easy to exercise the
> > mob feature's performance.
> >
> > [1] https://github.com/cloudera/hbase/commits/cdh5-1.0.0_5.4.0?page=2
> > (this is cdh5.4.0's hbase 1.0.0-based hbase)
> > [2] https://issues.apache.org/jira/browse/HBASE-13277
> >
> >
> > What coverage do we have for verifying the integrity of MOB references?
> > > Will the sweep tool detect, alert on, and optionally repair dangling
> > > references? (I could answer this for myself by looking at MOB branch,
> but
> > > hopefully someone here has an answer at the ready.) I assume we
> calculate
> > > and store checksums for MOB data itself so we know if values are
> corrupt.
> > > Does the sweep tool detect MOB value corruption? Can it be repaired? Do
> > we
> > > have a good ops story for why HBCK is no longer sufficient on its own,
> > > there's a separate tool with a whole new set of options - and a
> > requirement
> > > for a MR runtime! - for checking MOB data? That last one is a
> rhetorical
> > > question (smile), the ops story is... unsatisfying. It's like we've
> > taken a
> > > self sufficient HBase and bolted in parts of Hive, so now we need MR.
> > >
> > > Our internal compaction detects and alerts at warn level if there is a
> > missing link [3], and then returns a empty value [4]
>
>
> Ok, thanks
>
>
> > Mobs are stored in hfiles so we have the same checksumming all other
> hfiles
> > have.
>
>
> Ok, thanks
>
>
> >
> > In the other response, I answered about hbck and how something like
> > Hfile.main() could be a more appropriate checking tool to address this
> > situation.
>
>
> Ok. Replied there.
>
>
> >
> > I'm afraid then much of our complete operational story is "unsatisfying"
>
> even without mob because it still requires MR -- e.g. copytable, export,
> > import, walplayer, or verifyreplicaion mr jobs. While I'll agree that
> > having an external system is undesirable and unacceptable for what are
> > mandatory internal operations like compactions, I think requiring mr for
> a
> > verifiymob mr job would as acceptable as the verfiyreplication job.
>
>
> I think integrity checks are a different class of tool than all others and
> we shouldn't mandate the presence of a MR runtime to execute those. OTOH,
> it's reasonable to provide a standalone tool (if multithreaded) but
> then also a recommended MR version that can achieve better parallelism.
>
>
> >
> > [3]
> >
> >
> https://github.com/apache/hbase/blob/hbase-11339/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HMobStore.java#L400
> > [4]
> >
> >
> https://github.com/apache/hbase/blob/hbase-11339/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobCompactor.java#L224
> >
> > >
> > > On Fri, May 22, 2015 at 1:45 PM, Jonathan Hsieh <jon@cloudera.com
> > <javascript:;>> wrote:
> > >
> > > > In another thread andrew purtell brought up some concerns about the
> mob
> > > > feature:
> > > >
> > > > On Fri, May 22, 2015 at 12:40 PM, Andrew Purtell <
> apurtell@apache.org
> > <javascript:;>>
> > > > wrote:
> > > >
> > > > > Another point of clarification, sorry, I hit the send button too
> > early
> > > it
> > > > > seems: I don't believe MOB is fully integrated yet, for example the
> > > > > feature
> > > > > is an extension to store that lacks support for encryption (this
> > would
> > > > > technically be a feature regression); and HBCK. I have not been
> > > following
> > > > > MOB too closely so could be mistaken. These issues do not preclude
> a
> > > > merge
> > > > > of MOB into trunk, but do preclude a merge back of MOB from trunk
> to
> > > > > branch-1. I would veto the latter until such shortcomings in the
> > > > > implementation that could be described as regressions are
> addressed.
> > I
> > > > > would also like to see a performance analysis of a range of
> workloads
> > > > > before and after in as much detail as can be mustered, and would be
> > > happy
> > > > > to volunteer to help out with that.
> > > > >
> > > >
> > > > Here's info on the points brought up:
> > > >
> > > > Encryption support shortcoming is being addrsessed here:
> > > > https://issues.apache.org/jira/browse/HBASE-13693 (closed)
> > > > https://issues.apache.org/jira/browse/HBASE-13720 (in review)
> > > >
> > > > Hbck has been actually run against the integration test rigs while
> the
> > > > feature has been enabled but currently has no explicit unit test or
> > > simple
> > > > to run integration test. It currently doesn't report anything
> special
> > > > about the mob storage area. We can add unit tests that cover hbck
> when
> > > the
> > > > mob path is exercised.
> > > >
> > > > Another suggestion was a tool to check that mob references had
> > > > corresponding mob data. We currently include a mr-based sweeper job
> > that
> > > > could be used to perform this verification. We can add this tool and
> > > > testing for the tool.
> > > >
> > > > I've done some performance testing and Jingcheng and his colleagues
> > have
> > > > done significant amounts of performance testing. We currently have a
> > blog
> > > > post in progress that will share the results of this performance
> > testing.
> > > >
> > > > Jon.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Wed, May 20, 2015 at 7:38 PM, Ted Yu <yuzhihong@gmail.com
> > <javascript:;>> wrote:
> > > >
> > > > > This is a useful feature, Jon.
> > > > >
> > > > > I went over the mega-patch and left some comments on review board.
> > > > >
> > > > > I noticed that hbck was not included in the patch. Neither did I
> > find a
> > > > > sub-task of HBASE-11339 that covers hbck.
> > > > >
> > > > > Do you or Jingcheng plan to add MOB-aware capability for hbck ?
> > > > >
> > > > > Cheers
> > > > >
> > > > > On Wed, May 20, 2015 at 9:21 AM, Jonathan Hsieh <jon@cloudera.com
> > <javascript:;>>
> > > > wrote:
> > > > >
> > > > > > Hi folks,
> > > > > >
> > > > > > The Medium Object (MOB) Storage feature (HBASE-11339[1]) is
> > modified
> > > > I/O
> > > > > > and compaction path that allows individual moderately sized
> values
> > > > > > (10k-10MB) to be stored so that write amplification is reduced
> when
> > > > > > compared to the normal I/O path. At a high level, it provides
> > > > alternate
> > > > > > flush and compaction mechanisms that segregates large cells into
> a
> > > > > separate
> > > > > > area where they are not subject to potentially frequent
> compaction
> > > and
> > > > > > splits that can be encountered in the normal I/O path. A more
> > > detailed
> > > > > > design doc can be found on the hbase-11339 jira.
> > > > > >
> > > > > > Jingcheng Du has been working on the mob feature for a while and
> > > Anoop,
> > > > > Ram
> > > > > > and I have been shepherding him through the design revisions and
> > > > > > implementation of the feature in the hbase-11339 branch.[2]
> > > > > >
> > > > > > The branch we are proposing to merge into master is compatible
> with
> > > > > HBase's
> > > > > > core functionality including snapshots, replication, shell
> support,
> > > > > behaves
> > > > > > well with table alters, bulk loads and does not require external
> MR
> > > > > > processes. It has been documented, and subject to many
> integration
> > > test
> > > > > > runs (ITBLL, ITAcidGuarantees, ITIngest) including fault
> > injection.
> > > > > > Performance testing of the feature shows what can be a 2x-3x
> > > throughput
> > > > > > improvement for workloads that contain mobs. These results can be
> > > seen
> > > > on
> > > > > > the hbase 2.0 panel discussion slides from hbasecon (once
> > published).
> > > > > >
> > > > > > Recently there have been some hfile encryption related
> shortcomings
> > > > that
> > > > > we
> > > > > > could address in branch or in master.
> > > > > >
> > > > > > Earlier iterations of the feature has been tested in production
> by
> > > > users
> > > > > > that Jingcheng has been responsible for. A version has also been
> > > > > deployed
> > > > > > at users I have been responsible for. Some of the folks from
> > Huawei
> > > > > > (ashutosh) have also been submitting the recent encryption bug
> > > reports
> > > > > > against the hbase-11339 branch so there is some evidence of usage
> > by
> > > > > them.
> > > > > >
> > > > > > The four of us (Jingcheng, Ram, Anoop and I) are satisfied with
> > the
> > > > > > feature and feel it is a good time to call a merge vote. Ive
> > posted
> > > a
> > > > > > megapatch version for folks who want to peruse the code. [3]
> > > > > >
> > > > > > What do you all think?
> > > > > >
> > > > > > Thanks,
> > > > > > Jingcheng, Jon, Ram, and Anoop.
> > > > > >
> > > > > > [1] https://issues.apache.org/jira/browse/HBASE-11339
> > > > > > [2] https://github.com/apache/hbase/tree/hbase-11339
> > > > > > [3] https://reviews.apache.org/r/34475/
> > > > > > --
> > > > > > // Jonathan Hsieh (shay)
> > > > > > // HBase Tech Lead, Software Engineer, Cloudera
> > > > > > // jon@cloudera.com <javascript:;> // @jmhsieh
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > // Jonathan Hsieh (shay)
> > > > // HBase Tech Lead, Software Engineer, Cloudera
> > > > // jon@cloudera.com <javascript:;> // @jmhsieh
> > > >
> > >
> > >
> > >
> > > --
> > > Best regards,
> > >
> > > - Andy
> > >
> > > Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> > > (via Tom White)
> > >
> >
> >
> >
> > --
> > // Jonathan Hsieh (shay)
> > // HBase Tech Lead, Software Engineer, Cloudera
> > // jon@cloudera.com <javascript:;> // @jmhsieh
> >
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
Re: [DISCUSSION] Merge of the hbase-11339 mob branch into master.
Posted by Andrew Purtell <ap...@apache.org>.
Inline
On Wednesday, May 27, 2015, Jonathan Hsieh <jo...@cloudera.com> wrote:
> On Sat, May 23, 2015 at 9:40 AM, Andrew Purtell <apurtell@apache.org
> <javascript:;>> wrote:
>
> > Regarding performance testing: Whatever has been done on the MOB branch
> > will be interesting data points, and, potentially encouraging, but
> porting
> > to branch-1 will produce a new code base. Earlier results on other code
> > will not be applicable. We have to start over. Like I said elsewhere, I'm
> > happy to help with (re)characterizing the perf impact and improvements
> > produced by the changes.
> >
> >
> Thank you for offer for help -- we'd appreciated it!
>
>
You bet.
> Although most of my it tests and perf tests results were done against
> against trunk (from sept '14 and then later feb '15 -- we've been doing
> them roughly every two weeks now) Jingcheng's most recent performance
> testing and fault injection testing results were actually done against a
> version merged/rebased on to hbase 1.0.0[1]. Though not on the most recent
> branch-1, would this be close enough and sufficient or would you still want
> to redoing them?
Closer, yes.
Redo on the branch-1 merge proposal would be important as a confidence
builder still I believe.
>
> If we want to redo them when we have a 1.x backport is ready to propose,
> we'll include the augmented ltt[2] that will make it easy to exercise the
> mob feature's performance.
>
> [1] https://github.com/cloudera/hbase/commits/cdh5-1.0.0_5.4.0?page=2
> (this is cdh5.4.0's hbase 1.0.0-based hbase)
> [2] https://issues.apache.org/jira/browse/HBASE-13277
>
>
> What coverage do we have for verifying the integrity of MOB references?
> > Will the sweep tool detect, alert on, and optionally repair dangling
> > references? (I could answer this for myself by looking at MOB branch, but
> > hopefully someone here has an answer at the ready.) I assume we calculate
> > and store checksums for MOB data itself so we know if values are corrupt.
> > Does the sweep tool detect MOB value corruption? Can it be repaired? Do
> we
> > have a good ops story for why HBCK is no longer sufficient on its own,
> > there's a separate tool with a whole new set of options - and a
> requirement
> > for a MR runtime! - for checking MOB data? That last one is a rhetorical
> > question (smile), the ops story is... unsatisfying. It's like we've
> taken a
> > self sufficient HBase and bolted in parts of Hive, so now we need MR.
> >
> > Our internal compaction detects and alerts at warn level if there is a
> missing link [3], and then returns a empty value [4]
Ok, thanks
> Mobs are stored in hfiles so we have the same checksumming all other hfiles
> have.
Ok, thanks
>
> In the other response, I answered about hbck and how something like
> Hfile.main() could be a more appropriate checking tool to address this
> situation.
Ok. Replied there.
>
> I'm afraid then much of our complete operational story is "unsatisfying"
even without mob because it still requires MR -- e.g. copytable, export,
> import, walplayer, or verifyreplicaion mr jobs. While I'll agree that
> having an external system is undesirable and unacceptable for what are
> mandatory internal operations like compactions, I think requiring mr for a
> verifiymob mr job would as acceptable as the verfiyreplication job.
I think integrity checks are a different class of tool than all others and
we shouldn't mandate the presence of a MR runtime to execute those. OTOH,
it's reasonable to provide a standalone tool (if multithreaded) but
then also a recommended MR version that can achieve better parallelism.
>
> [3]
>
> https://github.com/apache/hbase/blob/hbase-11339/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HMobStore.java#L400
> [4]
>
> https://github.com/apache/hbase/blob/hbase-11339/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobCompactor.java#L224
>
> >
> > On Fri, May 22, 2015 at 1:45 PM, Jonathan Hsieh <jon@cloudera.com
> <javascript:;>> wrote:
> >
> > > In another thread andrew purtell brought up some concerns about the mob
> > > feature:
> > >
> > > On Fri, May 22, 2015 at 12:40 PM, Andrew Purtell <apurtell@apache.org
> <javascript:;>>
> > > wrote:
> > >
> > > > Another point of clarification, sorry, I hit the send button too
> early
> > it
> > > > seems: I don't believe MOB is fully integrated yet, for example the
> > > > feature
> > > > is an extension to store that lacks support for encryption (this
> would
> > > > technically be a feature regression); and HBCK. I have not been
> > following
> > > > MOB too closely so could be mistaken. These issues do not preclude a
> > > merge
> > > > of MOB into trunk, but do preclude a merge back of MOB from trunk to
> > > > branch-1. I would veto the latter until such shortcomings in the
> > > > implementation that could be described as regressions are addressed.
> I
> > > > would also like to see a performance analysis of a range of workloads
> > > > before and after in as much detail as can be mustered, and would be
> > happy
> > > > to volunteer to help out with that.
> > > >
> > >
> > > Here's info on the points brought up:
> > >
> > > Encryption support shortcoming is being addrsessed here:
> > > https://issues.apache.org/jira/browse/HBASE-13693 (closed)
> > > https://issues.apache.org/jira/browse/HBASE-13720 (in review)
> > >
> > > Hbck has been actually run against the integration test rigs while the
> > > feature has been enabled but currently has no explicit unit test or
> > simple
> > > to run integration test. It currently doesn't report anything special
> > > about the mob storage area. We can add unit tests that cover hbck when
> > the
> > > mob path is exercised.
> > >
> > > Another suggestion was a tool to check that mob references had
> > > corresponding mob data. We currently include a mr-based sweeper job
> that
> > > could be used to perform this verification. We can add this tool and
> > > testing for the tool.
> > >
> > > I've done some performance testing and Jingcheng and his colleagues
> have
> > > done significant amounts of performance testing. We currently have a
> blog
> > > post in progress that will share the results of this performance
> testing.
> > >
> > > Jon.
> > >
> > >
> > >
> > >
> > >
> > > On Wed, May 20, 2015 at 7:38 PM, Ted Yu <yuzhihong@gmail.com
> <javascript:;>> wrote:
> > >
> > > > This is a useful feature, Jon.
> > > >
> > > > I went over the mega-patch and left some comments on review board.
> > > >
> > > > I noticed that hbck was not included in the patch. Neither did I
> find a
> > > > sub-task of HBASE-11339 that covers hbck.
> > > >
> > > > Do you or Jingcheng plan to add MOB-aware capability for hbck ?
> > > >
> > > > Cheers
> > > >
> > > > On Wed, May 20, 2015 at 9:21 AM, Jonathan Hsieh <jon@cloudera.com
> <javascript:;>>
> > > wrote:
> > > >
> > > > > Hi folks,
> > > > >
> > > > > The Medium Object (MOB) Storage feature (HBASE-11339[1]) is
> modified
> > > I/O
> > > > > and compaction path that allows individual moderately sized values
> > > > > (10k-10MB) to be stored so that write amplification is reduced when
> > > > > compared to the normal I/O path. At a high level, it provides
> > > alternate
> > > > > flush and compaction mechanisms that segregates large cells into a
> > > > separate
> > > > > area where they are not subject to potentially frequent compaction
> > and
> > > > > splits that can be encountered in the normal I/O path. A more
> > detailed
> > > > > design doc can be found on the hbase-11339 jira.
> > > > >
> > > > > Jingcheng Du has been working on the mob feature for a while and
> > Anoop,
> > > > Ram
> > > > > and I have been shepherding him through the design revisions and
> > > > > implementation of the feature in the hbase-11339 branch.[2]
> > > > >
> > > > > The branch we are proposing to merge into master is compatible with
> > > > HBase's
> > > > > core functionality including snapshots, replication, shell support,
> > > > behaves
> > > > > well with table alters, bulk loads and does not require external MR
> > > > > processes. It has been documented, and subject to many integration
> > test
> > > > > runs (ITBLL, ITAcidGuarantees, ITIngest) including fault
> injection.
> > > > > Performance testing of the feature shows what can be a 2x-3x
> > throughput
> > > > > improvement for workloads that contain mobs. These results can be
> > seen
> > > on
> > > > > the hbase 2.0 panel discussion slides from hbasecon (once
> published).
> > > > >
> > > > > Recently there have been some hfile encryption related shortcomings
> > > that
> > > > we
> > > > > could address in branch or in master.
> > > > >
> > > > > Earlier iterations of the feature has been tested in production by
> > > users
> > > > > that Jingcheng has been responsible for. A version has also been
> > > > deployed
> > > > > at users I have been responsible for. Some of the folks from
> Huawei
> > > > > (ashutosh) have also been submitting the recent encryption bug
> > reports
> > > > > against the hbase-11339 branch so there is some evidence of usage
> by
> > > > them.
> > > > >
> > > > > The four of us (Jingcheng, Ram, Anoop and I) are satisfied with
> the
> > > > > feature and feel it is a good time to call a merge vote. Ive
> posted
> > a
> > > > > megapatch version for folks who want to peruse the code. [3]
> > > > >
> > > > > What do you all think?
> > > > >
> > > > > Thanks,
> > > > > Jingcheng, Jon, Ram, and Anoop.
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/HBASE-11339
> > > > > [2] https://github.com/apache/hbase/tree/hbase-11339
> > > > > [3] https://reviews.apache.org/r/34475/
> > > > > --
> > > > > // Jonathan Hsieh (shay)
> > > > > // HBase Tech Lead, Software Engineer, Cloudera
> > > > > // jon@cloudera.com <javascript:;> // @jmhsieh
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > // Jonathan Hsieh (shay)
> > > // HBase Tech Lead, Software Engineer, Cloudera
> > > // jon@cloudera.com <javascript:;> // @jmhsieh
> > >
> >
> >
> >
> > --
> > Best regards,
> >
> > - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>
>
>
> --
> // Jonathan Hsieh (shay)
> // HBase Tech Lead, Software Engineer, Cloudera
> // jon@cloudera.com <javascript:;> // @jmhsieh
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: [DISCUSSION] Merge of the hbase-11339 mob branch into master.
Posted by Jonathan Hsieh <jo...@cloudera.com>.
On Sat, May 23, 2015 at 9:40 AM, Andrew Purtell <ap...@apache.org> wrote:
> Regarding performance testing: Whatever has been done on the MOB branch
> will be interesting data points, and, potentially encouraging, but porting
> to branch-1 will produce a new code base. Earlier results on other code
> will not be applicable. We have to start over. Like I said elsewhere, I'm
> happy to help with (re)characterizing the perf impact and improvements
> produced by the changes.
>
>
Thank you for offer for help -- we'd appreciated it!
Although most of my it tests and perf tests results were done against
against trunk (from sept '14 and then later feb '15 -- we've been doing
them roughly every two weeks now) Jingcheng's most recent performance
testing and fault injection testing results were actually done against a
version merged/rebased on to hbase 1.0.0[1]. Though not on the most recent
branch-1, would this be close enough and sufficient or would you still want
to redoing them?
If we want to redo them when we have a 1.x backport is ready to propose,
we'll include the augmented ltt[2] that will make it easy to exercise the
mob feature's performance.
[1] https://github.com/cloudera/hbase/commits/cdh5-1.0.0_5.4.0?page=2
(this is cdh5.4.0's hbase 1.0.0-based hbase)
[2] https://issues.apache.org/jira/browse/HBASE-13277
What coverage do we have for verifying the integrity of MOB references?
> Will the sweep tool detect, alert on, and optionally repair dangling
> references? (I could answer this for myself by looking at MOB branch, but
> hopefully someone here has an answer at the ready.) I assume we calculate
> and store checksums for MOB data itself so we know if values are corrupt.
> Does the sweep tool detect MOB value corruption? Can it be repaired? Do we
> have a good ops story for why HBCK is no longer sufficient on its own,
> there's a separate tool with a whole new set of options - and a requirement
> for a MR runtime! - for checking MOB data? That last one is a rhetorical
> question (smile), the ops story is... unsatisfying. It's like we've taken a
> self sufficient HBase and bolted in parts of Hive, so now we need MR.
>
> Our internal compaction detects and alerts at warn level if there is a
missing link [3], and then returns a empty value [4]
Mobs are stored in hfiles so we have the same checksumming all other hfiles
have.
In the other response, I answered about hbck and how something like
Hfile.main() could be a more appropriate checking tool to address this
situation.
I'm afraid then much of our complete operational story is "unsatisfying"
even without mob because it still requires MR -- e.g. copytable, export,
import, walplayer, or verifyreplicaion mr jobs. While I'll agree that
having an external system is undesirable and unacceptable for what are
mandatory internal operations like compactions, I think requiring mr for a
verifiymob mr job would as acceptable as the verfiyreplication job.
[3]
https://github.com/apache/hbase/blob/hbase-11339/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HMobStore.java#L400
[4]
https://github.com/apache/hbase/blob/hbase-11339/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobCompactor.java#L224
>
> On Fri, May 22, 2015 at 1:45 PM, Jonathan Hsieh <jo...@cloudera.com> wrote:
>
> > In another thread andrew purtell brought up some concerns about the mob
> > feature:
> >
> > On Fri, May 22, 2015 at 12:40 PM, Andrew Purtell <ap...@apache.org>
> > wrote:
> >
> > > Another point of clarification, sorry, I hit the send button too early
> it
> > > seems: I don't believe MOB is fully integrated yet, for example the
> > > feature
> > > is an extension to store that lacks support for encryption (this would
> > > technically be a feature regression); and HBCK. I have not been
> following
> > > MOB too closely so could be mistaken. These issues do not preclude a
> > merge
> > > of MOB into trunk, but do preclude a merge back of MOB from trunk to
> > > branch-1. I would veto the latter until such shortcomings in the
> > > implementation that could be described as regressions are addressed. I
> > > would also like to see a performance analysis of a range of workloads
> > > before and after in as much detail as can be mustered, and would be
> happy
> > > to volunteer to help out with that.
> > >
> >
> > Here's info on the points brought up:
> >
> > Encryption support shortcoming is being addrsessed here:
> > https://issues.apache.org/jira/browse/HBASE-13693 (closed)
> > https://issues.apache.org/jira/browse/HBASE-13720 (in review)
> >
> > Hbck has been actually run against the integration test rigs while the
> > feature has been enabled but currently has no explicit unit test or
> simple
> > to run integration test. It currently doesn't report anything special
> > about the mob storage area. We can add unit tests that cover hbck when
> the
> > mob path is exercised.
> >
> > Another suggestion was a tool to check that mob references had
> > corresponding mob data. We currently include a mr-based sweeper job that
> > could be used to perform this verification. We can add this tool and
> > testing for the tool.
> >
> > I've done some performance testing and Jingcheng and his colleagues have
> > done significant amounts of performance testing. We currently have a blog
> > post in progress that will share the results of this performance testing.
> >
> > Jon.
> >
> >
> >
> >
> >
> > On Wed, May 20, 2015 at 7:38 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > This is a useful feature, Jon.
> > >
> > > I went over the mega-patch and left some comments on review board.
> > >
> > > I noticed that hbck was not included in the patch. Neither did I find a
> > > sub-task of HBASE-11339 that covers hbck.
> > >
> > > Do you or Jingcheng plan to add MOB-aware capability for hbck ?
> > >
> > > Cheers
> > >
> > > On Wed, May 20, 2015 at 9:21 AM, Jonathan Hsieh <jo...@cloudera.com>
> > wrote:
> > >
> > > > Hi folks,
> > > >
> > > > The Medium Object (MOB) Storage feature (HBASE-11339[1]) is modified
> > I/O
> > > > and compaction path that allows individual moderately sized values
> > > > (10k-10MB) to be stored so that write amplification is reduced when
> > > > compared to the normal I/O path. At a high level, it provides
> > alternate
> > > > flush and compaction mechanisms that segregates large cells into a
> > > separate
> > > > area where they are not subject to potentially frequent compaction
> and
> > > > splits that can be encountered in the normal I/O path. A more
> detailed
> > > > design doc can be found on the hbase-11339 jira.
> > > >
> > > > Jingcheng Du has been working on the mob feature for a while and
> Anoop,
> > > Ram
> > > > and I have been shepherding him through the design revisions and
> > > > implementation of the feature in the hbase-11339 branch.[2]
> > > >
> > > > The branch we are proposing to merge into master is compatible with
> > > HBase's
> > > > core functionality including snapshots, replication, shell support,
> > > behaves
> > > > well with table alters, bulk loads and does not require external MR
> > > > processes. It has been documented, and subject to many integration
> test
> > > > runs (ITBLL, ITAcidGuarantees, ITIngest) including fault injection.
> > > > Performance testing of the feature shows what can be a 2x-3x
> throughput
> > > > improvement for workloads that contain mobs. These results can be
> seen
> > on
> > > > the hbase 2.0 panel discussion slides from hbasecon (once published).
> > > >
> > > > Recently there have been some hfile encryption related shortcomings
> > that
> > > we
> > > > could address in branch or in master.
> > > >
> > > > Earlier iterations of the feature has been tested in production by
> > users
> > > > that Jingcheng has been responsible for. A version has also been
> > > deployed
> > > > at users I have been responsible for. Some of the folks from Huawei
> > > > (ashutosh) have also been submitting the recent encryption bug
> reports
> > > > against the hbase-11339 branch so there is some evidence of usage by
> > > them.
> > > >
> > > > The four of us (Jingcheng, Ram, Anoop and I) are satisfied with the
> > > > feature and feel it is a good time to call a merge vote. Ive posted
> a
> > > > megapatch version for folks who want to peruse the code. [3]
> > > >
> > > > What do you all think?
> > > >
> > > > Thanks,
> > > > Jingcheng, Jon, Ram, and Anoop.
> > > >
> > > > [1] https://issues.apache.org/jira/browse/HBASE-11339
> > > > [2] https://github.com/apache/hbase/tree/hbase-11339
> > > > [3] https://reviews.apache.org/r/34475/
> > > > --
> > > > // Jonathan Hsieh (shay)
> > > > // HBase Tech Lead, Software Engineer, Cloudera
> > > > // jon@cloudera.com // @jmhsieh
> > > >
> > >
> >
> >
> >
> > --
> > // Jonathan Hsieh (shay)
> > // HBase Tech Lead, Software Engineer, Cloudera
> > // jon@cloudera.com // @jmhsieh
> >
>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
--
// Jonathan Hsieh (shay)
// HBase Tech Lead, Software Engineer, Cloudera
// jon@cloudera.com // @jmhsieh
Re: [DISCUSSION] Merge of the hbase-11339 mob branch into master.
Posted by Jingcheng Du <ji...@intel.com>.
Andrew Purtell wrote
> What coverage do we have for verifying the integrity of MOB references?
> Will the sweep tool detect, alert on, and optionally repair dangling
> references? (I could answer this for myself by looking at MOB branch, but
> hopefully someone here has an answer at the ready.) I assume we calculate
> and store checksums for MOB data itself so we know if values are corrupt.
> Does the sweep tool detect MOB value corruption? Can it be repaired? Do we
> have a good ops story for why HBCK is no longer sufficient on its own,
> there's a separate tool with a whole new set of options - and a
> requirement
> for a MR runtime! - for checking MOB data? That last one is a rhetorical
> question (smile), the ops story is... unsatisfying. It's like we've taken
> a
> self sufficient HBase and bolted in parts of Hive, so now we need MR.
Now we have two mechanism to compact mob file (Merge the small files and
drop the deleted cells), one is the sweep tool (with MR), the other is
MobCompactor (without MR).
Both of them do not drop the dangling reference cells (if we have although I
think it hardly happens).
I think the code to detect the missing and corrupt can be added in HBCK and
HFile.main.
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/DISCUSSION-Merge-of-the-hbase-11339-mob-branch-into-master-tp4071644p4071912.html
Sent from the HBase Developer mailing list archive at Nabble.com.
Re: [DISCUSSION] Merge of the hbase-11339 mob branch into master.
Posted by Jingcheng Du <ji...@intel.com>.
Thanks Ted. The VOTE mail was posted a while ago.
Regards
Jingcheng
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/DISCUSSION-Merge-of-the-hbase-11339-mob-branch-into-master-tp4071644p4073309.html
Sent from the HBase Developer mailing list archive at Nabble.com.
Re: [DISCUSSION] Merge of the hbase-11339 mob branch into master.
Posted by Ted Yu <yu...@gmail.com>.
Jingcheng:
Let us know if you need anything for composition of the vote thread.
On Tue, Jul 14, 2015 at 3:58 PM, Andrew Purtell <ap...@apache.org> wrote:
> That's great, and I also appreciate the time spent discussing on this
> thread. I plan to vote yes.
>
> On Tue, Jul 14, 2015 at 3:57 PM, Jonathan Hsieh <jo...@cloudera.com> wrote:
>
> > Since Jingcheng did the majority of the work, I'd like him to stat the
> > thread vote thread. It should happen in the next few days.
> >
> > Jon
> >
> > On Tue, Jul 14, 2015 at 3:53 PM, Andrew Purtell <ap...@apache.org>
> > wrote:
> >
> > > Start a vote thread? This says DISCUSSION
> > >
> > > On Tue, Jul 14, 2015 at 2:05 PM, Ted Yu <yu...@gmail.com> wrote:
> > >
> > > > There have been several iterations since the first mega patch was
> > posted.
> > > >
> > > > Currently QA run is green.
> > > >
> > > > IntegrationTestIngestWithMOB has been run which passes.
> > > >
> > > > I want to give +1 for merging to master branch.
> > > >
> > > > On Tue, Jul 7, 2015 at 9:37 PM, Anoop John <an...@gmail.com>
> > > wrote:
> > > >
> > > > > +1
> > > > >
> > > > > We will work on making it a Vote thread.
> > > > >
> > > > > -Anoop-
> > > > >
> > > > > On Wed, Jul 8, 2015 at 9:46 AM, ramkrishna vasudevan <
> > > > > ramkrishna.s.vasudevan@gmail.com> wrote:
> > > > >
> > > > > > +1 to start the voting again.
> > > > > >
> > > > > > On Wed, Jul 8, 2015 at 9:07 AM, Ted Yu <yu...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > Looks like all MOB-related JIRAs have been resolved.
> > > > > > >
> > > > > > > Should the voting process be resumed ?
> > > > > > >
> > > > > > > Cheers
> > > > > > >
> > > > > > > On Thu, May 28, 2015 at 5:48 PM, Anoop John <
> > anoop.hbase@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Yes Andy. The sweep tool is completely optional now. We have
> a
> > > > chore
> > > > > > > doing
> > > > > > > > the compaction, like we trigger auto major compaction. We
> can
> > > > > > configure
> > > > > > > > the interval. Auto can be turned off and user can explicitly
> > call
> > > > > also.
> > > > > > > We
> > > > > > > > have shell and API support.
> > > > > > > >
> > > > > > > > Anoop
> > > > > > > >
> > > > > > > > On Friday, May 29, 2015, Andrew Purtell <apurtell@apache.org
> >
> > > > wrote:
> > > > > > > > > MOB references in cells won't find their value if the MOB
> > hfile
> > > > has
> > > > > > > been
> > > > > > > > > corrupted. Dealing with that would be like any other
> > corrupted
> > > > > HFile,
> > > > > > > > > understood. The dangling references make thinking about
> > > (partial)
> > > > > > > > recovery
> > > > > > > > > and repair interesting.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thursday, May 28, 2015, Jingcheng Du <
> > > jingcheng.du@intel.com>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Andrew Purtell wrote
> > > > > > > > >> > HBCK can check and sideline dangling reference files. I
> > > think
> > > > of
> > > > > > MOB
> > > > > > > > >> files
> > > > > > > > >> > as "core enough" auxiliary files that need some
> support. I
> > > > > suppose
> > > > > > > > unlike
> > > > > > > > >> > reference files their presence or absence won't produce
> a
> > > > region
> > > > > > > open
> > > > > > > > >> > failure, we would see dangling pointers later when tying
> > to
> > > > > > service
> > > > > > > > >> > queries. (Yes?) Will that abort the RS? Pardon the
> > ignorant
> > > > > > > question,
> > > > > > > > >> > normally I could check the code but I'm at the airport
> on
> > a
> > > > > phone.
> > > > > > > > >>
> > > > > > > > >> Thanks for comments!
> > > > > > > > >> In mob, usually the reference cells are committed after
> the
> > > mob
> > > > > > files
> > > > > > > > are
> > > > > > > > >> done. I think it hardly happens that a reference cell
> cannot
> > > > find
> > > > > > its
> > > > > > > > mob
> > > > > > > > >> file.
> > > > > > > > >> Even if there's a dangling reference cell, the RS won't be
> > > > > aborted,
> > > > > > a
> > > > > > > > empty
> > > > > > > > >> cell is returned instead.
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> Andrew Purtell wrote
> > > > > > > > >> > On that subject, I should file follow up issues for more
> > > check
> > > > > and
> > > > > > > > repair
> > > > > > > > >> > options for HFiles. We should be able to detect missing
> or
> > > > > corrupt
> > > > > > > > files
> > > > > > > > >> > of
> > > > > > > > >> > all variety: HFile, reference, MOB. This may require an
> > > > > expensive
> > > > > > > scan
> > > > > > > > >> > over
> > > > > > > > >> > lots of files, but this is like fsck full disk surface
> > scans
> > > > and
> > > > > > > those
> > > > > > > > >> > have
> > > > > > > > >> > similar costs. Providing MR based tools is fine but we
> > > should
> > > > > have
> > > > > > > > >> > multithreaded tools that can stand in if a MR runtime is
> > not
> > > > > > > > available.
> > > > > > > > >> > Import, Export, VerifyReplication...all of these tools
> are
> > > in
> > > > a
> > > > > > > > >> different,
> > > > > > > > >> > lesser, class than integrity and repair tools, in my
> > > opinion.
> > > > > > Since
> > > > > > > > MOB
> > > > > > > > >> > will likely be merged into trunk by then I'll be sure to
> > > > include
> > > > > > > it. I
> > > > > > > > >> > agree it's not fair to ask more of MOB then what we have
> > now
> > > > for
> > > > > > > > HFile.
> > > > > > > > >>
> > > > > > > > >> To detect the corrupt files, some code are needed in the
> > file
> > > > > > checker
> > > > > > > to
> > > > > > > > >> check mob files after knowing it's a mob-enabled column.
> > > > > > > > >> To detect the missing or dangling reference cells, I think
> > we
> > > > have
> > > > > > to
> > > > > > > do
> > > > > > > > a
> > > > > > > > >> full-table scan, like what is done now in HFile.main.
> > > > > > > > >> We can do that.
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> --
> > > > > > > > >> View this message in context:
> > > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://apache-hbase.679495.n3.nabble.com/DISCUSSION-Merge-of-the-hbase-11339-mob-branch-into-master-tp4071644p4071911.html
> > > > > > > > >> Sent from the HBase Developer mailing list archive at
> > > > Nabble.com.
> > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Best regards,
> > > > > > > > >
> > > > > > > > > - Andy
> > > > > > > > >
> > > > > > > > > Problems worthy of attack prove their worth by hitting
> back.
> > -
> > > > Piet
> > > > > > > Hein
> > > > > > > > > (via Tom White)
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Best regards,
> > >
> > > - Andy
> > >
> > > Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> > > (via Tom White)
> > >
> >
> >
> >
> > --
> > // Jonathan Hsieh (shay)
> > // HBase Tech Lead, Software Engineer, Cloudera
> > // jon@cloudera.com // @jmhsieh
> >
>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
Re: [DISCUSSION] Merge of the hbase-11339 mob branch into master.
Posted by Andrew Purtell <ap...@apache.org>.
That's great, and I also appreciate the time spent discussing on this
thread. I plan to vote yes.
On Tue, Jul 14, 2015 at 3:57 PM, Jonathan Hsieh <jo...@cloudera.com> wrote:
> Since Jingcheng did the majority of the work, I'd like him to stat the
> thread vote thread. It should happen in the next few days.
>
> Jon
>
> On Tue, Jul 14, 2015 at 3:53 PM, Andrew Purtell <ap...@apache.org>
> wrote:
>
> > Start a vote thread? This says DISCUSSION
> >
> > On Tue, Jul 14, 2015 at 2:05 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > There have been several iterations since the first mega patch was
> posted.
> > >
> > > Currently QA run is green.
> > >
> > > IntegrationTestIngestWithMOB has been run which passes.
> > >
> > > I want to give +1 for merging to master branch.
> > >
> > > On Tue, Jul 7, 2015 at 9:37 PM, Anoop John <an...@gmail.com>
> > wrote:
> > >
> > > > +1
> > > >
> > > > We will work on making it a Vote thread.
> > > >
> > > > -Anoop-
> > > >
> > > > On Wed, Jul 8, 2015 at 9:46 AM, ramkrishna vasudevan <
> > > > ramkrishna.s.vasudevan@gmail.com> wrote:
> > > >
> > > > > +1 to start the voting again.
> > > > >
> > > > > On Wed, Jul 8, 2015 at 9:07 AM, Ted Yu <yu...@gmail.com>
> wrote:
> > > > >
> > > > > > Looks like all MOB-related JIRAs have been resolved.
> > > > > >
> > > > > > Should the voting process be resumed ?
> > > > > >
> > > > > > Cheers
> > > > > >
> > > > > > On Thu, May 28, 2015 at 5:48 PM, Anoop John <
> anoop.hbase@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Yes Andy. The sweep tool is completely optional now. We have a
> > > chore
> > > > > > doing
> > > > > > > the compaction, like we trigger auto major compaction. We can
> > > > > configure
> > > > > > > the interval. Auto can be turned off and user can explicitly
> call
> > > > also.
> > > > > > We
> > > > > > > have shell and API support.
> > > > > > >
> > > > > > > Anoop
> > > > > > >
> > > > > > > On Friday, May 29, 2015, Andrew Purtell <ap...@apache.org>
> > > wrote:
> > > > > > > > MOB references in cells won't find their value if the MOB
> hfile
> > > has
> > > > > > been
> > > > > > > > corrupted. Dealing with that would be like any other
> corrupted
> > > > HFile,
> > > > > > > > understood. The dangling references make thinking about
> > (partial)
> > > > > > > recovery
> > > > > > > > and repair interesting.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thursday, May 28, 2015, Jingcheng Du <
> > jingcheng.du@intel.com>
> > > > > > wrote:
> > > > > > > >
> > > > > > > >> Andrew Purtell wrote
> > > > > > > >> > HBCK can check and sideline dangling reference files. I
> > think
> > > of
> > > > > MOB
> > > > > > > >> files
> > > > > > > >> > as "core enough" auxiliary files that need some support. I
> > > > suppose
> > > > > > > unlike
> > > > > > > >> > reference files their presence or absence won't produce a
> > > region
> > > > > > open
> > > > > > > >> > failure, we would see dangling pointers later when tying
> to
> > > > > service
> > > > > > > >> > queries. (Yes?) Will that abort the RS? Pardon the
> ignorant
> > > > > > question,
> > > > > > > >> > normally I could check the code but I'm at the airport on
> a
> > > > phone.
> > > > > > > >>
> > > > > > > >> Thanks for comments!
> > > > > > > >> In mob, usually the reference cells are committed after the
> > mob
> > > > > files
> > > > > > > are
> > > > > > > >> done. I think it hardly happens that a reference cell cannot
> > > find
> > > > > its
> > > > > > > mob
> > > > > > > >> file.
> > > > > > > >> Even if there's a dangling reference cell, the RS won't be
> > > > aborted,
> > > > > a
> > > > > > > empty
> > > > > > > >> cell is returned instead.
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> Andrew Purtell wrote
> > > > > > > >> > On that subject, I should file follow up issues for more
> > check
> > > > and
> > > > > > > repair
> > > > > > > >> > options for HFiles. We should be able to detect missing or
> > > > corrupt
> > > > > > > files
> > > > > > > >> > of
> > > > > > > >> > all variety: HFile, reference, MOB. This may require an
> > > > expensive
> > > > > > scan
> > > > > > > >> > over
> > > > > > > >> > lots of files, but this is like fsck full disk surface
> scans
> > > and
> > > > > > those
> > > > > > > >> > have
> > > > > > > >> > similar costs. Providing MR based tools is fine but we
> > should
> > > > have
> > > > > > > >> > multithreaded tools that can stand in if a MR runtime is
> not
> > > > > > > available.
> > > > > > > >> > Import, Export, VerifyReplication...all of these tools are
> > in
> > > a
> > > > > > > >> different,
> > > > > > > >> > lesser, class than integrity and repair tools, in my
> > opinion.
> > > > > Since
> > > > > > > MOB
> > > > > > > >> > will likely be merged into trunk by then I'll be sure to
> > > include
> > > > > > it. I
> > > > > > > >> > agree it's not fair to ask more of MOB then what we have
> now
> > > for
> > > > > > > HFile.
> > > > > > > >>
> > > > > > > >> To detect the corrupt files, some code are needed in the
> file
> > > > > checker
> > > > > > to
> > > > > > > >> check mob files after knowing it's a mob-enabled column.
> > > > > > > >> To detect the missing or dangling reference cells, I think
> we
> > > have
> > > > > to
> > > > > > do
> > > > > > > a
> > > > > > > >> full-table scan, like what is done now in HFile.main.
> > > > > > > >> We can do that.
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> --
> > > > > > > >> View this message in context:
> > > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://apache-hbase.679495.n3.nabble.com/DISCUSSION-Merge-of-the-hbase-11339-mob-branch-into-master-tp4071644p4071911.html
> > > > > > > >> Sent from the HBase Developer mailing list archive at
> > > Nabble.com.
> > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Best regards,
> > > > > > > >
> > > > > > > > - Andy
> > > > > > > >
> > > > > > > > Problems worthy of attack prove their worth by hitting back.
> -
> > > Piet
> > > > > > Hein
> > > > > > > > (via Tom White)
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Best regards,
> >
> > - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>
>
>
> --
> // Jonathan Hsieh (shay)
> // HBase Tech Lead, Software Engineer, Cloudera
> // jon@cloudera.com // @jmhsieh
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: [DISCUSSION] Merge of the hbase-11339 mob branch into master.
Posted by Jonathan Hsieh <jo...@cloudera.com>.
Since Jingcheng did the majority of the work, I'd like him to stat the
thread vote thread. It should happen in the next few days.
Jon
On Tue, Jul 14, 2015 at 3:53 PM, Andrew Purtell <ap...@apache.org> wrote:
> Start a vote thread? This says DISCUSSION
>
> On Tue, Jul 14, 2015 at 2:05 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > There have been several iterations since the first mega patch was posted.
> >
> > Currently QA run is green.
> >
> > IntegrationTestIngestWithMOB has been run which passes.
> >
> > I want to give +1 for merging to master branch.
> >
> > On Tue, Jul 7, 2015 at 9:37 PM, Anoop John <an...@gmail.com>
> wrote:
> >
> > > +1
> > >
> > > We will work on making it a Vote thread.
> > >
> > > -Anoop-
> > >
> > > On Wed, Jul 8, 2015 at 9:46 AM, ramkrishna vasudevan <
> > > ramkrishna.s.vasudevan@gmail.com> wrote:
> > >
> > > > +1 to start the voting again.
> > > >
> > > > On Wed, Jul 8, 2015 at 9:07 AM, Ted Yu <yu...@gmail.com> wrote:
> > > >
> > > > > Looks like all MOB-related JIRAs have been resolved.
> > > > >
> > > > > Should the voting process be resumed ?
> > > > >
> > > > > Cheers
> > > > >
> > > > > On Thu, May 28, 2015 at 5:48 PM, Anoop John <anoop.hbase@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > > Yes Andy. The sweep tool is completely optional now. We have a
> > chore
> > > > > doing
> > > > > > the compaction, like we trigger auto major compaction. We can
> > > > configure
> > > > > > the interval. Auto can be turned off and user can explicitly call
> > > also.
> > > > > We
> > > > > > have shell and API support.
> > > > > >
> > > > > > Anoop
> > > > > >
> > > > > > On Friday, May 29, 2015, Andrew Purtell <ap...@apache.org>
> > wrote:
> > > > > > > MOB references in cells won't find their value if the MOB hfile
> > has
> > > > > been
> > > > > > > corrupted. Dealing with that would be like any other corrupted
> > > HFile,
> > > > > > > understood. The dangling references make thinking about
> (partial)
> > > > > > recovery
> > > > > > > and repair interesting.
> > > > > > >
> > > > > > >
> > > > > > > On Thursday, May 28, 2015, Jingcheng Du <
> jingcheng.du@intel.com>
> > > > > wrote:
> > > > > > >
> > > > > > >> Andrew Purtell wrote
> > > > > > >> > HBCK can check and sideline dangling reference files. I
> think
> > of
> > > > MOB
> > > > > > >> files
> > > > > > >> > as "core enough" auxiliary files that need some support. I
> > > suppose
> > > > > > unlike
> > > > > > >> > reference files their presence or absence won't produce a
> > region
> > > > > open
> > > > > > >> > failure, we would see dangling pointers later when tying to
> > > > service
> > > > > > >> > queries. (Yes?) Will that abort the RS? Pardon the ignorant
> > > > > question,
> > > > > > >> > normally I could check the code but I'm at the airport on a
> > > phone.
> > > > > > >>
> > > > > > >> Thanks for comments!
> > > > > > >> In mob, usually the reference cells are committed after the
> mob
> > > > files
> > > > > > are
> > > > > > >> done. I think it hardly happens that a reference cell cannot
> > find
> > > > its
> > > > > > mob
> > > > > > >> file.
> > > > > > >> Even if there's a dangling reference cell, the RS won't be
> > > aborted,
> > > > a
> > > > > > empty
> > > > > > >> cell is returned instead.
> > > > > > >>
> > > > > > >>
> > > > > > >> Andrew Purtell wrote
> > > > > > >> > On that subject, I should file follow up issues for more
> check
> > > and
> > > > > > repair
> > > > > > >> > options for HFiles. We should be able to detect missing or
> > > corrupt
> > > > > > files
> > > > > > >> > of
> > > > > > >> > all variety: HFile, reference, MOB. This may require an
> > > expensive
> > > > > scan
> > > > > > >> > over
> > > > > > >> > lots of files, but this is like fsck full disk surface scans
> > and
> > > > > those
> > > > > > >> > have
> > > > > > >> > similar costs. Providing MR based tools is fine but we
> should
> > > have
> > > > > > >> > multithreaded tools that can stand in if a MR runtime is not
> > > > > > available.
> > > > > > >> > Import, Export, VerifyReplication...all of these tools are
> in
> > a
> > > > > > >> different,
> > > > > > >> > lesser, class than integrity and repair tools, in my
> opinion.
> > > > Since
> > > > > > MOB
> > > > > > >> > will likely be merged into trunk by then I'll be sure to
> > include
> > > > > it. I
> > > > > > >> > agree it's not fair to ask more of MOB then what we have now
> > for
> > > > > > HFile.
> > > > > > >>
> > > > > > >> To detect the corrupt files, some code are needed in the file
> > > > checker
> > > > > to
> > > > > > >> check mob files after knowing it's a mob-enabled column.
> > > > > > >> To detect the missing or dangling reference cells, I think we
> > have
> > > > to
> > > > > do
> > > > > > a
> > > > > > >> full-table scan, like what is done now in HFile.main.
> > > > > > >> We can do that.
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> --
> > > > > > >> View this message in context:
> > > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://apache-hbase.679495.n3.nabble.com/DISCUSSION-Merge-of-the-hbase-11339-mob-branch-into-master-tp4071644p4071911.html
> > > > > > >> Sent from the HBase Developer mailing list archive at
> > Nabble.com.
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Best regards,
> > > > > > >
> > > > > > > - Andy
> > > > > > >
> > > > > > > Problems worthy of attack prove their worth by hitting back. -
> > Piet
> > > > > Hein
> > > > > > > (via Tom White)
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
--
// Jonathan Hsieh (shay)
// HBase Tech Lead, Software Engineer, Cloudera
// jon@cloudera.com // @jmhsieh
Re: [DISCUSSION] Merge of the hbase-11339 mob branch into master.
Posted by Andrew Purtell <ap...@apache.org>.
Start a vote thread? This says DISCUSSION
On Tue, Jul 14, 2015 at 2:05 PM, Ted Yu <yu...@gmail.com> wrote:
> There have been several iterations since the first mega patch was posted.
>
> Currently QA run is green.
>
> IntegrationTestIngestWithMOB has been run which passes.
>
> I want to give +1 for merging to master branch.
>
> On Tue, Jul 7, 2015 at 9:37 PM, Anoop John <an...@gmail.com> wrote:
>
> > +1
> >
> > We will work on making it a Vote thread.
> >
> > -Anoop-
> >
> > On Wed, Jul 8, 2015 at 9:46 AM, ramkrishna vasudevan <
> > ramkrishna.s.vasudevan@gmail.com> wrote:
> >
> > > +1 to start the voting again.
> > >
> > > On Wed, Jul 8, 2015 at 9:07 AM, Ted Yu <yu...@gmail.com> wrote:
> > >
> > > > Looks like all MOB-related JIRAs have been resolved.
> > > >
> > > > Should the voting process be resumed ?
> > > >
> > > > Cheers
> > > >
> > > > On Thu, May 28, 2015 at 5:48 PM, Anoop John <an...@gmail.com>
> > > wrote:
> > > >
> > > > > Yes Andy. The sweep tool is completely optional now. We have a
> chore
> > > > doing
> > > > > the compaction, like we trigger auto major compaction. We can
> > > configure
> > > > > the interval. Auto can be turned off and user can explicitly call
> > also.
> > > > We
> > > > > have shell and API support.
> > > > >
> > > > > Anoop
> > > > >
> > > > > On Friday, May 29, 2015, Andrew Purtell <ap...@apache.org>
> wrote:
> > > > > > MOB references in cells won't find their value if the MOB hfile
> has
> > > > been
> > > > > > corrupted. Dealing with that would be like any other corrupted
> > HFile,
> > > > > > understood. The dangling references make thinking about (partial)
> > > > > recovery
> > > > > > and repair interesting.
> > > > > >
> > > > > >
> > > > > > On Thursday, May 28, 2015, Jingcheng Du <ji...@intel.com>
> > > > wrote:
> > > > > >
> > > > > >> Andrew Purtell wrote
> > > > > >> > HBCK can check and sideline dangling reference files. I think
> of
> > > MOB
> > > > > >> files
> > > > > >> > as "core enough" auxiliary files that need some support. I
> > suppose
> > > > > unlike
> > > > > >> > reference files their presence or absence won't produce a
> region
> > > > open
> > > > > >> > failure, we would see dangling pointers later when tying to
> > > service
> > > > > >> > queries. (Yes?) Will that abort the RS? Pardon the ignorant
> > > > question,
> > > > > >> > normally I could check the code but I'm at the airport on a
> > phone.
> > > > > >>
> > > > > >> Thanks for comments!
> > > > > >> In mob, usually the reference cells are committed after the mob
> > > files
> > > > > are
> > > > > >> done. I think it hardly happens that a reference cell cannot
> find
> > > its
> > > > > mob
> > > > > >> file.
> > > > > >> Even if there's a dangling reference cell, the RS won't be
> > aborted,
> > > a
> > > > > empty
> > > > > >> cell is returned instead.
> > > > > >>
> > > > > >>
> > > > > >> Andrew Purtell wrote
> > > > > >> > On that subject, I should file follow up issues for more check
> > and
> > > > > repair
> > > > > >> > options for HFiles. We should be able to detect missing or
> > corrupt
> > > > > files
> > > > > >> > of
> > > > > >> > all variety: HFile, reference, MOB. This may require an
> > expensive
> > > > scan
> > > > > >> > over
> > > > > >> > lots of files, but this is like fsck full disk surface scans
> and
> > > > those
> > > > > >> > have
> > > > > >> > similar costs. Providing MR based tools is fine but we should
> > have
> > > > > >> > multithreaded tools that can stand in if a MR runtime is not
> > > > > available.
> > > > > >> > Import, Export, VerifyReplication...all of these tools are in
> a
> > > > > >> different,
> > > > > >> > lesser, class than integrity and repair tools, in my opinion.
> > > Since
> > > > > MOB
> > > > > >> > will likely be merged into trunk by then I'll be sure to
> include
> > > > it. I
> > > > > >> > agree it's not fair to ask more of MOB then what we have now
> for
> > > > > HFile.
> > > > > >>
> > > > > >> To detect the corrupt files, some code are needed in the file
> > > checker
> > > > to
> > > > > >> check mob files after knowing it's a mob-enabled column.
> > > > > >> To detect the missing or dangling reference cells, I think we
> have
> > > to
> > > > do
> > > > > a
> > > > > >> full-table scan, like what is done now in HFile.main.
> > > > > >> We can do that.
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >> View this message in context:
> > > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
> http://apache-hbase.679495.n3.nabble.com/DISCUSSION-Merge-of-the-hbase-11339-mob-branch-into-master-tp4071644p4071911.html
> > > > > >> Sent from the HBase Developer mailing list archive at
> Nabble.com.
> > > > > >>
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Best regards,
> > > > > >
> > > > > > - Andy
> > > > > >
> > > > > > Problems worthy of attack prove their worth by hitting back. -
> Piet
> > > > Hein
> > > > > > (via Tom White)
> > > > > >
> > > > >
> > > >
> > >
> >
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: [DISCUSSION] Merge of the hbase-11339 mob branch into master.
Posted by Ted Yu <yu...@gmail.com>.
There have been several iterations since the first mega patch was posted.
Currently QA run is green.
IntegrationTestIngestWithMOB has been run which passes.
I want to give +1 for merging to master branch.
On Tue, Jul 7, 2015 at 9:37 PM, Anoop John <an...@gmail.com> wrote:
> +1
>
> We will work on making it a Vote thread.
>
> -Anoop-
>
> On Wed, Jul 8, 2015 at 9:46 AM, ramkrishna vasudevan <
> ramkrishna.s.vasudevan@gmail.com> wrote:
>
> > +1 to start the voting again.
> >
> > On Wed, Jul 8, 2015 at 9:07 AM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > Looks like all MOB-related JIRAs have been resolved.
> > >
> > > Should the voting process be resumed ?
> > >
> > > Cheers
> > >
> > > On Thu, May 28, 2015 at 5:48 PM, Anoop John <an...@gmail.com>
> > wrote:
> > >
> > > > Yes Andy. The sweep tool is completely optional now. We have a chore
> > > doing
> > > > the compaction, like we trigger auto major compaction. We can
> > configure
> > > > the interval. Auto can be turned off and user can explicitly call
> also.
> > > We
> > > > have shell and API support.
> > > >
> > > > Anoop
> > > >
> > > > On Friday, May 29, 2015, Andrew Purtell <ap...@apache.org> wrote:
> > > > > MOB references in cells won't find their value if the MOB hfile has
> > > been
> > > > > corrupted. Dealing with that would be like any other corrupted
> HFile,
> > > > > understood. The dangling references make thinking about (partial)
> > > > recovery
> > > > > and repair interesting.
> > > > >
> > > > >
> > > > > On Thursday, May 28, 2015, Jingcheng Du <ji...@intel.com>
> > > wrote:
> > > > >
> > > > >> Andrew Purtell wrote
> > > > >> > HBCK can check and sideline dangling reference files. I think of
> > MOB
> > > > >> files
> > > > >> > as "core enough" auxiliary files that need some support. I
> suppose
> > > > unlike
> > > > >> > reference files their presence or absence won't produce a region
> > > open
> > > > >> > failure, we would see dangling pointers later when tying to
> > service
> > > > >> > queries. (Yes?) Will that abort the RS? Pardon the ignorant
> > > question,
> > > > >> > normally I could check the code but I'm at the airport on a
> phone.
> > > > >>
> > > > >> Thanks for comments!
> > > > >> In mob, usually the reference cells are committed after the mob
> > files
> > > > are
> > > > >> done. I think it hardly happens that a reference cell cannot find
> > its
> > > > mob
> > > > >> file.
> > > > >> Even if there's a dangling reference cell, the RS won't be
> aborted,
> > a
> > > > empty
> > > > >> cell is returned instead.
> > > > >>
> > > > >>
> > > > >> Andrew Purtell wrote
> > > > >> > On that subject, I should file follow up issues for more check
> and
> > > > repair
> > > > >> > options for HFiles. We should be able to detect missing or
> corrupt
> > > > files
> > > > >> > of
> > > > >> > all variety: HFile, reference, MOB. This may require an
> expensive
> > > scan
> > > > >> > over
> > > > >> > lots of files, but this is like fsck full disk surface scans and
> > > those
> > > > >> > have
> > > > >> > similar costs. Providing MR based tools is fine but we should
> have
> > > > >> > multithreaded tools that can stand in if a MR runtime is not
> > > > available.
> > > > >> > Import, Export, VerifyReplication...all of these tools are in a
> > > > >> different,
> > > > >> > lesser, class than integrity and repair tools, in my opinion.
> > Since
> > > > MOB
> > > > >> > will likely be merged into trunk by then I'll be sure to include
> > > it. I
> > > > >> > agree it's not fair to ask more of MOB then what we have now for
> > > > HFile.
> > > > >>
> > > > >> To detect the corrupt files, some code are needed in the file
> > checker
> > > to
> > > > >> check mob files after knowing it's a mob-enabled column.
> > > > >> To detect the missing or dangling reference cells, I think we have
> > to
> > > do
> > > > a
> > > > >> full-table scan, like what is done now in HFile.main.
> > > > >> We can do that.
> > > > >>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> View this message in context:
> > > > >>
> > > >
> > > >
> > >
> >
> http://apache-hbase.679495.n3.nabble.com/DISCUSSION-Merge-of-the-hbase-11339-mob-branch-into-master-tp4071644p4071911.html
> > > > >> Sent from the HBase Developer mailing list archive at Nabble.com.
> > > > >>
> > > > >
> > > > >
> > > > > --
> > > > > Best regards,
> > > > >
> > > > > - Andy
> > > > >
> > > > > Problems worthy of attack prove their worth by hitting back. - Piet
> > > Hein
> > > > > (via Tom White)
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSSION] Merge of the hbase-11339 mob branch into master.
Posted by Anoop John <an...@gmail.com>.
+1
We will work on making it a Vote thread.
-Anoop-
On Wed, Jul 8, 2015 at 9:46 AM, ramkrishna vasudevan <
ramkrishna.s.vasudevan@gmail.com> wrote:
> +1 to start the voting again.
>
> On Wed, Jul 8, 2015 at 9:07 AM, Ted Yu <yu...@gmail.com> wrote:
>
> > Looks like all MOB-related JIRAs have been resolved.
> >
> > Should the voting process be resumed ?
> >
> > Cheers
> >
> > On Thu, May 28, 2015 at 5:48 PM, Anoop John <an...@gmail.com>
> wrote:
> >
> > > Yes Andy. The sweep tool is completely optional now. We have a chore
> > doing
> > > the compaction, like we trigger auto major compaction. We can
> configure
> > > the interval. Auto can be turned off and user can explicitly call also.
> > We
> > > have shell and API support.
> > >
> > > Anoop
> > >
> > > On Friday, May 29, 2015, Andrew Purtell <ap...@apache.org> wrote:
> > > > MOB references in cells won't find their value if the MOB hfile has
> > been
> > > > corrupted. Dealing with that would be like any other corrupted HFile,
> > > > understood. The dangling references make thinking about (partial)
> > > recovery
> > > > and repair interesting.
> > > >
> > > >
> > > > On Thursday, May 28, 2015, Jingcheng Du <ji...@intel.com>
> > wrote:
> > > >
> > > >> Andrew Purtell wrote
> > > >> > HBCK can check and sideline dangling reference files. I think of
> MOB
> > > >> files
> > > >> > as "core enough" auxiliary files that need some support. I suppose
> > > unlike
> > > >> > reference files their presence or absence won't produce a region
> > open
> > > >> > failure, we would see dangling pointers later when tying to
> service
> > > >> > queries. (Yes?) Will that abort the RS? Pardon the ignorant
> > question,
> > > >> > normally I could check the code but I'm at the airport on a phone.
> > > >>
> > > >> Thanks for comments!
> > > >> In mob, usually the reference cells are committed after the mob
> files
> > > are
> > > >> done. I think it hardly happens that a reference cell cannot find
> its
> > > mob
> > > >> file.
> > > >> Even if there's a dangling reference cell, the RS won't be aborted,
> a
> > > empty
> > > >> cell is returned instead.
> > > >>
> > > >>
> > > >> Andrew Purtell wrote
> > > >> > On that subject, I should file follow up issues for more check and
> > > repair
> > > >> > options for HFiles. We should be able to detect missing or corrupt
> > > files
> > > >> > of
> > > >> > all variety: HFile, reference, MOB. This may require an expensive
> > scan
> > > >> > over
> > > >> > lots of files, but this is like fsck full disk surface scans and
> > those
> > > >> > have
> > > >> > similar costs. Providing MR based tools is fine but we should have
> > > >> > multithreaded tools that can stand in if a MR runtime is not
> > > available.
> > > >> > Import, Export, VerifyReplication...all of these tools are in a
> > > >> different,
> > > >> > lesser, class than integrity and repair tools, in my opinion.
> Since
> > > MOB
> > > >> > will likely be merged into trunk by then I'll be sure to include
> > it. I
> > > >> > agree it's not fair to ask more of MOB then what we have now for
> > > HFile.
> > > >>
> > > >> To detect the corrupt files, some code are needed in the file
> checker
> > to
> > > >> check mob files after knowing it's a mob-enabled column.
> > > >> To detect the missing or dangling reference cells, I think we have
> to
> > do
> > > a
> > > >> full-table scan, like what is done now in HFile.main.
> > > >> We can do that.
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> View this message in context:
> > > >>
> > >
> > >
> >
> http://apache-hbase.679495.n3.nabble.com/DISCUSSION-Merge-of-the-hbase-11339-mob-branch-into-master-tp4071644p4071911.html
> > > >> Sent from the HBase Developer mailing list archive at Nabble.com.
> > > >>
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > >
> > > > - Andy
> > > >
> > > > Problems worthy of attack prove their worth by hitting back. - Piet
> > Hein
> > > > (via Tom White)
> > > >
> > >
> >
>
Re: [DISCUSSION] Merge of the hbase-11339 mob branch into master.
Posted by ramkrishna vasudevan <ra...@gmail.com>.
+1 to start the voting again.
On Wed, Jul 8, 2015 at 9:07 AM, Ted Yu <yu...@gmail.com> wrote:
> Looks like all MOB-related JIRAs have been resolved.
>
> Should the voting process be resumed ?
>
> Cheers
>
> On Thu, May 28, 2015 at 5:48 PM, Anoop John <an...@gmail.com> wrote:
>
> > Yes Andy. The sweep tool is completely optional now. We have a chore
> doing
> > the compaction, like we trigger auto major compaction. We can configure
> > the interval. Auto can be turned off and user can explicitly call also.
> We
> > have shell and API support.
> >
> > Anoop
> >
> > On Friday, May 29, 2015, Andrew Purtell <ap...@apache.org> wrote:
> > > MOB references in cells won't find their value if the MOB hfile has
> been
> > > corrupted. Dealing with that would be like any other corrupted HFile,
> > > understood. The dangling references make thinking about (partial)
> > recovery
> > > and repair interesting.
> > >
> > >
> > > On Thursday, May 28, 2015, Jingcheng Du <ji...@intel.com>
> wrote:
> > >
> > >> Andrew Purtell wrote
> > >> > HBCK can check and sideline dangling reference files. I think of MOB
> > >> files
> > >> > as "core enough" auxiliary files that need some support. I suppose
> > unlike
> > >> > reference files their presence or absence won't produce a region
> open
> > >> > failure, we would see dangling pointers later when tying to service
> > >> > queries. (Yes?) Will that abort the RS? Pardon the ignorant
> question,
> > >> > normally I could check the code but I'm at the airport on a phone.
> > >>
> > >> Thanks for comments!
> > >> In mob, usually the reference cells are committed after the mob files
> > are
> > >> done. I think it hardly happens that a reference cell cannot find its
> > mob
> > >> file.
> > >> Even if there's a dangling reference cell, the RS won't be aborted, a
> > empty
> > >> cell is returned instead.
> > >>
> > >>
> > >> Andrew Purtell wrote
> > >> > On that subject, I should file follow up issues for more check and
> > repair
> > >> > options for HFiles. We should be able to detect missing or corrupt
> > files
> > >> > of
> > >> > all variety: HFile, reference, MOB. This may require an expensive
> scan
> > >> > over
> > >> > lots of files, but this is like fsck full disk surface scans and
> those
> > >> > have
> > >> > similar costs. Providing MR based tools is fine but we should have
> > >> > multithreaded tools that can stand in if a MR runtime is not
> > available.
> > >> > Import, Export, VerifyReplication...all of these tools are in a
> > >> different,
> > >> > lesser, class than integrity and repair tools, in my opinion. Since
> > MOB
> > >> > will likely be merged into trunk by then I'll be sure to include
> it. I
> > >> > agree it's not fair to ask more of MOB then what we have now for
> > HFile.
> > >>
> > >> To detect the corrupt files, some code are needed in the file checker
> to
> > >> check mob files after knowing it's a mob-enabled column.
> > >> To detect the missing or dangling reference cells, I think we have to
> do
> > a
> > >> full-table scan, like what is done now in HFile.main.
> > >> We can do that.
> > >>
> > >>
> > >>
> > >> --
> > >> View this message in context:
> > >>
> >
> >
> http://apache-hbase.679495.n3.nabble.com/DISCUSSION-Merge-of-the-hbase-11339-mob-branch-into-master-tp4071644p4071911.html
> > >> Sent from the HBase Developer mailing list archive at Nabble.com.
> > >>
> > >
> > >
> > > --
> > > Best regards,
> > >
> > > - Andy
> > >
> > > Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> > > (via Tom White)
> > >
> >
>
Re: [DISCUSSION] Merge of the hbase-11339 mob branch into master.
Posted by Ted Yu <yu...@gmail.com>.
Looks like all MOB-related JIRAs have been resolved.
Should the voting process be resumed ?
Cheers
On Thu, May 28, 2015 at 5:48 PM, Anoop John <an...@gmail.com> wrote:
> Yes Andy. The sweep tool is completely optional now. We have a chore doing
> the compaction, like we trigger auto major compaction. We can configure
> the interval. Auto can be turned off and user can explicitly call also. We
> have shell and API support.
>
> Anoop
>
> On Friday, May 29, 2015, Andrew Purtell <ap...@apache.org> wrote:
> > MOB references in cells won't find their value if the MOB hfile has been
> > corrupted. Dealing with that would be like any other corrupted HFile,
> > understood. The dangling references make thinking about (partial)
> recovery
> > and repair interesting.
> >
> >
> > On Thursday, May 28, 2015, Jingcheng Du <ji...@intel.com> wrote:
> >
> >> Andrew Purtell wrote
> >> > HBCK can check and sideline dangling reference files. I think of MOB
> >> files
> >> > as "core enough" auxiliary files that need some support. I suppose
> unlike
> >> > reference files their presence or absence won't produce a region open
> >> > failure, we would see dangling pointers later when tying to service
> >> > queries. (Yes?) Will that abort the RS? Pardon the ignorant question,
> >> > normally I could check the code but I'm at the airport on a phone.
> >>
> >> Thanks for comments!
> >> In mob, usually the reference cells are committed after the mob files
> are
> >> done. I think it hardly happens that a reference cell cannot find its
> mob
> >> file.
> >> Even if there's a dangling reference cell, the RS won't be aborted, a
> empty
> >> cell is returned instead.
> >>
> >>
> >> Andrew Purtell wrote
> >> > On that subject, I should file follow up issues for more check and
> repair
> >> > options for HFiles. We should be able to detect missing or corrupt
> files
> >> > of
> >> > all variety: HFile, reference, MOB. This may require an expensive scan
> >> > over
> >> > lots of files, but this is like fsck full disk surface scans and those
> >> > have
> >> > similar costs. Providing MR based tools is fine but we should have
> >> > multithreaded tools that can stand in if a MR runtime is not
> available.
> >> > Import, Export, VerifyReplication...all of these tools are in a
> >> different,
> >> > lesser, class than integrity and repair tools, in my opinion. Since
> MOB
> >> > will likely be merged into trunk by then I'll be sure to include it. I
> >> > agree it's not fair to ask more of MOB then what we have now for
> HFile.
> >>
> >> To detect the corrupt files, some code are needed in the file checker to
> >> check mob files after knowing it's a mob-enabled column.
> >> To detect the missing or dangling reference cells, I think we have to do
> a
> >> full-table scan, like what is done now in HFile.main.
> >> We can do that.
> >>
> >>
> >>
> >> --
> >> View this message in context:
> >>
>
> http://apache-hbase.679495.n3.nabble.com/DISCUSSION-Merge-of-the-hbase-11339-mob-branch-into-master-tp4071644p4071911.html
> >> Sent from the HBase Developer mailing list archive at Nabble.com.
> >>
> >
> >
> > --
> > Best regards,
> >
> > - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>
Re: [DISCUSSION] Merge of the hbase-11339 mob branch into master.
Posted by Anoop John <an...@gmail.com>.
Yes Andy. The sweep tool is completely optional now. We have a chore doing
the compaction, like we trigger auto major compaction. We can configure
the interval. Auto can be turned off and user can explicitly call also. We
have shell and API support.
Anoop
On Friday, May 29, 2015, Andrew Purtell <ap...@apache.org> wrote:
> MOB references in cells won't find their value if the MOB hfile has been
> corrupted. Dealing with that would be like any other corrupted HFile,
> understood. The dangling references make thinking about (partial) recovery
> and repair interesting.
>
>
> On Thursday, May 28, 2015, Jingcheng Du <ji...@intel.com> wrote:
>
>> Andrew Purtell wrote
>> > HBCK can check and sideline dangling reference files. I think of MOB
>> files
>> > as "core enough" auxiliary files that need some support. I suppose
unlike
>> > reference files their presence or absence won't produce a region open
>> > failure, we would see dangling pointers later when tying to service
>> > queries. (Yes?) Will that abort the RS? Pardon the ignorant question,
>> > normally I could check the code but I'm at the airport on a phone.
>>
>> Thanks for comments!
>> In mob, usually the reference cells are committed after the mob files are
>> done. I think it hardly happens that a reference cell cannot find its mob
>> file.
>> Even if there's a dangling reference cell, the RS won't be aborted, a
empty
>> cell is returned instead.
>>
>>
>> Andrew Purtell wrote
>> > On that subject, I should file follow up issues for more check and
repair
>> > options for HFiles. We should be able to detect missing or corrupt
files
>> > of
>> > all variety: HFile, reference, MOB. This may require an expensive scan
>> > over
>> > lots of files, but this is like fsck full disk surface scans and those
>> > have
>> > similar costs. Providing MR based tools is fine but we should have
>> > multithreaded tools that can stand in if a MR runtime is not available.
>> > Import, Export, VerifyReplication...all of these tools are in a
>> different,
>> > lesser, class than integrity and repair tools, in my opinion. Since MOB
>> > will likely be merged into trunk by then I'll be sure to include it. I
>> > agree it's not fair to ask more of MOB then what we have now for HFile.
>>
>> To detect the corrupt files, some code are needed in the file checker to
>> check mob files after knowing it's a mob-enabled column.
>> To detect the missing or dangling reference cells, I think we have to do
a
>> full-table scan, like what is done now in HFile.main.
>> We can do that.
>>
>>
>>
>> --
>> View this message in context:
>>
http://apache-hbase.679495.n3.nabble.com/DISCUSSION-Merge-of-the-hbase-11339-mob-branch-into-master-tp4071644p4071911.html
>> Sent from the HBase Developer mailing list archive at Nabble.com.
>>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
Re: [DISCUSSION] Merge of the hbase-11339 mob branch into master.
Posted by Andrew Purtell <ap...@apache.org>.
MOB references in cells won't find their value if the MOB hfile has been
corrupted. Dealing with that would be like any other corrupted HFile,
understood. The dangling references make thinking about (partial) recovery
and repair interesting.
On Thursday, May 28, 2015, Jingcheng Du <ji...@intel.com> wrote:
> Andrew Purtell wrote
> > HBCK can check and sideline dangling reference files. I think of MOB
> files
> > as "core enough" auxiliary files that need some support. I suppose unlike
> > reference files their presence or absence won't produce a region open
> > failure, we would see dangling pointers later when tying to service
> > queries. (Yes?) Will that abort the RS? Pardon the ignorant question,
> > normally I could check the code but I'm at the airport on a phone.
>
> Thanks for comments!
> In mob, usually the reference cells are committed after the mob files are
> done. I think it hardly happens that a reference cell cannot find its mob
> file.
> Even if there's a dangling reference cell, the RS won't be aborted, a empty
> cell is returned instead.
>
>
> Andrew Purtell wrote
> > On that subject, I should file follow up issues for more check and repair
> > options for HFiles. We should be able to detect missing or corrupt files
> > of
> > all variety: HFile, reference, MOB. This may require an expensive scan
> > over
> > lots of files, but this is like fsck full disk surface scans and those
> > have
> > similar costs. Providing MR based tools is fine but we should have
> > multithreaded tools that can stand in if a MR runtime is not available.
> > Import, Export, VerifyReplication...all of these tools are in a
> different,
> > lesser, class than integrity and repair tools, in my opinion. Since MOB
> > will likely be merged into trunk by then I'll be sure to include it. I
> > agree it's not fair to ask more of MOB then what we have now for HFile.
>
> To detect the corrupt files, some code are needed in the file checker to
> check mob files after knowing it's a mob-enabled column.
> To detect the missing or dangling reference cells, I think we have to do a
> full-table scan, like what is done now in HFile.main.
> We can do that.
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/DISCUSSION-Merge-of-the-hbase-11339-mob-branch-into-master-tp4071644p4071911.html
> Sent from the HBase Developer mailing list archive at Nabble.com.
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: [DISCUSSION] Merge of the hbase-11339 mob branch into master.
Posted by Jingcheng Du <ji...@intel.com>.
Andrew Purtell wrote
> HBCK can check and sideline dangling reference files. I think of MOB files
> as "core enough" auxiliary files that need some support. I suppose unlike
> reference files their presence or absence won't produce a region open
> failure, we would see dangling pointers later when tying to service
> queries. (Yes?) Will that abort the RS? Pardon the ignorant question,
> normally I could check the code but I'm at the airport on a phone.
Thanks for comments!
In mob, usually the reference cells are committed after the mob files are
done. I think it hardly happens that a reference cell cannot find its mob
file.
Even if there's a dangling reference cell, the RS won't be aborted, a empty
cell is returned instead.
Andrew Purtell wrote
> On that subject, I should file follow up issues for more check and repair
> options for HFiles. We should be able to detect missing or corrupt files
> of
> all variety: HFile, reference, MOB. This may require an expensive scan
> over
> lots of files, but this is like fsck full disk surface scans and those
> have
> similar costs. Providing MR based tools is fine but we should have
> multithreaded tools that can stand in if a MR runtime is not available.
> Import, Export, VerifyReplication...all of these tools are in a different,
> lesser, class than integrity and repair tools, in my opinion. Since MOB
> will likely be merged into trunk by then I'll be sure to include it. I
> agree it's not fair to ask more of MOB then what we have now for HFile.
To detect the corrupt files, some code are needed in the file checker to
check mob files after knowing it's a mob-enabled column.
To detect the missing or dangling reference cells, I think we have to do a
full-table scan, like what is done now in HFile.main.
We can do that.
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/DISCUSSION-Merge-of-the-hbase-11339-mob-branch-into-master-tp4071644p4071911.html
Sent from the HBase Developer mailing list archive at Nabble.com.
[DISCUSSION] Merge of the hbase-11339 mob branch into master.
Posted by Andrew Purtell <ap...@apache.org>.
Inline
On Wednesday, May 27, 2015, Jonathan Hsieh <jon@cloudera.com
<javascript:_e(%7B%7D,'cvml','jon@cloudera.com');>> wrote:
> On Sat, May 23, 2015 at 2:51 PM, Andrew Purtell <ap...@apache.org>
> wrote:
>
> > I was responding to this comment from Jon's email:
> >
> > > Another suggestion was a tool to check that mob references had
> > > corresponding mob data. We currently include a mr-based sweeper job
> > > that could be used to perform this verification. We can add this tool
> > and
> > > testing for the tool.
> >
> > So for those of us not intimately familiar with the MOB work, is there a
> > tool that checks for MOB integrity, or a tool that can be adapted for
> that
> > purpose, and does it require MR or not? More generally: Can MOB integrity
> > checks be added or folded into HBCK? I think you can see what my concerns
> > are but if they are unclear please let me know and I will clarify them
> > further.
> >
> >
> > The main purpose of hbck is to see if region metadata consistent in all
> locations. hbck checks metadata by scaning hbase:meta, reading .regioninfo
> metadata files, and interrogating region servers. It does not actually
> access the bulk of the data found in hfiles. It never had the facilities
> to check validity of data and as designed hbase doesn't have the ability
> doesn't catch things like deleted, missing or truncated hfiles.
>
>
HBCK can check and sideline dangling reference files. I think of MOB files
as "core enough" auxiliary files that need some support. I suppose unlike
reference files their presence or absence won't produce a region open
failure, we would see dangling pointers later when tying to service
queries. (Yes?) Will that abort the RS? Pardon the ignorant question,
normally I could check the code but I'm at the airport on a phone.
Good point below on the difference between HFile.main and HBCK and the
limitations of these tools.
On that subject, I should file follow up issues for more check and repair
options for HFiles. We should be able to detect missing or corrupt files of
all variety: HFile, reference, MOB. This may require an expensive scan over
lots of files, but this is like fsck full disk surface scans and those have
similar costs. Providing MR based tools is fine but we should have
multithreaded tools that can stand in if a MR runtime is not available.
Import, Export, VerifyReplication...all of these tools are in a different,
lesser, class than integrity and repair tools, in my opinion. Since MOB
will likely be merged into trunk by then I'll be sure to include it. I
agree it's not fair to ask more of MOB then what we have now for HFile.
> As such, I don't think integration into hbck is appropriate for mob (or
> checking the validity of any other tag-based feature).
When the tag based features become widely used this will be a distinction
without a difference. All the tools will need to learn about them.
> I do think it makes sense to have a tool more akin to a modified version of
> HFile.main() that can read an hfile and detect bad link between a mob ref
> and a mob value. Howeve, I actually don't think there is a out of the box
> tool that we can use today to recover a truncated hfile and I don't think
> there is any tool that checks the validity in any of the other tag-based
> features (please correct me if I am wrong)
>
> An alternative since this is a infrequently run operation, would be a tool
> a map reduce job that uses the mob scanner filter to check an entire
> table. The skeleton for this is essentially written already -- the mob
> sweeper could be modified to output mob integrity violations. Similarly, I
> don't think validation of a full table's tag data was required for other
> tag-based features (please correct me if I am wrong).
>
> On Sat, May 23, 2015 at 11:15 AM, Matteo Bertozzi <theo.bertozzi@gmail.com
> >
> > wrote:
> >
> > > as far as I know MOB does not depend anymore on MR
> > > the old MR sweeper tool is still around, and you can use it to compact
> > > manually
> > > but it is not called by the normal RS compaction code.
> > >
> > > also, the MOB code is more or less isolated.
> > > if your family is not using MOB you still have your old code path.
> > > so, I'd say that if we don't break compatibility and
> > > the few changes in the core-path, to do the if mobIsEnabled, do not
> > impact
> > > the perf of the traditional path
> > > we can probably get the feature in 1.2 as "experimental".
> > > brave users can experiment with it, report bugs and suggestions
> > > and then we will mark it as stable in 1.3, 1.4 or whenever is ready.
> > >
> > >
> > > Matteo
> > >
> > >
> > > On Sat, May 23, 2015 at 9:47 AM, Andrew Purtell <ap...@apache.org>
> > > wrote:
> > >
> > > > Maybe we can remove the dependency on a MR runtime for MOB
> maintenance
> > by
> > > > reimplementing those parallel tasks using Procedure V2? We wouldn't
> be
> > > > looking at MOB for 1.2 but maybe 1.3? I'm also not sure the community
> > as
> > > a
> > > > whole has the necessary bandwidth for perf and stability testing of
> MOB
> > > in
> > > > the 1.2 timeframe, but 1.3 would be more likely.
> > > >
> > > >
> > > > On Sat, May 23, 2015 at 9:40 AM, Andrew Purtell <apurtell@apache.org
> >
> > > > wrote:
> > > >
> > > > > Regarding performance testing: Whatever has been done on the MOB
> > branch
> > > > > will be interesting data points, and, potentially encouraging, but
> > > > porting
> > > > > to branch-1 will produce a new code base. Earlier results on other
> > code
> > > > > will not be applicable. We have to start over. Like I said
> elsewhere,
> > > I'm
> > > > > happy to help with (re)characterizing the perf impact and
> > improvements
> > > > > produced by the changes.
> > > > >
> > > > > What coverage do we have for verifying the integrity of MOB
> > references?
> > > > > Will the sweep tool detect, alert on, and optionally repair
> dangling
> > > > > references? (I could answer this for myself by looking at MOB
> branch,
> > > but
> > > > > hopefully someone here has an answer at the ready.) I assume we
> > > calculate
> > > > > and store checksums for MOB data itself so we know if values are
> > > corrupt.
> > > > > Does the sweep tool detect MOB value corruption? Can it be
> repaired?
> > Do
> > > > we
> > > > > have a good ops story for why HBCK is no longer sufficient on its
> > own,
> > > > > there's a separate tool with a whole new set of options - and a
> > > > requirement
> > > > > for a MR runtime! - for checking MOB data? That last one is a
> > > rhetorical
> > > > > question (smile), the ops story is... unsatisfying. It's like we've
> > > > taken a
> > > > > self sufficient HBase and bolted in parts of Hive, so now we need
> MR.
> > > > >
> > > > >
> > > > > On Fri, May 22, 2015 at 1:45 PM, Jonathan Hsieh <jo...@cloudera.com>
> > > > wrote:
> > > > >
> > > > >> In another thread andrew purtell brought up some concerns about
> the
> > > mob
> > > > >> feature:
> > > > >>
> > > > >> On Fri, May 22, 2015 at 12:40 PM, Andrew Purtell <
> > apurtell@apache.org
> > > >
> > > > >> wrote:
> > > > >>
> > > > >> > Another point of clarification, sorry, I hit the send button too
> > > early
> > > > >> it
> > > > >> > seems: I don't believe MOB is fully integrated yet, for example
> > the
> > > > >> > feature
> > > > >> > is an extension to store that lacks support for encryption (this
> > > would
> > > > >> > technically be a feature regression); and HBCK. I have not been
> > > > >> following
> > > > >> > MOB too closely so could be mistaken. These issues do not
> > preclude a
> > > > >> merge
> > > > >> > of MOB into trunk, but do preclude a merge back of MOB from
> trunk
> > to
> > > > >> > branch-1. I would veto the latter until such shortcomings in the
> > > > >> > implementation that could be described as regressions are
> > > addressed. I
> > > > >> > would also like to see a performance analysis of a range of
> > > workloads
> > > > >> > before and after in as much detail as can be mustered, and would
> > be
> > > > >> happy
> > > > >> > to volunteer to help out with that.
> > > > >> >
> > > > >>
> > > > >> Here's info on the points brought up:
> > > > >>
> > > > >> Encryption support shortcoming is being addrsessed here:
> > > > >> https://issues.apache.org/jira/browse/HBASE-13693 (closed)
> > > > >> https://issues.apache.org/jira/browse/HBASE-13720 (in review)
> > > > >>
> > > > >> Hbck has been actually run against the integration test rigs while
> > the
> > > > >> feature has been enabled but currently has no explicit unit test
> or
> > > > simple
> > > > >> to run integration test. It currently doesn't report anything
> > special
> > > > >> about the mob storage area. We can add unit tests that cover hbck
> > when
> > > > the
> > > > >> mob path is exercised.
> > > > >>
> > > > >> Another suggestion was a tool to check that mob references had
> > > > >> corresponding mob data. We currently include a mr-based sweeper
> job
> > > > that
> > > > >> could be used to perform this verification. We can add this tool
> > and
> > > > >> testing for the tool.
> > > > >>
> > > > >> I've done some performance testing and Jingcheng and his
> colleagues
> > > have
> > > > >> done significant amounts of performance testing. We currently
> have a
> > > > blog
> > > > >> post in progress that will share the results of this performance
> > > > testing.
> > > > >>
> > > > >> Jon.
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Wed, May 20, 2015 at 7:38 PM, Ted Yu <yu...@gmail.com>
> > wrote:
> > > > >>
> > > > >> > This is a useful feature, Jon.
> > > > >> >
> > > > >> > I went over the mega-patch and left some comments on review
> board.
> > > > >> >
> > > > >> > I noticed that hbck was not included in the patch. Neither did I
> > > find
> > > > a
> > > > >> > sub-task of HBASE-11339 that covers hbck.
> > > > >> >
> > > > >> > Do you or Jingcheng plan to add MOB-aware capability for hbck ?
> > > > >> >
> > > > >> > Cheers
> > > > >> >
> > > > >> > On Wed, May 20, 2015 at 9:21 AM, Jonathan Hsieh <
> jon@cloudera.com
> > >
> > > > >> wrote:
> > > > >> >
> > > > >> > > Hi folks,
> > > > >> > >
> > > > >> > > The Medium Object (MOB) Storage feature (HBASE-11339[1]) is
> > > modified
> > > > >> I/O
> > > > >> > > and compaction path that allows individual moderately sized
> > values
> > > > >> > > (10k-10MB) to be stored so that write amplification is reduced
> > > when
> > > > >> > > compared to the normal I/O path. At a high level, it
> provides
> > > > >> alternate
> > > > >> > > flush and compaction mechanisms that segregates large cells
> > into a
> > > > >> > separate
> > > > >> > > area where they are not subject to potentially frequent
> > compaction
> > > > and
> > > > >> > > splits that can be encountered in the normal I/O path. A more
> > > > detailed
> > > > >> > > design doc can be found on the hbase-11339 jira.
> > > > >> > >
> > > > >> > > Jingcheng Du has been working on the mob feature for a while
> and
> > > > >> Anoop,
> > > > >> > Ram
> > > > >> > > and I have been shepherding him through the design revisions
> and
> > > > >> > > implementation of the feature in the hbase-11339 branch.[2]
> > > > >> > >
> > > > >> > > The branch we are proposing to merge into master is compatible
> > > with
> > > > >> > HBase's
> > > > >> > > core functionality including snapshots, replication, shell
> > > support,
> > > > >> > behaves
> > > > >> > > well with table alters, bulk loads and does not require
> external
> > > MR
> > > > >> > > processes. It has been documented, and subject to many
> > integration
> > > > >> test
> > > > >> > > runs (ITBLL, ITAcidGuarantees, ITIngest) including fault
> > > injection.
> > > > >> > > Performance testing of the feature shows what can be a 2x-3x
> > > > >> throughput
> > > > >> > > improvement for workloads that contain mobs. These results can
> > be
> > > > >> seen on
> > > > >> > > the hbase 2.0 panel discussion slides from hbasecon (once
> > > > published).
> > > > >> > >
> > > > >> > > Recently there have been some hfile encryption related
> > > shortcomings
> > > > >> that
> > > > >> > we
> > > > >> > > could address in branch or in master.
> > > > >> > >
> > > > >> > > Earlier iterations of the feature has been tested in
> production
> > by
> > > > >> users
> > > > >> > > that Jingcheng has been responsible for. A version has also
> > been
> > > > >> > deployed
> > > > >> > > at users I have been responsible for. Some of the folks from
> > > Huawei
> > > > >> > > (ashutosh) have also been submitting the recent encryption bug
> > > > reports
> > > > >> > > against the hbase-11339 branch so there is some evidence of
> > usage
> > > by
> > > > >> > them.
> > > > >> > >
> > > > >> > > The four of us (Jingcheng, Ram, Anoop and I) are satisfied
> with
> > > the
> > > > >> > > feature and feel it is a good time to call a merge vote. Ive
> > > > posted a
> > > > >> > > megapatch version for folks who want to peruse the code. [3]
> > > > >> > >
> > > > >> > > What do you all think?
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > > Jingcheng, Jon, Ram, and Anoop.
> > > > >> > >
> > > > >> > > [1] https://issues.apache.org/jira/browse/HBASE-11339
> > > > >> > > [2] https://github.com/apache/hbase/tree/hbase-11339
> > > > >> > > [3] https://reviews.apache.org/r/34475/
> > > > >> > > --
> > > > >> > > // Jonathan Hsieh (shay)
> > > > >> > > // HBase Tech Lead, Software Engineer, Cloudera
> > > > >> > > // jon@cloudera.com // @jmhsieh
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> // Jonathan Hsieh (shay)
> > > > >> // HBase Tech Lead, Software Engineer, Cloudera
> > > > >> // jon@cloudera.com // @jmhsieh
> > > > >>
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Best regards,
> >
> > - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>
>
>
> --
> // Jonathan Hsieh (shay)
> // HBase Tech Lead, Software Engineer, Cloudera
> // jon@cloudera.com // @jmhsieh
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: [DISCUSSION] Merge of the hbase-11339 mob branch into master.
Posted by Jonathan Hsieh <jo...@cloudera.com>.
On Sat, May 23, 2015 at 2:51 PM, Andrew Purtell <ap...@apache.org> wrote:
> I was responding to this comment from Jon's email:
>
> > Another suggestion was a tool to check that mob references had
> > corresponding mob data. We currently include a mr-based sweeper job
> > that could be used to perform this verification. We can add this tool
> and
> > testing for the tool.
>
> So for those of us not intimately familiar with the MOB work, is there a
> tool that checks for MOB integrity, or a tool that can be adapted for that
> purpose, and does it require MR or not? More generally: Can MOB integrity
> checks be added or folded into HBCK? I think you can see what my concerns
> are but if they are unclear please let me know and I will clarify them
> further.
>
>
> The main purpose of hbck is to see if region metadata consistent in all
locations. hbck checks metadata by scaning hbase:meta, reading .regioninfo
metadata files, and interrogating region servers. It does not actually
access the bulk of the data found in hfiles. It never had the facilities
to check validity of data and as designed hbase doesn't have the ability
doesn't catch things like deleted, missing or truncated hfiles.
As such, I don't think integration into hbck is appropriate for mob (or
checking the validity of any other tag-based feature).
I do think it makes sense to have a tool more akin to a modified version of
HFile.main() that can read an hfile and detect bad link between a mob ref
and a mob value. Howeve, I actually don't think there is a out of the box
tool that we can use today to recover a truncated hfile and I don't think
there is any tool that checks the validity in any of the other tag-based
features (please correct me if I am wrong)
An alternative since this is a infrequently run operation, would be a tool
a map reduce job that uses the mob scanner filter to check an entire
table. The skeleton for this is essentially written already -- the mob
sweeper could be modified to output mob integrity violations. Similarly, I
don't think validation of a full table's tag data was required for other
tag-based features (please correct me if I am wrong).
On Sat, May 23, 2015 at 11:15 AM, Matteo Bertozzi <th...@gmail.com>
> wrote:
>
> > as far as I know MOB does not depend anymore on MR
> > the old MR sweeper tool is still around, and you can use it to compact
> > manually
> > but it is not called by the normal RS compaction code.
> >
> > also, the MOB code is more or less isolated.
> > if your family is not using MOB you still have your old code path.
> > so, I'd say that if we don't break compatibility and
> > the few changes in the core-path, to do the if mobIsEnabled, do not
> impact
> > the perf of the traditional path
> > we can probably get the feature in 1.2 as "experimental".
> > brave users can experiment with it, report bugs and suggestions
> > and then we will mark it as stable in 1.3, 1.4 or whenever is ready.
> >
> >
> > Matteo
> >
> >
> > On Sat, May 23, 2015 at 9:47 AM, Andrew Purtell <ap...@apache.org>
> > wrote:
> >
> > > Maybe we can remove the dependency on a MR runtime for MOB maintenance
> by
> > > reimplementing those parallel tasks using Procedure V2? We wouldn't be
> > > looking at MOB for 1.2 but maybe 1.3? I'm also not sure the community
> as
> > a
> > > whole has the necessary bandwidth for perf and stability testing of MOB
> > in
> > > the 1.2 timeframe, but 1.3 would be more likely.
> > >
> > >
> > > On Sat, May 23, 2015 at 9:40 AM, Andrew Purtell <ap...@apache.org>
> > > wrote:
> > >
> > > > Regarding performance testing: Whatever has been done on the MOB
> branch
> > > > will be interesting data points, and, potentially encouraging, but
> > > porting
> > > > to branch-1 will produce a new code base. Earlier results on other
> code
> > > > will not be applicable. We have to start over. Like I said elsewhere,
> > I'm
> > > > happy to help with (re)characterizing the perf impact and
> improvements
> > > > produced by the changes.
> > > >
> > > > What coverage do we have for verifying the integrity of MOB
> references?
> > > > Will the sweep tool detect, alert on, and optionally repair dangling
> > > > references? (I could answer this for myself by looking at MOB branch,
> > but
> > > > hopefully someone here has an answer at the ready.) I assume we
> > calculate
> > > > and store checksums for MOB data itself so we know if values are
> > corrupt.
> > > > Does the sweep tool detect MOB value corruption? Can it be repaired?
> Do
> > > we
> > > > have a good ops story for why HBCK is no longer sufficient on its
> own,
> > > > there's a separate tool with a whole new set of options - and a
> > > requirement
> > > > for a MR runtime! - for checking MOB data? That last one is a
> > rhetorical
> > > > question (smile), the ops story is... unsatisfying. It's like we've
> > > taken a
> > > > self sufficient HBase and bolted in parts of Hive, so now we need MR.
> > > >
> > > >
> > > > On Fri, May 22, 2015 at 1:45 PM, Jonathan Hsieh <jo...@cloudera.com>
> > > wrote:
> > > >
> > > >> In another thread andrew purtell brought up some concerns about the
> > mob
> > > >> feature:
> > > >>
> > > >> On Fri, May 22, 2015 at 12:40 PM, Andrew Purtell <
> apurtell@apache.org
> > >
> > > >> wrote:
> > > >>
> > > >> > Another point of clarification, sorry, I hit the send button too
> > early
> > > >> it
> > > >> > seems: I don't believe MOB is fully integrated yet, for example
> the
> > > >> > feature
> > > >> > is an extension to store that lacks support for encryption (this
> > would
> > > >> > technically be a feature regression); and HBCK. I have not been
> > > >> following
> > > >> > MOB too closely so could be mistaken. These issues do not
> preclude a
> > > >> merge
> > > >> > of MOB into trunk, but do preclude a merge back of MOB from trunk
> to
> > > >> > branch-1. I would veto the latter until such shortcomings in the
> > > >> > implementation that could be described as regressions are
> > addressed. I
> > > >> > would also like to see a performance analysis of a range of
> > workloads
> > > >> > before and after in as much detail as can be mustered, and would
> be
> > > >> happy
> > > >> > to volunteer to help out with that.
> > > >> >
> > > >>
> > > >> Here's info on the points brought up:
> > > >>
> > > >> Encryption support shortcoming is being addrsessed here:
> > > >> https://issues.apache.org/jira/browse/HBASE-13693 (closed)
> > > >> https://issues.apache.org/jira/browse/HBASE-13720 (in review)
> > > >>
> > > >> Hbck has been actually run against the integration test rigs while
> the
> > > >> feature has been enabled but currently has no explicit unit test or
> > > simple
> > > >> to run integration test. It currently doesn't report anything
> special
> > > >> about the mob storage area. We can add unit tests that cover hbck
> when
> > > the
> > > >> mob path is exercised.
> > > >>
> > > >> Another suggestion was a tool to check that mob references had
> > > >> corresponding mob data. We currently include a mr-based sweeper job
> > > that
> > > >> could be used to perform this verification. We can add this tool
> and
> > > >> testing for the tool.
> > > >>
> > > >> I've done some performance testing and Jingcheng and his colleagues
> > have
> > > >> done significant amounts of performance testing. We currently have a
> > > blog
> > > >> post in progress that will share the results of this performance
> > > testing.
> > > >>
> > > >> Jon.
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> On Wed, May 20, 2015 at 7:38 PM, Ted Yu <yu...@gmail.com>
> wrote:
> > > >>
> > > >> > This is a useful feature, Jon.
> > > >> >
> > > >> > I went over the mega-patch and left some comments on review board.
> > > >> >
> > > >> > I noticed that hbck was not included in the patch. Neither did I
> > find
> > > a
> > > >> > sub-task of HBASE-11339 that covers hbck.
> > > >> >
> > > >> > Do you or Jingcheng plan to add MOB-aware capability for hbck ?
> > > >> >
> > > >> > Cheers
> > > >> >
> > > >> > On Wed, May 20, 2015 at 9:21 AM, Jonathan Hsieh <jon@cloudera.com
> >
> > > >> wrote:
> > > >> >
> > > >> > > Hi folks,
> > > >> > >
> > > >> > > The Medium Object (MOB) Storage feature (HBASE-11339[1]) is
> > modified
> > > >> I/O
> > > >> > > and compaction path that allows individual moderately sized
> values
> > > >> > > (10k-10MB) to be stored so that write amplification is reduced
> > when
> > > >> > > compared to the normal I/O path. At a high level, it provides
> > > >> alternate
> > > >> > > flush and compaction mechanisms that segregates large cells
> into a
> > > >> > separate
> > > >> > > area where they are not subject to potentially frequent
> compaction
> > > and
> > > >> > > splits that can be encountered in the normal I/O path. A more
> > > detailed
> > > >> > > design doc can be found on the hbase-11339 jira.
> > > >> > >
> > > >> > > Jingcheng Du has been working on the mob feature for a while and
> > > >> Anoop,
> > > >> > Ram
> > > >> > > and I have been shepherding him through the design revisions and
> > > >> > > implementation of the feature in the hbase-11339 branch.[2]
> > > >> > >
> > > >> > > The branch we are proposing to merge into master is compatible
> > with
> > > >> > HBase's
> > > >> > > core functionality including snapshots, replication, shell
> > support,
> > > >> > behaves
> > > >> > > well with table alters, bulk loads and does not require external
> > MR
> > > >> > > processes. It has been documented, and subject to many
> integration
> > > >> test
> > > >> > > runs (ITBLL, ITAcidGuarantees, ITIngest) including fault
> > injection.
> > > >> > > Performance testing of the feature shows what can be a 2x-3x
> > > >> throughput
> > > >> > > improvement for workloads that contain mobs. These results can
> be
> > > >> seen on
> > > >> > > the hbase 2.0 panel discussion slides from hbasecon (once
> > > published).
> > > >> > >
> > > >> > > Recently there have been some hfile encryption related
> > shortcomings
> > > >> that
> > > >> > we
> > > >> > > could address in branch or in master.
> > > >> > >
> > > >> > > Earlier iterations of the feature has been tested in production
> by
> > > >> users
> > > >> > > that Jingcheng has been responsible for. A version has also
> been
> > > >> > deployed
> > > >> > > at users I have been responsible for. Some of the folks from
> > Huawei
> > > >> > > (ashutosh) have also been submitting the recent encryption bug
> > > reports
> > > >> > > against the hbase-11339 branch so there is some evidence of
> usage
> > by
> > > >> > them.
> > > >> > >
> > > >> > > The four of us (Jingcheng, Ram, Anoop and I) are satisfied with
> > the
> > > >> > > feature and feel it is a good time to call a merge vote. Ive
> > > posted a
> > > >> > > megapatch version for folks who want to peruse the code. [3]
> > > >> > >
> > > >> > > What do you all think?
> > > >> > >
> > > >> > > Thanks,
> > > >> > > Jingcheng, Jon, Ram, and Anoop.
> > > >> > >
> > > >> > > [1] https://issues.apache.org/jira/browse/HBASE-11339
> > > >> > > [2] https://github.com/apache/hbase/tree/hbase-11339
> > > >> > > [3] https://reviews.apache.org/r/34475/
> > > >> > > --
> > > >> > > // Jonathan Hsieh (shay)
> > > >> > > // HBase Tech Lead, Software Engineer, Cloudera
> > > >> > > // jon@cloudera.com // @jmhsieh
> > > >> > >
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> // Jonathan Hsieh (shay)
> > > >> // HBase Tech Lead, Software Engineer, Cloudera
> > > >> // jon@cloudera.com // @jmhsieh
> > > >>
> > > >
> > >
> >
>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
--
// Jonathan Hsieh (shay)
// HBase Tech Lead, Software Engineer, Cloudera
// jon@cloudera.com // @jmhsieh
Re: [DISCUSSION] Merge of the hbase-11339 mob branch into master.
Posted by Andrew Purtell <ap...@apache.org>.
I was responding to this comment from Jon's email:
> Another suggestion was a tool to check that mob references had
> corresponding mob data. We currently include a mr-based sweeper job
> that could be used to perform this verification. We can add this tool and
> testing for the tool.
So for those of us not intimately familiar with the MOB work, is there a
tool that checks for MOB integrity, or a tool that can be adapted for that
purpose, and does it require MR or not? More generally: Can MOB integrity
checks be added or folded into HBCK? I think you can see what my concerns
are but if they are unclear please let me know and I will clarify them
further.
On Sat, May 23, 2015 at 11:15 AM, Matteo Bertozzi <th...@gmail.com>
wrote:
> as far as I know MOB does not depend anymore on MR
> the old MR sweeper tool is still around, and you can use it to compact
> manually
> but it is not called by the normal RS compaction code.
>
> also, the MOB code is more or less isolated.
> if your family is not using MOB you still have your old code path.
> so, I'd say that if we don't break compatibility and
> the few changes in the core-path, to do the if mobIsEnabled, do not impact
> the perf of the traditional path
> we can probably get the feature in 1.2 as "experimental".
> brave users can experiment with it, report bugs and suggestions
> and then we will mark it as stable in 1.3, 1.4 or whenever is ready.
>
>
> Matteo
>
>
> On Sat, May 23, 2015 at 9:47 AM, Andrew Purtell <ap...@apache.org>
> wrote:
>
> > Maybe we can remove the dependency on a MR runtime for MOB maintenance by
> > reimplementing those parallel tasks using Procedure V2? We wouldn't be
> > looking at MOB for 1.2 but maybe 1.3? I'm also not sure the community as
> a
> > whole has the necessary bandwidth for perf and stability testing of MOB
> in
> > the 1.2 timeframe, but 1.3 would be more likely.
> >
> >
> > On Sat, May 23, 2015 at 9:40 AM, Andrew Purtell <ap...@apache.org>
> > wrote:
> >
> > > Regarding performance testing: Whatever has been done on the MOB branch
> > > will be interesting data points, and, potentially encouraging, but
> > porting
> > > to branch-1 will produce a new code base. Earlier results on other code
> > > will not be applicable. We have to start over. Like I said elsewhere,
> I'm
> > > happy to help with (re)characterizing the perf impact and improvements
> > > produced by the changes.
> > >
> > > What coverage do we have for verifying the integrity of MOB references?
> > > Will the sweep tool detect, alert on, and optionally repair dangling
> > > references? (I could answer this for myself by looking at MOB branch,
> but
> > > hopefully someone here has an answer at the ready.) I assume we
> calculate
> > > and store checksums for MOB data itself so we know if values are
> corrupt.
> > > Does the sweep tool detect MOB value corruption? Can it be repaired? Do
> > we
> > > have a good ops story for why HBCK is no longer sufficient on its own,
> > > there's a separate tool with a whole new set of options - and a
> > requirement
> > > for a MR runtime! - for checking MOB data? That last one is a
> rhetorical
> > > question (smile), the ops story is... unsatisfying. It's like we've
> > taken a
> > > self sufficient HBase and bolted in parts of Hive, so now we need MR.
> > >
> > >
> > > On Fri, May 22, 2015 at 1:45 PM, Jonathan Hsieh <jo...@cloudera.com>
> > wrote:
> > >
> > >> In another thread andrew purtell brought up some concerns about the
> mob
> > >> feature:
> > >>
> > >> On Fri, May 22, 2015 at 12:40 PM, Andrew Purtell <apurtell@apache.org
> >
> > >> wrote:
> > >>
> > >> > Another point of clarification, sorry, I hit the send button too
> early
> > >> it
> > >> > seems: I don't believe MOB is fully integrated yet, for example the
> > >> > feature
> > >> > is an extension to store that lacks support for encryption (this
> would
> > >> > technically be a feature regression); and HBCK. I have not been
> > >> following
> > >> > MOB too closely so could be mistaken. These issues do not preclude a
> > >> merge
> > >> > of MOB into trunk, but do preclude a merge back of MOB from trunk to
> > >> > branch-1. I would veto the latter until such shortcomings in the
> > >> > implementation that could be described as regressions are
> addressed. I
> > >> > would also like to see a performance analysis of a range of
> workloads
> > >> > before and after in as much detail as can be mustered, and would be
> > >> happy
> > >> > to volunteer to help out with that.
> > >> >
> > >>
> > >> Here's info on the points brought up:
> > >>
> > >> Encryption support shortcoming is being addrsessed here:
> > >> https://issues.apache.org/jira/browse/HBASE-13693 (closed)
> > >> https://issues.apache.org/jira/browse/HBASE-13720 (in review)
> > >>
> > >> Hbck has been actually run against the integration test rigs while the
> > >> feature has been enabled but currently has no explicit unit test or
> > simple
> > >> to run integration test. It currently doesn't report anything special
> > >> about the mob storage area. We can add unit tests that cover hbck when
> > the
> > >> mob path is exercised.
> > >>
> > >> Another suggestion was a tool to check that mob references had
> > >> corresponding mob data. We currently include a mr-based sweeper job
> > that
> > >> could be used to perform this verification. We can add this tool and
> > >> testing for the tool.
> > >>
> > >> I've done some performance testing and Jingcheng and his colleagues
> have
> > >> done significant amounts of performance testing. We currently have a
> > blog
> > >> post in progress that will share the results of this performance
> > testing.
> > >>
> > >> Jon.
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> On Wed, May 20, 2015 at 7:38 PM, Ted Yu <yu...@gmail.com> wrote:
> > >>
> > >> > This is a useful feature, Jon.
> > >> >
> > >> > I went over the mega-patch and left some comments on review board.
> > >> >
> > >> > I noticed that hbck was not included in the patch. Neither did I
> find
> > a
> > >> > sub-task of HBASE-11339 that covers hbck.
> > >> >
> > >> > Do you or Jingcheng plan to add MOB-aware capability for hbck ?
> > >> >
> > >> > Cheers
> > >> >
> > >> > On Wed, May 20, 2015 at 9:21 AM, Jonathan Hsieh <jo...@cloudera.com>
> > >> wrote:
> > >> >
> > >> > > Hi folks,
> > >> > >
> > >> > > The Medium Object (MOB) Storage feature (HBASE-11339[1]) is
> modified
> > >> I/O
> > >> > > and compaction path that allows individual moderately sized values
> > >> > > (10k-10MB) to be stored so that write amplification is reduced
> when
> > >> > > compared to the normal I/O path. At a high level, it provides
> > >> alternate
> > >> > > flush and compaction mechanisms that segregates large cells into a
> > >> > separate
> > >> > > area where they are not subject to potentially frequent compaction
> > and
> > >> > > splits that can be encountered in the normal I/O path. A more
> > detailed
> > >> > > design doc can be found on the hbase-11339 jira.
> > >> > >
> > >> > > Jingcheng Du has been working on the mob feature for a while and
> > >> Anoop,
> > >> > Ram
> > >> > > and I have been shepherding him through the design revisions and
> > >> > > implementation of the feature in the hbase-11339 branch.[2]
> > >> > >
> > >> > > The branch we are proposing to merge into master is compatible
> with
> > >> > HBase's
> > >> > > core functionality including snapshots, replication, shell
> support,
> > >> > behaves
> > >> > > well with table alters, bulk loads and does not require external
> MR
> > >> > > processes. It has been documented, and subject to many integration
> > >> test
> > >> > > runs (ITBLL, ITAcidGuarantees, ITIngest) including fault
> injection.
> > >> > > Performance testing of the feature shows what can be a 2x-3x
> > >> throughput
> > >> > > improvement for workloads that contain mobs. These results can be
> > >> seen on
> > >> > > the hbase 2.0 panel discussion slides from hbasecon (once
> > published).
> > >> > >
> > >> > > Recently there have been some hfile encryption related
> shortcomings
> > >> that
> > >> > we
> > >> > > could address in branch or in master.
> > >> > >
> > >> > > Earlier iterations of the feature has been tested in production by
> > >> users
> > >> > > that Jingcheng has been responsible for. A version has also been
> > >> > deployed
> > >> > > at users I have been responsible for. Some of the folks from
> Huawei
> > >> > > (ashutosh) have also been submitting the recent encryption bug
> > reports
> > >> > > against the hbase-11339 branch so there is some evidence of usage
> by
> > >> > them.
> > >> > >
> > >> > > The four of us (Jingcheng, Ram, Anoop and I) are satisfied with
> the
> > >> > > feature and feel it is a good time to call a merge vote. Ive
> > posted a
> > >> > > megapatch version for folks who want to peruse the code. [3]
> > >> > >
> > >> > > What do you all think?
> > >> > >
> > >> > > Thanks,
> > >> > > Jingcheng, Jon, Ram, and Anoop.
> > >> > >
> > >> > > [1] https://issues.apache.org/jira/browse/HBASE-11339
> > >> > > [2] https://github.com/apache/hbase/tree/hbase-11339
> > >> > > [3] https://reviews.apache.org/r/34475/
> > >> > > --
> > >> > > // Jonathan Hsieh (shay)
> > >> > > // HBase Tech Lead, Software Engineer, Cloudera
> > >> > > // jon@cloudera.com // @jmhsieh
> > >> > >
> > >> >
> > >>
> > >>
> > >>
> > >> --
> > >> // Jonathan Hsieh (shay)
> > >> // HBase Tech Lead, Software Engineer, Cloudera
> > >> // jon@cloudera.com // @jmhsieh
> > >>
> > >
> >
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: [DISCUSSION] Merge of the hbase-11339 mob branch into master.
Posted by Matteo Bertozzi <th...@gmail.com>.
as far as I know MOB does not depend anymore on MR
the old MR sweeper tool is still around, and you can use it to compact
manually
but it is not called by the normal RS compaction code.
also, the MOB code is more or less isolated.
if your family is not using MOB you still have your old code path.
so, I'd say that if we don't break compatibility and
the few changes in the core-path, to do the if mobIsEnabled, do not impact
the perf of the traditional path
we can probably get the feature in 1.2 as "experimental".
brave users can experiment with it, report bugs and suggestions
and then we will mark it as stable in 1.3, 1.4 or whenever is ready.
Matteo
On Sat, May 23, 2015 at 9:47 AM, Andrew Purtell <ap...@apache.org> wrote:
> Maybe we can remove the dependency on a MR runtime for MOB maintenance by
> reimplementing those parallel tasks using Procedure V2? We wouldn't be
> looking at MOB for 1.2 but maybe 1.3? I'm also not sure the community as a
> whole has the necessary bandwidth for perf and stability testing of MOB in
> the 1.2 timeframe, but 1.3 would be more likely.
>
>
> On Sat, May 23, 2015 at 9:40 AM, Andrew Purtell <ap...@apache.org>
> wrote:
>
> > Regarding performance testing: Whatever has been done on the MOB branch
> > will be interesting data points, and, potentially encouraging, but
> porting
> > to branch-1 will produce a new code base. Earlier results on other code
> > will not be applicable. We have to start over. Like I said elsewhere, I'm
> > happy to help with (re)characterizing the perf impact and improvements
> > produced by the changes.
> >
> > What coverage do we have for verifying the integrity of MOB references?
> > Will the sweep tool detect, alert on, and optionally repair dangling
> > references? (I could answer this for myself by looking at MOB branch, but
> > hopefully someone here has an answer at the ready.) I assume we calculate
> > and store checksums for MOB data itself so we know if values are corrupt.
> > Does the sweep tool detect MOB value corruption? Can it be repaired? Do
> we
> > have a good ops story for why HBCK is no longer sufficient on its own,
> > there's a separate tool with a whole new set of options - and a
> requirement
> > for a MR runtime! - for checking MOB data? That last one is a rhetorical
> > question (smile), the ops story is... unsatisfying. It's like we've
> taken a
> > self sufficient HBase and bolted in parts of Hive, so now we need MR.
> >
> >
> > On Fri, May 22, 2015 at 1:45 PM, Jonathan Hsieh <jo...@cloudera.com>
> wrote:
> >
> >> In another thread andrew purtell brought up some concerns about the mob
> >> feature:
> >>
> >> On Fri, May 22, 2015 at 12:40 PM, Andrew Purtell <ap...@apache.org>
> >> wrote:
> >>
> >> > Another point of clarification, sorry, I hit the send button too early
> >> it
> >> > seems: I don't believe MOB is fully integrated yet, for example the
> >> > feature
> >> > is an extension to store that lacks support for encryption (this would
> >> > technically be a feature regression); and HBCK. I have not been
> >> following
> >> > MOB too closely so could be mistaken. These issues do not preclude a
> >> merge
> >> > of MOB into trunk, but do preclude a merge back of MOB from trunk to
> >> > branch-1. I would veto the latter until such shortcomings in the
> >> > implementation that could be described as regressions are addressed. I
> >> > would also like to see a performance analysis of a range of workloads
> >> > before and after in as much detail as can be mustered, and would be
> >> happy
> >> > to volunteer to help out with that.
> >> >
> >>
> >> Here's info on the points brought up:
> >>
> >> Encryption support shortcoming is being addrsessed here:
> >> https://issues.apache.org/jira/browse/HBASE-13693 (closed)
> >> https://issues.apache.org/jira/browse/HBASE-13720 (in review)
> >>
> >> Hbck has been actually run against the integration test rigs while the
> >> feature has been enabled but currently has no explicit unit test or
> simple
> >> to run integration test. It currently doesn't report anything special
> >> about the mob storage area. We can add unit tests that cover hbck when
> the
> >> mob path is exercised.
> >>
> >> Another suggestion was a tool to check that mob references had
> >> corresponding mob data. We currently include a mr-based sweeper job
> that
> >> could be used to perform this verification. We can add this tool and
> >> testing for the tool.
> >>
> >> I've done some performance testing and Jingcheng and his colleagues have
> >> done significant amounts of performance testing. We currently have a
> blog
> >> post in progress that will share the results of this performance
> testing.
> >>
> >> Jon.
> >>
> >>
> >>
> >>
> >>
> >> On Wed, May 20, 2015 at 7:38 PM, Ted Yu <yu...@gmail.com> wrote:
> >>
> >> > This is a useful feature, Jon.
> >> >
> >> > I went over the mega-patch and left some comments on review board.
> >> >
> >> > I noticed that hbck was not included in the patch. Neither did I find
> a
> >> > sub-task of HBASE-11339 that covers hbck.
> >> >
> >> > Do you or Jingcheng plan to add MOB-aware capability for hbck ?
> >> >
> >> > Cheers
> >> >
> >> > On Wed, May 20, 2015 at 9:21 AM, Jonathan Hsieh <jo...@cloudera.com>
> >> wrote:
> >> >
> >> > > Hi folks,
> >> > >
> >> > > The Medium Object (MOB) Storage feature (HBASE-11339[1]) is modified
> >> I/O
> >> > > and compaction path that allows individual moderately sized values
> >> > > (10k-10MB) to be stored so that write amplification is reduced when
> >> > > compared to the normal I/O path. At a high level, it provides
> >> alternate
> >> > > flush and compaction mechanisms that segregates large cells into a
> >> > separate
> >> > > area where they are not subject to potentially frequent compaction
> and
> >> > > splits that can be encountered in the normal I/O path. A more
> detailed
> >> > > design doc can be found on the hbase-11339 jira.
> >> > >
> >> > > Jingcheng Du has been working on the mob feature for a while and
> >> Anoop,
> >> > Ram
> >> > > and I have been shepherding him through the design revisions and
> >> > > implementation of the feature in the hbase-11339 branch.[2]
> >> > >
> >> > > The branch we are proposing to merge into master is compatible with
> >> > HBase's
> >> > > core functionality including snapshots, replication, shell support,
> >> > behaves
> >> > > well with table alters, bulk loads and does not require external MR
> >> > > processes. It has been documented, and subject to many integration
> >> test
> >> > > runs (ITBLL, ITAcidGuarantees, ITIngest) including fault injection.
> >> > > Performance testing of the feature shows what can be a 2x-3x
> >> throughput
> >> > > improvement for workloads that contain mobs. These results can be
> >> seen on
> >> > > the hbase 2.0 panel discussion slides from hbasecon (once
> published).
> >> > >
> >> > > Recently there have been some hfile encryption related shortcomings
> >> that
> >> > we
> >> > > could address in branch or in master.
> >> > >
> >> > > Earlier iterations of the feature has been tested in production by
> >> users
> >> > > that Jingcheng has been responsible for. A version has also been
> >> > deployed
> >> > > at users I have been responsible for. Some of the folks from Huawei
> >> > > (ashutosh) have also been submitting the recent encryption bug
> reports
> >> > > against the hbase-11339 branch so there is some evidence of usage by
> >> > them.
> >> > >
> >> > > The four of us (Jingcheng, Ram, Anoop and I) are satisfied with the
> >> > > feature and feel it is a good time to call a merge vote. Ive
> posted a
> >> > > megapatch version for folks who want to peruse the code. [3]
> >> > >
> >> > > What do you all think?
> >> > >
> >> > > Thanks,
> >> > > Jingcheng, Jon, Ram, and Anoop.
> >> > >
> >> > > [1] https://issues.apache.org/jira/browse/HBASE-11339
> >> > > [2] https://github.com/apache/hbase/tree/hbase-11339
> >> > > [3] https://reviews.apache.org/r/34475/
> >> > > --
> >> > > // Jonathan Hsieh (shay)
> >> > > // HBase Tech Lead, Software Engineer, Cloudera
> >> > > // jon@cloudera.com // @jmhsieh
> >> > >
> >> >
> >>
> >>
> >>
> >> --
> >> // Jonathan Hsieh (shay)
> >> // HBase Tech Lead, Software Engineer, Cloudera
> >> // jon@cloudera.com // @jmhsieh
> >>
> >
>
Re: [DISCUSSION] Merge of the hbase-11339 mob branch into master.
Posted by Andrew Purtell <ap...@apache.org>.
Maybe we can remove the dependency on a MR runtime for MOB maintenance by
reimplementing those parallel tasks using Procedure V2? We wouldn't be
looking at MOB for 1.2 but maybe 1.3? I'm also not sure the community as a
whole has the necessary bandwidth for perf and stability testing of MOB in
the 1.2 timeframe, but 1.3 would be more likely.
On Sat, May 23, 2015 at 9:40 AM, Andrew Purtell <ap...@apache.org> wrote:
> Regarding performance testing: Whatever has been done on the MOB branch
> will be interesting data points, and, potentially encouraging, but porting
> to branch-1 will produce a new code base. Earlier results on other code
> will not be applicable. We have to start over. Like I said elsewhere, I'm
> happy to help with (re)characterizing the perf impact and improvements
> produced by the changes.
>
> What coverage do we have for verifying the integrity of MOB references?
> Will the sweep tool detect, alert on, and optionally repair dangling
> references? (I could answer this for myself by looking at MOB branch, but
> hopefully someone here has an answer at the ready.) I assume we calculate
> and store checksums for MOB data itself so we know if values are corrupt.
> Does the sweep tool detect MOB value corruption? Can it be repaired? Do we
> have a good ops story for why HBCK is no longer sufficient on its own,
> there's a separate tool with a whole new set of options - and a requirement
> for a MR runtime! - for checking MOB data? That last one is a rhetorical
> question (smile), the ops story is... unsatisfying. It's like we've taken a
> self sufficient HBase and bolted in parts of Hive, so now we need MR.
>
>
> On Fri, May 22, 2015 at 1:45 PM, Jonathan Hsieh <jo...@cloudera.com> wrote:
>
>> In another thread andrew purtell brought up some concerns about the mob
>> feature:
>>
>> On Fri, May 22, 2015 at 12:40 PM, Andrew Purtell <ap...@apache.org>
>> wrote:
>>
>> > Another point of clarification, sorry, I hit the send button too early
>> it
>> > seems: I don't believe MOB is fully integrated yet, for example the
>> > feature
>> > is an extension to store that lacks support for encryption (this would
>> > technically be a feature regression); and HBCK. I have not been
>> following
>> > MOB too closely so could be mistaken. These issues do not preclude a
>> merge
>> > of MOB into trunk, but do preclude a merge back of MOB from trunk to
>> > branch-1. I would veto the latter until such shortcomings in the
>> > implementation that could be described as regressions are addressed. I
>> > would also like to see a performance analysis of a range of workloads
>> > before and after in as much detail as can be mustered, and would be
>> happy
>> > to volunteer to help out with that.
>> >
>>
>> Here's info on the points brought up:
>>
>> Encryption support shortcoming is being addrsessed here:
>> https://issues.apache.org/jira/browse/HBASE-13693 (closed)
>> https://issues.apache.org/jira/browse/HBASE-13720 (in review)
>>
>> Hbck has been actually run against the integration test rigs while the
>> feature has been enabled but currently has no explicit unit test or simple
>> to run integration test. It currently doesn't report anything special
>> about the mob storage area. We can add unit tests that cover hbck when the
>> mob path is exercised.
>>
>> Another suggestion was a tool to check that mob references had
>> corresponding mob data. We currently include a mr-based sweeper job that
>> could be used to perform this verification. We can add this tool and
>> testing for the tool.
>>
>> I've done some performance testing and Jingcheng and his colleagues have
>> done significant amounts of performance testing. We currently have a blog
>> post in progress that will share the results of this performance testing.
>>
>> Jon.
>>
>>
>>
>>
>>
>> On Wed, May 20, 2015 at 7:38 PM, Ted Yu <yu...@gmail.com> wrote:
>>
>> > This is a useful feature, Jon.
>> >
>> > I went over the mega-patch and left some comments on review board.
>> >
>> > I noticed that hbck was not included in the patch. Neither did I find a
>> > sub-task of HBASE-11339 that covers hbck.
>> >
>> > Do you or Jingcheng plan to add MOB-aware capability for hbck ?
>> >
>> > Cheers
>> >
>> > On Wed, May 20, 2015 at 9:21 AM, Jonathan Hsieh <jo...@cloudera.com>
>> wrote:
>> >
>> > > Hi folks,
>> > >
>> > > The Medium Object (MOB) Storage feature (HBASE-11339[1]) is modified
>> I/O
>> > > and compaction path that allows individual moderately sized values
>> > > (10k-10MB) to be stored so that write amplification is reduced when
>> > > compared to the normal I/O path. At a high level, it provides
>> alternate
>> > > flush and compaction mechanisms that segregates large cells into a
>> > separate
>> > > area where they are not subject to potentially frequent compaction and
>> > > splits that can be encountered in the normal I/O path. A more detailed
>> > > design doc can be found on the hbase-11339 jira.
>> > >
>> > > Jingcheng Du has been working on the mob feature for a while and
>> Anoop,
>> > Ram
>> > > and I have been shepherding him through the design revisions and
>> > > implementation of the feature in the hbase-11339 branch.[2]
>> > >
>> > > The branch we are proposing to merge into master is compatible with
>> > HBase's
>> > > core functionality including snapshots, replication, shell support,
>> > behaves
>> > > well with table alters, bulk loads and does not require external MR
>> > > processes. It has been documented, and subject to many integration
>> test
>> > > runs (ITBLL, ITAcidGuarantees, ITIngest) including fault injection.
>> > > Performance testing of the feature shows what can be a 2x-3x
>> throughput
>> > > improvement for workloads that contain mobs. These results can be
>> seen on
>> > > the hbase 2.0 panel discussion slides from hbasecon (once published).
>> > >
>> > > Recently there have been some hfile encryption related shortcomings
>> that
>> > we
>> > > could address in branch or in master.
>> > >
>> > > Earlier iterations of the feature has been tested in production by
>> users
>> > > that Jingcheng has been responsible for. A version has also been
>> > deployed
>> > > at users I have been responsible for. Some of the folks from Huawei
>> > > (ashutosh) have also been submitting the recent encryption bug reports
>> > > against the hbase-11339 branch so there is some evidence of usage by
>> > them.
>> > >
>> > > The four of us (Jingcheng, Ram, Anoop and I) are satisfied with the
>> > > feature and feel it is a good time to call a merge vote. Ive posted a
>> > > megapatch version for folks who want to peruse the code. [3]
>> > >
>> > > What do you all think?
>> > >
>> > > Thanks,
>> > > Jingcheng, Jon, Ram, and Anoop.
>> > >
>> > > [1] https://issues.apache.org/jira/browse/HBASE-11339
>> > > [2] https://github.com/apache/hbase/tree/hbase-11339
>> > > [3] https://reviews.apache.org/r/34475/
>> > > --
>> > > // Jonathan Hsieh (shay)
>> > > // HBase Tech Lead, Software Engineer, Cloudera
>> > > // jon@cloudera.com // @jmhsieh
>> > >
>> >
>>
>>
>>
>> --
>> // Jonathan Hsieh (shay)
>> // HBase Tech Lead, Software Engineer, Cloudera
>> // jon@cloudera.com // @jmhsieh
>>
>
Re: [DISCUSSION] Merge of the hbase-11339 mob branch into master.
Posted by Andrew Purtell <ap...@apache.org>.
Regarding performance testing: Whatever has been done on the MOB branch
will be interesting data points, and, potentially encouraging, but porting
to branch-1 will produce a new code base. Earlier results on other code
will not be applicable. We have to start over. Like I said elsewhere, I'm
happy to help with (re)characterizing the perf impact and improvements
produced by the changes.
What coverage do we have for verifying the integrity of MOB references?
Will the sweep tool detect, alert on, and optionally repair dangling
references? (I could answer this for myself by looking at MOB branch, but
hopefully someone here has an answer at the ready.) I assume we calculate
and store checksums for MOB data itself so we know if values are corrupt.
Does the sweep tool detect MOB value corruption? Can it be repaired? Do we
have a good ops story for why HBCK is no longer sufficient on its own,
there's a separate tool with a whole new set of options - and a requirement
for a MR runtime! - for checking MOB data? That last one is a rhetorical
question (smile), the ops story is... unsatisfying. It's like we've taken a
self sufficient HBase and bolted in parts of Hive, so now we need MR.
On Fri, May 22, 2015 at 1:45 PM, Jonathan Hsieh <jo...@cloudera.com> wrote:
> In another thread andrew purtell brought up some concerns about the mob
> feature:
>
> On Fri, May 22, 2015 at 12:40 PM, Andrew Purtell <ap...@apache.org>
> wrote:
>
> > Another point of clarification, sorry, I hit the send button too early it
> > seems: I don't believe MOB is fully integrated yet, for example the
> > feature
> > is an extension to store that lacks support for encryption (this would
> > technically be a feature regression); and HBCK. I have not been following
> > MOB too closely so could be mistaken. These issues do not preclude a
> merge
> > of MOB into trunk, but do preclude a merge back of MOB from trunk to
> > branch-1. I would veto the latter until such shortcomings in the
> > implementation that could be described as regressions are addressed. I
> > would also like to see a performance analysis of a range of workloads
> > before and after in as much detail as can be mustered, and would be happy
> > to volunteer to help out with that.
> >
>
> Here's info on the points brought up:
>
> Encryption support shortcoming is being addrsessed here:
> https://issues.apache.org/jira/browse/HBASE-13693 (closed)
> https://issues.apache.org/jira/browse/HBASE-13720 (in review)
>
> Hbck has been actually run against the integration test rigs while the
> feature has been enabled but currently has no explicit unit test or simple
> to run integration test. It currently doesn't report anything special
> about the mob storage area. We can add unit tests that cover hbck when the
> mob path is exercised.
>
> Another suggestion was a tool to check that mob references had
> corresponding mob data. We currently include a mr-based sweeper job that
> could be used to perform this verification. We can add this tool and
> testing for the tool.
>
> I've done some performance testing and Jingcheng and his colleagues have
> done significant amounts of performance testing. We currently have a blog
> post in progress that will share the results of this performance testing.
>
> Jon.
>
>
>
>
>
> On Wed, May 20, 2015 at 7:38 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > This is a useful feature, Jon.
> >
> > I went over the mega-patch and left some comments on review board.
> >
> > I noticed that hbck was not included in the patch. Neither did I find a
> > sub-task of HBASE-11339 that covers hbck.
> >
> > Do you or Jingcheng plan to add MOB-aware capability for hbck ?
> >
> > Cheers
> >
> > On Wed, May 20, 2015 at 9:21 AM, Jonathan Hsieh <jo...@cloudera.com>
> wrote:
> >
> > > Hi folks,
> > >
> > > The Medium Object (MOB) Storage feature (HBASE-11339[1]) is modified
> I/O
> > > and compaction path that allows individual moderately sized values
> > > (10k-10MB) to be stored so that write amplification is reduced when
> > > compared to the normal I/O path. At a high level, it provides
> alternate
> > > flush and compaction mechanisms that segregates large cells into a
> > separate
> > > area where they are not subject to potentially frequent compaction and
> > > splits that can be encountered in the normal I/O path. A more detailed
> > > design doc can be found on the hbase-11339 jira.
> > >
> > > Jingcheng Du has been working on the mob feature for a while and Anoop,
> > Ram
> > > and I have been shepherding him through the design revisions and
> > > implementation of the feature in the hbase-11339 branch.[2]
> > >
> > > The branch we are proposing to merge into master is compatible with
> > HBase's
> > > core functionality including snapshots, replication, shell support,
> > behaves
> > > well with table alters, bulk loads and does not require external MR
> > > processes. It has been documented, and subject to many integration test
> > > runs (ITBLL, ITAcidGuarantees, ITIngest) including fault injection.
> > > Performance testing of the feature shows what can be a 2x-3x throughput
> > > improvement for workloads that contain mobs. These results can be seen
> on
> > > the hbase 2.0 panel discussion slides from hbasecon (once published).
> > >
> > > Recently there have been some hfile encryption related shortcomings
> that
> > we
> > > could address in branch or in master.
> > >
> > > Earlier iterations of the feature has been tested in production by
> users
> > > that Jingcheng has been responsible for. A version has also been
> > deployed
> > > at users I have been responsible for. Some of the folks from Huawei
> > > (ashutosh) have also been submitting the recent encryption bug reports
> > > against the hbase-11339 branch so there is some evidence of usage by
> > them.
> > >
> > > The four of us (Jingcheng, Ram, Anoop and I) are satisfied with the
> > > feature and feel it is a good time to call a merge vote. Ive posted a
> > > megapatch version for folks who want to peruse the code. [3]
> > >
> > > What do you all think?
> > >
> > > Thanks,
> > > Jingcheng, Jon, Ram, and Anoop.
> > >
> > > [1] https://issues.apache.org/jira/browse/HBASE-11339
> > > [2] https://github.com/apache/hbase/tree/hbase-11339
> > > [3] https://reviews.apache.org/r/34475/
> > > --
> > > // Jonathan Hsieh (shay)
> > > // HBase Tech Lead, Software Engineer, Cloudera
> > > // jon@cloudera.com // @jmhsieh
> > >
> >
>
>
>
> --
> // Jonathan Hsieh (shay)
> // HBase Tech Lead, Software Engineer, Cloudera
> // jon@cloudera.com // @jmhsieh
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: [DISCUSSION] Merge of the hbase-11339 mob branch into master.
Posted by Jonathan Hsieh <jo...@cloudera.com>.
In another thread andrew purtell brought up some concerns about the mob
feature:
On Fri, May 22, 2015 at 12:40 PM, Andrew Purtell <ap...@apache.org>
wrote:
> Another point of clarification, sorry, I hit the send button too early it
> seems: I don't believe MOB is fully integrated yet, for example the
> feature
> is an extension to store that lacks support for encryption (this would
> technically be a feature regression); and HBCK. I have not been following
> MOB too closely so could be mistaken. These issues do not preclude a merge
> of MOB into trunk, but do preclude a merge back of MOB from trunk to
> branch-1. I would veto the latter until such shortcomings in the
> implementation that could be described as regressions are addressed. I
> would also like to see a performance analysis of a range of workloads
> before and after in as much detail as can be mustered, and would be happy
> to volunteer to help out with that.
>
Here's info on the points brought up:
Encryption support shortcoming is being addrsessed here:
https://issues.apache.org/jira/browse/HBASE-13693 (closed)
https://issues.apache.org/jira/browse/HBASE-13720 (in review)
Hbck has been actually run against the integration test rigs while the
feature has been enabled but currently has no explicit unit test or simple
to run integration test. It currently doesn't report anything special
about the mob storage area. We can add unit tests that cover hbck when the
mob path is exercised.
Another suggestion was a tool to check that mob references had
corresponding mob data. We currently include a mr-based sweeper job that
could be used to perform this verification. We can add this tool and
testing for the tool.
I've done some performance testing and Jingcheng and his colleagues have
done significant amounts of performance testing. We currently have a blog
post in progress that will share the results of this performance testing.
Jon.
On Wed, May 20, 2015 at 7:38 PM, Ted Yu <yu...@gmail.com> wrote:
> This is a useful feature, Jon.
>
> I went over the mega-patch and left some comments on review board.
>
> I noticed that hbck was not included in the patch. Neither did I find a
> sub-task of HBASE-11339 that covers hbck.
>
> Do you or Jingcheng plan to add MOB-aware capability for hbck ?
>
> Cheers
>
> On Wed, May 20, 2015 at 9:21 AM, Jonathan Hsieh <jo...@cloudera.com> wrote:
>
> > Hi folks,
> >
> > The Medium Object (MOB) Storage feature (HBASE-11339[1]) is modified I/O
> > and compaction path that allows individual moderately sized values
> > (10k-10MB) to be stored so that write amplification is reduced when
> > compared to the normal I/O path. At a high level, it provides alternate
> > flush and compaction mechanisms that segregates large cells into a
> separate
> > area where they are not subject to potentially frequent compaction and
> > splits that can be encountered in the normal I/O path. A more detailed
> > design doc can be found on the hbase-11339 jira.
> >
> > Jingcheng Du has been working on the mob feature for a while and Anoop,
> Ram
> > and I have been shepherding him through the design revisions and
> > implementation of the feature in the hbase-11339 branch.[2]
> >
> > The branch we are proposing to merge into master is compatible with
> HBase's
> > core functionality including snapshots, replication, shell support,
> behaves
> > well with table alters, bulk loads and does not require external MR
> > processes. It has been documented, and subject to many integration test
> > runs (ITBLL, ITAcidGuarantees, ITIngest) including fault injection.
> > Performance testing of the feature shows what can be a 2x-3x throughput
> > improvement for workloads that contain mobs. These results can be seen on
> > the hbase 2.0 panel discussion slides from hbasecon (once published).
> >
> > Recently there have been some hfile encryption related shortcomings that
> we
> > could address in branch or in master.
> >
> > Earlier iterations of the feature has been tested in production by users
> > that Jingcheng has been responsible for. A version has also been
> deployed
> > at users I have been responsible for. Some of the folks from Huawei
> > (ashutosh) have also been submitting the recent encryption bug reports
> > against the hbase-11339 branch so there is some evidence of usage by
> them.
> >
> > The four of us (Jingcheng, Ram, Anoop and I) are satisfied with the
> > feature and feel it is a good time to call a merge vote. Ive posted a
> > megapatch version for folks who want to peruse the code. [3]
> >
> > What do you all think?
> >
> > Thanks,
> > Jingcheng, Jon, Ram, and Anoop.
> >
> > [1] https://issues.apache.org/jira/browse/HBASE-11339
> > [2] https://github.com/apache/hbase/tree/hbase-11339
> > [3] https://reviews.apache.org/r/34475/
> > --
> > // Jonathan Hsieh (shay)
> > // HBase Tech Lead, Software Engineer, Cloudera
> > // jon@cloudera.com // @jmhsieh
> >
>
--
// Jonathan Hsieh (shay)
// HBase Tech Lead, Software Engineer, Cloudera
// jon@cloudera.com // @jmhsieh
Re: [DISCUSSION] Merge of the hbase-11339 mob branch into master.
Posted by Ted Yu <yu...@gmail.com>.
bq. I don't think this is a hard requirement for merge
Agreed.
Can be done post-merge.
On Thu, May 21, 2015 at 5:01 PM, Jonathan Hsieh <jo...@cloudera.com> wrote:
> Comments inline
>
> On Thu, May 21, 2015 at 11:20 AM, Ted Yu <yu...@gmail.com> wrote:
>
> > w.r.t. hbck, the two actions listed below look good.
> >
> > Cheers
> >
> > On Thu, May 21, 2015 at 11:04 AM, Jonathan Hsieh <jo...@cloudera.com>
> wrote:
> >
> > > On Wed, May 20, 2015 at 7:38 PM, Ted Yu <yu...@gmail.com> wrote:
> > >
> > > > This is a useful feature, Jon.
> > > >
> > > > I went over the mega-patch and left some comments on review board.
> > > >
> > > > I noticed that hbck was not included in the patch. Neither did I
> find a
> > > > sub-task of HBASE-11339 that covers hbck.
> > > >
> > > > Do you or Jingcheng plan to add MOB-aware capability for hbck ?
> > > >
> > > > Ted -- what in particular are you thinking about for a mob-aware
> hbck?
> > >
> > > here are a few things I can think of:
> > >
> > > - hbck doesn't berak mob
> >
>
> hbck not getting broken by mob should be a check we do and will be done
> before merge.
>
> > - a full table scan job that verifies all mob refs have values
> > >
> >
> I don't think this is a hard requirement for merge.
>
> wdyt ted?
>
>
> > > --
> > > // Jonathan Hsieh (shay)
> > > // HBase Tech Lead, Software Engineer, Cloudera
> > > // jon@cloudera.com // @jmhsieh
> > >
> >
>
>
>
> --
> // Jonathan Hsieh (shay)
> // HBase Tech Lead, Software Engineer, Cloudera
> // jon@cloudera.com // @jmhsieh
>
Re: [DISCUSSION] Merge of the hbase-11339 mob branch into master.
Posted by Jonathan Hsieh <jo...@cloudera.com>.
Comments inline
On Thu, May 21, 2015 at 11:20 AM, Ted Yu <yu...@gmail.com> wrote:
> w.r.t. hbck, the two actions listed below look good.
>
> Cheers
>
> On Thu, May 21, 2015 at 11:04 AM, Jonathan Hsieh <jo...@cloudera.com> wrote:
>
> > On Wed, May 20, 2015 at 7:38 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > This is a useful feature, Jon.
> > >
> > > I went over the mega-patch and left some comments on review board.
> > >
> > > I noticed that hbck was not included in the patch. Neither did I find a
> > > sub-task of HBASE-11339 that covers hbck.
> > >
> > > Do you or Jingcheng plan to add MOB-aware capability for hbck ?
> > >
> > > Ted -- what in particular are you thinking about for a mob-aware hbck?
> >
> > here are a few things I can think of:
> >
> > - hbck doesn't berak mob
>
hbck not getting broken by mob should be a check we do and will be done
before merge.
> - a full table scan job that verifies all mob refs have values
> >
>
I don't think this is a hard requirement for merge.
wdyt ted?
> > --
> > // Jonathan Hsieh (shay)
> > // HBase Tech Lead, Software Engineer, Cloudera
> > // jon@cloudera.com // @jmhsieh
> >
>
--
// Jonathan Hsieh (shay)
// HBase Tech Lead, Software Engineer, Cloudera
// jon@cloudera.com // @jmhsieh
Re: [DISCUSSION] Merge of the hbase-11339 mob branch into master.
Posted by Ted Yu <yu...@gmail.com>.
w.r.t. hbck, the two actions listed below look good.
Cheers
On Thu, May 21, 2015 at 11:04 AM, Jonathan Hsieh <jo...@cloudera.com> wrote:
> On Wed, May 20, 2015 at 7:38 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > This is a useful feature, Jon.
> >
> > I went over the mega-patch and left some comments on review board.
> >
> > I noticed that hbck was not included in the patch. Neither did I find a
> > sub-task of HBASE-11339 that covers hbck.
> >
> > Do you or Jingcheng plan to add MOB-aware capability for hbck ?
> >
> > Ted -- what in particular are you thinking about for a mob-aware hbck?
>
> here are a few things I can think of:
>
> - hbck doesn't berak mob
> - a full table scan job that verifies all mob refs have values
>
> --
> // Jonathan Hsieh (shay)
> // HBase Tech Lead, Software Engineer, Cloudera
> // jon@cloudera.com // @jmhsieh
>
Re: [DISCUSSION] Merge of the hbase-11339 mob branch into master.
Posted by Jonathan Hsieh <jo...@cloudera.com>.
On Wed, May 20, 2015 at 7:38 PM, Ted Yu <yu...@gmail.com> wrote:
> This is a useful feature, Jon.
>
> I went over the mega-patch and left some comments on review board.
>
> I noticed that hbck was not included in the patch. Neither did I find a
> sub-task of HBASE-11339 that covers hbck.
>
> Do you or Jingcheng plan to add MOB-aware capability for hbck ?
>
> Ted -- what in particular are you thinking about for a mob-aware hbck?
here are a few things I can think of:
- hbck doesn't berak mob
- a full table scan job that verifies all mob refs have values
--
// Jonathan Hsieh (shay)
// HBase Tech Lead, Software Engineer, Cloudera
// jon@cloudera.com // @jmhsieh
Re: [DISCUSSION] Merge of the hbase-11339 mob branch into master.
Posted by Ted Yu <yu...@gmail.com>.
This is a useful feature, Jon.
I went over the mega-patch and left some comments on review board.
I noticed that hbck was not included in the patch. Neither did I find a
sub-task of HBASE-11339 that covers hbck.
Do you or Jingcheng plan to add MOB-aware capability for hbck ?
Cheers
On Wed, May 20, 2015 at 9:21 AM, Jonathan Hsieh <jo...@cloudera.com> wrote:
> Hi folks,
>
> The Medium Object (MOB) Storage feature (HBASE-11339[1]) is modified I/O
> and compaction path that allows individual moderately sized values
> (10k-10MB) to be stored so that write amplification is reduced when
> compared to the normal I/O path. At a high level, it provides alternate
> flush and compaction mechanisms that segregates large cells into a separate
> area where they are not subject to potentially frequent compaction and
> splits that can be encountered in the normal I/O path. A more detailed
> design doc can be found on the hbase-11339 jira.
>
> Jingcheng Du has been working on the mob feature for a while and Anoop, Ram
> and I have been shepherding him through the design revisions and
> implementation of the feature in the hbase-11339 branch.[2]
>
> The branch we are proposing to merge into master is compatible with HBase's
> core functionality including snapshots, replication, shell support, behaves
> well with table alters, bulk loads and does not require external MR
> processes. It has been documented, and subject to many integration test
> runs (ITBLL, ITAcidGuarantees, ITIngest) including fault injection.
> Performance testing of the feature shows what can be a 2x-3x throughput
> improvement for workloads that contain mobs. These results can be seen on
> the hbase 2.0 panel discussion slides from hbasecon (once published).
>
> Recently there have been some hfile encryption related shortcomings that we
> could address in branch or in master.
>
> Earlier iterations of the feature has been tested in production by users
> that Jingcheng has been responsible for. A version has also been deployed
> at users I have been responsible for. Some of the folks from Huawei
> (ashutosh) have also been submitting the recent encryption bug reports
> against the hbase-11339 branch so there is some evidence of usage by them.
>
> The four of us (Jingcheng, Ram, Anoop and I) are satisfied with the
> feature and feel it is a good time to call a merge vote. Ive posted a
> megapatch version for folks who want to peruse the code. [3]
>
> What do you all think?
>
> Thanks,
> Jingcheng, Jon, Ram, and Anoop.
>
> [1] https://issues.apache.org/jira/browse/HBASE-11339
> [2] https://github.com/apache/hbase/tree/hbase-11339
> [3] https://reviews.apache.org/r/34475/
> --
> // Jonathan Hsieh (shay)
> // HBase Tech Lead, Software Engineer, Cloudera
> // jon@cloudera.com // @jmhsieh
>