You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Matteo Bertozzi <mb...@apache.org> on 2016/09/22 20:50:47 UTC
[DISCUSSION] MR jobs started by Master or RS
What's the latest opinion around running MR jobs from hbase (Master or RS)?
I remember in the past that there was discussion about not having MR has
direct dependency of hbase.
I think some of discussion where around MOB that had a MR job to compact,
that later was transformed in a non-MR job to be merged, I think we had a
similar discussion for log split/replay.
the latest is the new Backup feature (HBASE-7912), that runs a MR job from
the master to copy data or restore data.
(backup is also "not really core" as in.. if you don't use backup you'll
not end up running MR jobs, but this was probably true for MOB as in "if
you don't enable MOB you don't need MR")
any thoughts? do we a rule that says "we don't want to have hbase run MR
jobs, only tool started manually by the user can do that". or can we start
adding MR calls around without problems?
Re: [DISCUSSION] MR jobs started by Master or RS
Posted by Andrew Purtell <an...@gmail.com>.
I would be -1 a requirement for MR for something core to HBase.
> On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi <mb...@apache.org> wrote:
>
> What's the latest opinion around running MR jobs from hbase (Master or RS)?
>
> I remember in the past that there was discussion about not having MR has
> direct dependency of hbase.
>
> I think some of discussion where around MOB that had a MR job to compact,
> that later was transformed in a non-MR job to be merged, I think we had a
> similar discussion for log split/replay.
>
> the latest is the new Backup feature (HBASE-7912), that runs a MR job from
> the master to copy data or restore data.
> (backup is also "not really core" as in.. if you don't use backup you'll
> not end up running MR jobs, but this was probably true for MOB as in "if
> you don't enable MOB you don't need MR")
>
> any thoughts? do we a rule that says "we don't want to have hbase run MR
> jobs, only tool started manually by the user can do that". or can we start
> adding MR calls around without problems?
Re: [DISCUSSION] MR jobs started by Master or RS
Posted by Andrew Purtell <an...@gmail.com>.
(Back with a sore throat.)
Also for what it is worth - it may well be that the attempt to bolt containers-as-executors to YARN is too little too late and coordination of container based services and applications (such as distributed map-reduce workflows or more likely Spark) will be handled by the native management tooling we are all using to build container based infrastructure. It's unclear to me how long HBase will live into this new regime but I'd optimistically wager long enough so even having YARN around is a suspect assumption.
> On Sep 22, 2016, at 6:58 PM, Andrew Purtell <an...@gmail.com> wrote:
>
> > We should also do compactions using MR (just saying :)
>
> No, never. It's not a good idea to wed any of our core function to something that independently evolves, that some of us don't have commit rights on (and never will), and has varying degrees of utility depending on deploy. Like JM says in some places not having the MR runtime around is virtuous.
>
> (Runs away screaming.)
>
> On Sep 22, 2016, at 2:49 PM, Devaraj Das <dd...@hortonworks.com> wrote:
>
>> Very much agree; for tools like ExportSnapshot / Backup / Restore, it's fine to be dependent on MR. MR is the right framework for such. We should also do compactions using MR (just saying :) )
>> ________________________________________
>> From: Ted Yu <yu...@gmail.com>
>> Sent: Thursday, September 22, 2016 2:00 PM
>> To: dev@hbase.apache.org
>> Subject: Re: [DISCUSSION] MR jobs started by Master or RS
>>
>> I agree - backup / restore is in the same category as import / export.
>>
>> On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell <an...@gmail.com>
>> wrote:
>>
>>> Backup is extra tooling around core in my opinion. Like import or export.
>>> Or the optional MOB tool. It's fine.
>>>
>>>> On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi <mb...@apache.org>
>>> wrote:
>>>>
>>>> What's the latest opinion around running MR jobs from hbase (Master or
>>> RS)?
>>>>
>>>> I remember in the past that there was discussion about not having MR has
>>>> direct dependency of hbase.
>>>>
>>>> I think some of discussion where around MOB that had a MR job to compact,
>>>> that later was transformed in a non-MR job to be merged, I think we had a
>>>> similar discussion for log split/replay.
>>>>
>>>> the latest is the new Backup feature (HBASE-7912), that runs a MR job
>>> from
>>>> the master to copy data or restore data.
>>>> (backup is also "not really core" as in.. if you don't use backup you'll
>>>> not end up running MR jobs, but this was probably true for MOB as in "if
>>>> you don't enable MOB you don't need MR")
>>>>
>>>> any thoughts? do we a rule that says "we don't want to have hbase run MR
>>>> jobs, only tool started manually by the user can do that". or can we
>>> start
>>>> adding MR calls around without problems?
>>>
Re: [DISCUSSION] MR jobs started by Master or RS
Posted by Andrew Purtell <an...@gmail.com>.
> We should also do compactions using MR (just saying :)
No, never. It's not a good idea to wed any of our core function to something that independently evolves, that some of us don't have commit rights on (and never will), and has varying degrees of utility depending on deploy. Like JM says in some places not having the MR runtime around is virtuous.
(Runs away screaming.)
> On Sep 22, 2016, at 2:49 PM, Devaraj Das <dd...@hortonworks.com> wrote:
>
> Very much agree; for tools like ExportSnapshot / Backup / Restore, it's fine to be dependent on MR. MR is the right framework for such. We should also do compactions using MR (just saying :) )
> ________________________________________
> From: Ted Yu <yu...@gmail.com>
> Sent: Thursday, September 22, 2016 2:00 PM
> To: dev@hbase.apache.org
> Subject: Re: [DISCUSSION] MR jobs started by Master or RS
>
> I agree - backup / restore is in the same category as import / export.
>
> On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell <an...@gmail.com>
> wrote:
>
>> Backup is extra tooling around core in my opinion. Like import or export.
>> Or the optional MOB tool. It's fine.
>>
>>> On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi <mb...@apache.org>
>> wrote:
>>>
>>> What's the latest opinion around running MR jobs from hbase (Master or
>> RS)?
>>>
>>> I remember in the past that there was discussion about not having MR has
>>> direct dependency of hbase.
>>>
>>> I think some of discussion where around MOB that had a MR job to compact,
>>> that later was transformed in a non-MR job to be merged, I think we had a
>>> similar discussion for log split/replay.
>>>
>>> the latest is the new Backup feature (HBASE-7912), that runs a MR job
>> from
>>> the master to copy data or restore data.
>>> (backup is also "not really core" as in.. if you don't use backup you'll
>>> not end up running MR jobs, but this was probably true for MOB as in "if
>>> you don't enable MOB you don't need MR")
>>>
>>> any thoughts? do we a rule that says "we don't want to have hbase run MR
>>> jobs, only tool started manually by the user can do that". or can we
>> start
>>> adding MR calls around without problems?
>>
Re: [DISCUSSION] MR jobs started by Master or RS
Posted by Vladimir Rodionov <vl...@gmail.com>.
I think the rationale behind a decision to move code to Master was a
security/access control. Enis will correct me if I am wrong.
Master does not have direct dependecy on mapreduce - only on backup.
-Vlad
On Thu, Sep 22, 2016 at 3:44 PM, Matteo Bertozzi <th...@gmail.com>
wrote:
> just a remark. my query was not about tools using MR (everyone i think is
> ok with those).
> the topic was about: "are we ok with running MR jobs from Master and RSs
> code?" since this will be the first time we do this
>
> Matteo
>
>
> On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das <dd...@hortonworks.com> wrote:
>
> > Very much agree; for tools like ExportSnapshot / Backup / Restore, it's
> > fine to be dependent on MR. MR is the right framework for such. We should
> > also do compactions using MR (just saying :) )
> > ________________________________________
> > From: Ted Yu <yu...@gmail.com>
> > Sent: Thursday, September 22, 2016 2:00 PM
> > To: dev@hbase.apache.org
> > Subject: Re: [DISCUSSION] MR jobs started by Master or RS
> >
> > I agree - backup / restore is in the same category as import / export.
> >
> > On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell <
> andrew.purtell@gmail.com>
> > wrote:
> >
> > > Backup is extra tooling around core in my opinion. Like import or
> export.
> > > Or the optional MOB tool. It's fine.
> > >
> > > > On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi <mb...@apache.org>
> > > wrote:
> > > >
> > > > What's the latest opinion around running MR jobs from hbase (Master
> or
> > > RS)?
> > > >
> > > > I remember in the past that there was discussion about not having MR
> > has
> > > > direct dependency of hbase.
> > > >
> > > > I think some of discussion where around MOB that had a MR job to
> > compact,
> > > > that later was transformed in a non-MR job to be merged, I think we
> > had a
> > > > similar discussion for log split/replay.
> > > >
> > > > the latest is the new Backup feature (HBASE-7912), that runs a MR job
> > > from
> > > > the master to copy data or restore data.
> > > > (backup is also "not really core" as in.. if you don't use backup
> > you'll
> > > > not end up running MR jobs, but this was probably true for MOB as in
> > "if
> > > > you don't enable MOB you don't need MR")
> > > >
> > > > any thoughts? do we a rule that says "we don't want to have hbase run
> > MR
> > > > jobs, only tool started manually by the user can do that". or can we
> > > start
> > > > adding MR calls around without problems?
> > >
> >
>
Re: [DISCUSSION] MR jobs started by Master or RS
Posted by Devaraj Das <dd...@hortonworks.com>.
Matteo, the Master won't spawn the job unless someone actually wants to use the backup/restore. So I'd argue we still don't have a 'hard' dependency - it's still much like the other tools that you consider as being outside the core.
________________________________________
From: Matteo Bertozzi <th...@gmail.com>
Sent: Thursday, September 22, 2016 3:44 PM
To: dev@hbase.apache.org
Subject: Re: [DISCUSSION] MR jobs started by Master or RS
just a remark. my query was not about tools using MR (everyone i think is
ok with those).
the topic was about: "are we ok with running MR jobs from Master and RSs
code?" since this will be the first time we do this
Matteo
On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das <dd...@hortonworks.com> wrote:
> Very much agree; for tools like ExportSnapshot / Backup / Restore, it's
> fine to be dependent on MR. MR is the right framework for such. We should
> also do compactions using MR (just saying :) )
> ________________________________________
> From: Ted Yu <yu...@gmail.com>
> Sent: Thursday, September 22, 2016 2:00 PM
> To: dev@hbase.apache.org
> Subject: Re: [DISCUSSION] MR jobs started by Master or RS
>
> I agree - backup / restore is in the same category as import / export.
>
> On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell <an...@gmail.com>
> wrote:
>
> > Backup is extra tooling around core in my opinion. Like import or export.
> > Or the optional MOB tool. It's fine.
> >
> > > On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi <mb...@apache.org>
> > wrote:
> > >
> > > What's the latest opinion around running MR jobs from hbase (Master or
> > RS)?
> > >
> > > I remember in the past that there was discussion about not having MR
> has
> > > direct dependency of hbase.
> > >
> > > I think some of discussion where around MOB that had a MR job to
> compact,
> > > that later was transformed in a non-MR job to be merged, I think we
> had a
> > > similar discussion for log split/replay.
> > >
> > > the latest is the new Backup feature (HBASE-7912), that runs a MR job
> > from
> > > the master to copy data or restore data.
> > > (backup is also "not really core" as in.. if you don't use backup
> you'll
> > > not end up running MR jobs, but this was probably true for MOB as in
> "if
> > > you don't enable MOB you don't need MR")
> > >
> > > any thoughts? do we a rule that says "we don't want to have hbase run
> MR
> > > jobs, only tool started manually by the user can do that". or can we
> > start
> > > adding MR calls around without problems?
> >
>
Re: [DISCUSSION] MR jobs started by Master or RS
Posted by Vladimir Rodionov <vl...@gmail.com>.
>> If MR is not strong dependency for Master/RS, it is OK for me.
There is no strong MR dependency for Master/RS. They will function as
usual, until you try backup,
it will fail but Master won't.
-Vlad
On Thu, Sep 22, 2016 at 8:03 PM, Vladimir Rodionov <vl...@gmail.com>
wrote:
> >> No, never.
>
> No need for M/R here, just a simple compaction-server colocated with RS on
> a same node.
> You save a lot on GC in RS. Ideally, it can be IO "nice" in Linux (by
> setting IO priority). But offtopic, of course :)
>
> -Vlad
>
> On Thu, Sep 22, 2016 at 7:57 PM, Vladimir Rodionov <vladrodionov@gmail.com
> > wrote:
>
>> >> And if MR not deployed, Backup/Restore feature could not be used,
>> right?
>>
>> Yes.
>>
>> On Thu, Sep 22, 2016 at 7:53 PM, Heng Chen <he...@gmail.com>
>> wrote:
>>
>>> {quote}
>>> If MR framework is not deployed in the cluster, hbase still functions
>>> normally (post merge).
>>> {quote}
>>>
>>> If MR is not strong dependency for Master/RS, it is OK for me.
>>> And if MR not deployed, Backup/Restore feature could not be used, right?
>>>
>>> 2016-09-23 10:49 GMT+08:00 Ted Yu <yu...@gmail.com>:
>>> > If MR framework is not deployed in the cluster, hbase still functions
>>> > normally (post merge).
>>> >
>>> > In terms of build time dependency, we have long been depending on
>>> > mapreduce. Take a look at ExportSnapshot.
>>> >
>>> > Cheers
>>> >
>>> > On Thu, Sep 22, 2016 at 7:42 PM, Heng Chen <he...@gmail.com>
>>> wrote:
>>> >
>>> >> In our production cluster, it is a common case we just have HDFS and
>>> >> HBase deployed.
>>> >> If our Master/RS depend on MR framework (especially some features we
>>> >> have not used at all), it introduced another cost for maintain. I
>>> >> don't think it is a good idea.
>>> >>
>>> >> 2016-09-23 10:28 GMT+08:00 张铎 <pa...@gmail.com>:
>>> >> > To be specific, for example, our nice Backup/Restore feature, if we
>>> think
>>> >> > this is not a core feature of HBase, then we could make it depend
>>> on MR,
>>> >> > and start a standalone BackupManager instance that submits MR jobs
>>> to do
>>> >> > periodical maintenance job. And if we think this is a core feature
>>> that
>>> >> > everyone should use it, then we'd better implement it without MR
>>> >> > dependency, like DLS.
>>> >> >
>>> >> > Thanks.
>>> >> >
>>> >> > 2016-09-23 10:11 GMT+08:00 张铎 <pa...@gmail.com>:
>>> >> >
>>> >> >> I‘m -1 on let master or rs launch MR jobs. It is OK that some of
>>> our
>>> >> >> features depend on MR but I think the bottom line is that we should
>>> >> launch
>>> >> >> the jobs from outside manually or by other services.
>>> >> >>
>>> >> >> 2016-09-23 9:47 GMT+08:00 Andrew Purtell <andrew.purtell@gmail.com
>>> >:
>>> >> >>
>>> >> >>> Ok, got it. Well "shelling out" is on the line I think, so a fair
>>> >> >>> question.
>>> >> >>>
>>> >> >>> Can this be driven by a utility derived from Tool like our other
>>> MR
>>> >> apps?
>>> >> >>> The issue is needing the AccessController to decide if allowed?
>>> But
>>> >> nothing
>>> >> >>> prevents the user from running the job manually/independently,
>>> right?
>>> >> >>>
>>> >> >>> > On Sep 22, 2016, at 3:44 PM, Matteo Bertozzi <
>>> >> theo.bertozzi@gmail.com>
>>> >> >>> wrote:
>>> >> >>> >
>>> >> >>> > just a remark. my query was not about tools using MR (everyone i
>>> >> think
>>> >> >>> is
>>> >> >>> > ok with those).
>>> >> >>> > the topic was about: "are we ok with running MR jobs from
>>> Master and
>>> >> RSs
>>> >> >>> > code?" since this will be the first time we do this
>>> >> >>> >
>>> >> >>> > Matteo
>>> >> >>> >
>>> >> >>> >
>>> >> >>> >> On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das <
>>> ddas@hortonworks.com>
>>> >> >>> wrote:
>>> >> >>> >>
>>> >> >>> >> Very much agree; for tools like ExportSnapshot / Backup /
>>> Restore,
>>> >> it's
>>> >> >>> >> fine to be dependent on MR. MR is the right framework for
>>> such. We
>>> >> >>> should
>>> >> >>> >> also do compactions using MR (just saying :) )
>>> >> >>> >> ________________________________________
>>> >> >>> >> From: Ted Yu <yu...@gmail.com>
>>> >> >>> >> Sent: Thursday, September 22, 2016 2:00 PM
>>> >> >>> >> To: dev@hbase.apache.org
>>> >> >>> >> Subject: Re: [DISCUSSION] MR jobs started by Master or RS
>>> >> >>> >>
>>> >> >>> >> I agree - backup / restore is in the same category as import /
>>> >> export.
>>> >> >>> >>
>>> >> >>> >> On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell <
>>> >> >>> andrew.purtell@gmail.com>
>>> >> >>> >> wrote:
>>> >> >>> >>
>>> >> >>> >>> Backup is extra tooling around core in my opinion. Like
>>> import or
>>> >> >>> export.
>>> >> >>> >>> Or the optional MOB tool. It's fine.
>>> >> >>> >>>
>>> >> >>> >>>> On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi <
>>> >> mbertozzi@apache.org>
>>> >> >>> >>> wrote:
>>> >> >>> >>>>
>>> >> >>> >>>> What's the latest opinion around running MR jobs from hbase
>>> >> (Master
>>> >> >>> or
>>> >> >>> >>> RS)?
>>> >> >>> >>>>
>>> >> >>> >>>> I remember in the past that there was discussion about not
>>> having
>>> >> MR
>>> >> >>> >> has
>>> >> >>> >>>> direct dependency of hbase.
>>> >> >>> >>>>
>>> >> >>> >>>> I think some of discussion where around MOB that had a MR
>>> job to
>>> >> >>> >> compact,
>>> >> >>> >>>> that later was transformed in a non-MR job to be merged, I
>>> think
>>> >> we
>>> >> >>> >> had a
>>> >> >>> >>>> similar discussion for log split/replay.
>>> >> >>> >>>>
>>> >> >>> >>>> the latest is the new Backup feature (HBASE-7912), that runs
>>> a MR
>>> >> job
>>> >> >>> >>> from
>>> >> >>> >>>> the master to copy data or restore data.
>>> >> >>> >>>> (backup is also "not really core" as in.. if you don't use
>>> backup
>>> >> >>> >> you'll
>>> >> >>> >>>> not end up running MR jobs, but this was probably true for
>>> MOB as
>>> >> in
>>> >> >>> >> "if
>>> >> >>> >>>> you don't enable MOB you don't need MR")
>>> >> >>> >>>>
>>> >> >>> >>>> any thoughts? do we a rule that says "we don't want to have
>>> hbase
>>> >> run
>>> >> >>> >> MR
>>> >> >>> >>>> jobs, only tool started manually by the user can do that".
>>> or can
>>> >> we
>>> >> >>> >>> start
>>> >> >>> >>>> adding MR calls around without problems?
>>> >> >>> >>>
>>> >> >>> >>
>>> >> >>>
>>> >> >>
>>> >> >>
>>> >>
>>>
>>
>>
>
Re: [DISCUSSION] MR jobs started by Master or RS
Posted by Andrew Purtell <an...@gmail.com>.
Agreed, this would be interesting to contemplate.
On Sep 22, 2016, at 8:03 PM, Vladimir Rodionov <vl...@gmail.com> wrote:
>>> No, never.
>
> No need for M/R here, just a simple compaction-server colocated with RS on
> a same node.
> You save a lot on GC in RS. Ideally, it can be IO "nice" in Linux (by
> setting IO priority). But offtopic, of course :)
>
> -Vlad
>
> On Thu, Sep 22, 2016 at 7:57 PM, Vladimir Rodionov <vl...@gmail.com>
> wrote:
>
>>>> And if MR not deployed, Backup/Restore feature could not be used,
>> right?
>>
>> Yes.
>>
>> On Thu, Sep 22, 2016 at 7:53 PM, Heng Chen <he...@gmail.com>
>> wrote:
>>
>>> {quote}
>>> If MR framework is not deployed in the cluster, hbase still functions
>>> normally (post merge).
>>> {quote}
>>>
>>> If MR is not strong dependency for Master/RS, it is OK for me.
>>> And if MR not deployed, Backup/Restore feature could not be used, right?
>>>
>>> 2016-09-23 10:49 GMT+08:00 Ted Yu <yu...@gmail.com>:
>>>> If MR framework is not deployed in the cluster, hbase still functions
>>>> normally (post merge).
>>>>
>>>> In terms of build time dependency, we have long been depending on
>>>> mapreduce. Take a look at ExportSnapshot.
>>>>
>>>> Cheers
>>>>
>>>> On Thu, Sep 22, 2016 at 7:42 PM, Heng Chen <he...@gmail.com>
>>> wrote:
>>>>
>>>>> In our production cluster, it is a common case we just have HDFS and
>>>>> HBase deployed.
>>>>> If our Master/RS depend on MR framework (especially some features we
>>>>> have not used at all), it introduced another cost for maintain. I
>>>>> don't think it is a good idea.
>>>>>
>>>>> 2016-09-23 10:28 GMT+08:00 张铎 <pa...@gmail.com>:
>>>>>> To be specific, for example, our nice Backup/Restore feature, if we
>>> think
>>>>>> this is not a core feature of HBase, then we could make it depend on
>>> MR,
>>>>>> and start a standalone BackupManager instance that submits MR jobs
>>> to do
>>>>>> periodical maintenance job. And if we think this is a core feature
>>> that
>>>>>> everyone should use it, then we'd better implement it without MR
>>>>>> dependency, like DLS.
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>> 2016-09-23 10:11 GMT+08:00 张铎 <pa...@gmail.com>:
>>>>>>
>>>>>>> I‘m -1 on let master or rs launch MR jobs. It is OK that some of our
>>>>>>> features depend on MR but I think the bottom line is that we should
>>>>> launch
>>>>>>> the jobs from outside manually or by other services.
>>>>>>>
>>>>>>> 2016-09-23 9:47 GMT+08:00 Andrew Purtell <andrew.purtell@gmail.com
>>>> :
>>>>>>>
>>>>>>>> Ok, got it. Well "shelling out" is on the line I think, so a fair
>>>>>>>> question.
>>>>>>>>
>>>>>>>> Can this be driven by a utility derived from Tool like our other MR
>>>>> apps?
>>>>>>>> The issue is needing the AccessController to decide if allowed? But
>>>>> nothing
>>>>>>>> prevents the user from running the job manually/independently,
>>> right?
>>>>>>>>
>>>>>>>>> On Sep 22, 2016, at 3:44 PM, Matteo Bertozzi <
>>>>> theo.bertozzi@gmail.com>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> just a remark. my query was not about tools using MR (everyone i
>>>>> think
>>>>>>>> is
>>>>>>>>> ok with those).
>>>>>>>>> the topic was about: "are we ok with running MR jobs from Master
>>> and
>>>>> RSs
>>>>>>>>> code?" since this will be the first time we do this
>>>>>>>>>
>>>>>>>>> Matteo
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das <
>>> ddas@hortonworks.com>
>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Very much agree; for tools like ExportSnapshot / Backup /
>>> Restore,
>>>>> it's
>>>>>>>>>> fine to be dependent on MR. MR is the right framework for such.
>>> We
>>>>>>>> should
>>>>>>>>>> also do compactions using MR (just saying :) )
>>>>>>>>>> ________________________________________
>>>>>>>>>> From: Ted Yu <yu...@gmail.com>
>>>>>>>>>> Sent: Thursday, September 22, 2016 2:00 PM
>>>>>>>>>> To: dev@hbase.apache.org
>>>>>>>>>> Subject: Re: [DISCUSSION] MR jobs started by Master or RS
>>>>>>>>>>
>>>>>>>>>> I agree - backup / restore is in the same category as import /
>>>>> export.
>>>>>>>>>>
>>>>>>>>>> On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell <
>>>>>>>> andrew.purtell@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Backup is extra tooling around core in my opinion. Like import
>>> or
>>>>>>>> export.
>>>>>>>>>>> Or the optional MOB tool. It's fine.
>>>>>>>>>>>
>>>>>>>>>>>> On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi <
>>>>> mbertozzi@apache.org>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> What's the latest opinion around running MR jobs from hbase
>>>>> (Master
>>>>>>>> or
>>>>>>>>>>> RS)?
>>>>>>>>>>>>
>>>>>>>>>>>> I remember in the past that there was discussion about not
>>> having
>>>>> MR
>>>>>>>>>> has
>>>>>>>>>>>> direct dependency of hbase.
>>>>>>>>>>>>
>>>>>>>>>>>> I think some of discussion where around MOB that had a MR job
>>> to
>>>>>>>>>> compact,
>>>>>>>>>>>> that later was transformed in a non-MR job to be merged, I
>>> think
>>>>> we
>>>>>>>>>> had a
>>>>>>>>>>>> similar discussion for log split/replay.
>>>>>>>>>>>>
>>>>>>>>>>>> the latest is the new Backup feature (HBASE-7912), that runs
>>> a MR
>>>>> job
>>>>>>>>>>> from
>>>>>>>>>>>> the master to copy data or restore data.
>>>>>>>>>>>> (backup is also "not really core" as in.. if you don't use
>>> backup
>>>>>>>>>> you'll
>>>>>>>>>>>> not end up running MR jobs, but this was probably true for
>>> MOB as
>>>>> in
>>>>>>>>>> "if
>>>>>>>>>>>> you don't enable MOB you don't need MR")
>>>>>>>>>>>>
>>>>>>>>>>>> any thoughts? do we a rule that says "we don't want to have
>>> hbase
>>>>> run
>>>>>>>>>> MR
>>>>>>>>>>>> jobs, only tool started manually by the user can do that". or
>>> can
>>>>> we
>>>>>>>>>>> start
>>>>>>>>>>>> adding MR calls around without problems?
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>
>>
>>
Re: [DISCUSSION] MR jobs started by Master or RS
Posted by Vladimir Rodionov <vl...@gmail.com>.
>> No, never.
No need for M/R here, just a simple compaction-server colocated with RS on
a same node.
You save a lot on GC in RS. Ideally, it can be IO "nice" in Linux (by
setting IO priority). But offtopic, of course :)
-Vlad
On Thu, Sep 22, 2016 at 7:57 PM, Vladimir Rodionov <vl...@gmail.com>
wrote:
> >> And if MR not deployed, Backup/Restore feature could not be used,
> right?
>
> Yes.
>
> On Thu, Sep 22, 2016 at 7:53 PM, Heng Chen <he...@gmail.com>
> wrote:
>
>> {quote}
>> If MR framework is not deployed in the cluster, hbase still functions
>> normally (post merge).
>> {quote}
>>
>> If MR is not strong dependency for Master/RS, it is OK for me.
>> And if MR not deployed, Backup/Restore feature could not be used, right?
>>
>> 2016-09-23 10:49 GMT+08:00 Ted Yu <yu...@gmail.com>:
>> > If MR framework is not deployed in the cluster, hbase still functions
>> > normally (post merge).
>> >
>> > In terms of build time dependency, we have long been depending on
>> > mapreduce. Take a look at ExportSnapshot.
>> >
>> > Cheers
>> >
>> > On Thu, Sep 22, 2016 at 7:42 PM, Heng Chen <he...@gmail.com>
>> wrote:
>> >
>> >> In our production cluster, it is a common case we just have HDFS and
>> >> HBase deployed.
>> >> If our Master/RS depend on MR framework (especially some features we
>> >> have not used at all), it introduced another cost for maintain. I
>> >> don't think it is a good idea.
>> >>
>> >> 2016-09-23 10:28 GMT+08:00 张铎 <pa...@gmail.com>:
>> >> > To be specific, for example, our nice Backup/Restore feature, if we
>> think
>> >> > this is not a core feature of HBase, then we could make it depend on
>> MR,
>> >> > and start a standalone BackupManager instance that submits MR jobs
>> to do
>> >> > periodical maintenance job. And if we think this is a core feature
>> that
>> >> > everyone should use it, then we'd better implement it without MR
>> >> > dependency, like DLS.
>> >> >
>> >> > Thanks.
>> >> >
>> >> > 2016-09-23 10:11 GMT+08:00 张铎 <pa...@gmail.com>:
>> >> >
>> >> >> I‘m -1 on let master or rs launch MR jobs. It is OK that some of our
>> >> >> features depend on MR but I think the bottom line is that we should
>> >> launch
>> >> >> the jobs from outside manually or by other services.
>> >> >>
>> >> >> 2016-09-23 9:47 GMT+08:00 Andrew Purtell <andrew.purtell@gmail.com
>> >:
>> >> >>
>> >> >>> Ok, got it. Well "shelling out" is on the line I think, so a fair
>> >> >>> question.
>> >> >>>
>> >> >>> Can this be driven by a utility derived from Tool like our other MR
>> >> apps?
>> >> >>> The issue is needing the AccessController to decide if allowed? But
>> >> nothing
>> >> >>> prevents the user from running the job manually/independently,
>> right?
>> >> >>>
>> >> >>> > On Sep 22, 2016, at 3:44 PM, Matteo Bertozzi <
>> >> theo.bertozzi@gmail.com>
>> >> >>> wrote:
>> >> >>> >
>> >> >>> > just a remark. my query was not about tools using MR (everyone i
>> >> think
>> >> >>> is
>> >> >>> > ok with those).
>> >> >>> > the topic was about: "are we ok with running MR jobs from Master
>> and
>> >> RSs
>> >> >>> > code?" since this will be the first time we do this
>> >> >>> >
>> >> >>> > Matteo
>> >> >>> >
>> >> >>> >
>> >> >>> >> On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das <
>> ddas@hortonworks.com>
>> >> >>> wrote:
>> >> >>> >>
>> >> >>> >> Very much agree; for tools like ExportSnapshot / Backup /
>> Restore,
>> >> it's
>> >> >>> >> fine to be dependent on MR. MR is the right framework for such.
>> We
>> >> >>> should
>> >> >>> >> also do compactions using MR (just saying :) )
>> >> >>> >> ________________________________________
>> >> >>> >> From: Ted Yu <yu...@gmail.com>
>> >> >>> >> Sent: Thursday, September 22, 2016 2:00 PM
>> >> >>> >> To: dev@hbase.apache.org
>> >> >>> >> Subject: Re: [DISCUSSION] MR jobs started by Master or RS
>> >> >>> >>
>> >> >>> >> I agree - backup / restore is in the same category as import /
>> >> export.
>> >> >>> >>
>> >> >>> >> On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell <
>> >> >>> andrew.purtell@gmail.com>
>> >> >>> >> wrote:
>> >> >>> >>
>> >> >>> >>> Backup is extra tooling around core in my opinion. Like import
>> or
>> >> >>> export.
>> >> >>> >>> Or the optional MOB tool. It's fine.
>> >> >>> >>>
>> >> >>> >>>> On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi <
>> >> mbertozzi@apache.org>
>> >> >>> >>> wrote:
>> >> >>> >>>>
>> >> >>> >>>> What's the latest opinion around running MR jobs from hbase
>> >> (Master
>> >> >>> or
>> >> >>> >>> RS)?
>> >> >>> >>>>
>> >> >>> >>>> I remember in the past that there was discussion about not
>> having
>> >> MR
>> >> >>> >> has
>> >> >>> >>>> direct dependency of hbase.
>> >> >>> >>>>
>> >> >>> >>>> I think some of discussion where around MOB that had a MR job
>> to
>> >> >>> >> compact,
>> >> >>> >>>> that later was transformed in a non-MR job to be merged, I
>> think
>> >> we
>> >> >>> >> had a
>> >> >>> >>>> similar discussion for log split/replay.
>> >> >>> >>>>
>> >> >>> >>>> the latest is the new Backup feature (HBASE-7912), that runs
>> a MR
>> >> job
>> >> >>> >>> from
>> >> >>> >>>> the master to copy data or restore data.
>> >> >>> >>>> (backup is also "not really core" as in.. if you don't use
>> backup
>> >> >>> >> you'll
>> >> >>> >>>> not end up running MR jobs, but this was probably true for
>> MOB as
>> >> in
>> >> >>> >> "if
>> >> >>> >>>> you don't enable MOB you don't need MR")
>> >> >>> >>>>
>> >> >>> >>>> any thoughts? do we a rule that says "we don't want to have
>> hbase
>> >> run
>> >> >>> >> MR
>> >> >>> >>>> jobs, only tool started manually by the user can do that". or
>> can
>> >> we
>> >> >>> >>> start
>> >> >>> >>>> adding MR calls around without problems?
>> >> >>> >>>
>> >> >>> >>
>> >> >>>
>> >> >>
>> >> >>
>> >>
>>
>
>
Re: [DISCUSSION] MR jobs started by Master or RS
Posted by Vladimir Rodionov <vl...@gmail.com>.
>> And if MR not deployed, Backup/Restore feature could not be used, right?
Yes.
On Thu, Sep 22, 2016 at 7:53 PM, Heng Chen <he...@gmail.com> wrote:
> {quote}
> If MR framework is not deployed in the cluster, hbase still functions
> normally (post merge).
> {quote}
>
> If MR is not strong dependency for Master/RS, it is OK for me.
> And if MR not deployed, Backup/Restore feature could not be used, right?
>
> 2016-09-23 10:49 GMT+08:00 Ted Yu <yu...@gmail.com>:
> > If MR framework is not deployed in the cluster, hbase still functions
> > normally (post merge).
> >
> > In terms of build time dependency, we have long been depending on
> > mapreduce. Take a look at ExportSnapshot.
> >
> > Cheers
> >
> > On Thu, Sep 22, 2016 at 7:42 PM, Heng Chen <he...@gmail.com>
> wrote:
> >
> >> In our production cluster, it is a common case we just have HDFS and
> >> HBase deployed.
> >> If our Master/RS depend on MR framework (especially some features we
> >> have not used at all), it introduced another cost for maintain. I
> >> don't think it is a good idea.
> >>
> >> 2016-09-23 10:28 GMT+08:00 张铎 <pa...@gmail.com>:
> >> > To be specific, for example, our nice Backup/Restore feature, if we
> think
> >> > this is not a core feature of HBase, then we could make it depend on
> MR,
> >> > and start a standalone BackupManager instance that submits MR jobs to
> do
> >> > periodical maintenance job. And if we think this is a core feature
> that
> >> > everyone should use it, then we'd better implement it without MR
> >> > dependency, like DLS.
> >> >
> >> > Thanks.
> >> >
> >> > 2016-09-23 10:11 GMT+08:00 张铎 <pa...@gmail.com>:
> >> >
> >> >> I‘m -1 on let master or rs launch MR jobs. It is OK that some of our
> >> >> features depend on MR but I think the bottom line is that we should
> >> launch
> >> >> the jobs from outside manually or by other services.
> >> >>
> >> >> 2016-09-23 9:47 GMT+08:00 Andrew Purtell <an...@gmail.com>:
> >> >>
> >> >>> Ok, got it. Well "shelling out" is on the line I think, so a fair
> >> >>> question.
> >> >>>
> >> >>> Can this be driven by a utility derived from Tool like our other MR
> >> apps?
> >> >>> The issue is needing the AccessController to decide if allowed? But
> >> nothing
> >> >>> prevents the user from running the job manually/independently,
> right?
> >> >>>
> >> >>> > On Sep 22, 2016, at 3:44 PM, Matteo Bertozzi <
> >> theo.bertozzi@gmail.com>
> >> >>> wrote:
> >> >>> >
> >> >>> > just a remark. my query was not about tools using MR (everyone i
> >> think
> >> >>> is
> >> >>> > ok with those).
> >> >>> > the topic was about: "are we ok with running MR jobs from Master
> and
> >> RSs
> >> >>> > code?" since this will be the first time we do this
> >> >>> >
> >> >>> > Matteo
> >> >>> >
> >> >>> >
> >> >>> >> On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das <
> ddas@hortonworks.com>
> >> >>> wrote:
> >> >>> >>
> >> >>> >> Very much agree; for tools like ExportSnapshot / Backup /
> Restore,
> >> it's
> >> >>> >> fine to be dependent on MR. MR is the right framework for such.
> We
> >> >>> should
> >> >>> >> also do compactions using MR (just saying :) )
> >> >>> >> ________________________________________
> >> >>> >> From: Ted Yu <yu...@gmail.com>
> >> >>> >> Sent: Thursday, September 22, 2016 2:00 PM
> >> >>> >> To: dev@hbase.apache.org
> >> >>> >> Subject: Re: [DISCUSSION] MR jobs started by Master or RS
> >> >>> >>
> >> >>> >> I agree - backup / restore is in the same category as import /
> >> export.
> >> >>> >>
> >> >>> >> On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell <
> >> >>> andrew.purtell@gmail.com>
> >> >>> >> wrote:
> >> >>> >>
> >> >>> >>> Backup is extra tooling around core in my opinion. Like import
> or
> >> >>> export.
> >> >>> >>> Or the optional MOB tool. It's fine.
> >> >>> >>>
> >> >>> >>>> On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi <
> >> mbertozzi@apache.org>
> >> >>> >>> wrote:
> >> >>> >>>>
> >> >>> >>>> What's the latest opinion around running MR jobs from hbase
> >> (Master
> >> >>> or
> >> >>> >>> RS)?
> >> >>> >>>>
> >> >>> >>>> I remember in the past that there was discussion about not
> having
> >> MR
> >> >>> >> has
> >> >>> >>>> direct dependency of hbase.
> >> >>> >>>>
> >> >>> >>>> I think some of discussion where around MOB that had a MR job
> to
> >> >>> >> compact,
> >> >>> >>>> that later was transformed in a non-MR job to be merged, I
> think
> >> we
> >> >>> >> had a
> >> >>> >>>> similar discussion for log split/replay.
> >> >>> >>>>
> >> >>> >>>> the latest is the new Backup feature (HBASE-7912), that runs a
> MR
> >> job
> >> >>> >>> from
> >> >>> >>>> the master to copy data or restore data.
> >> >>> >>>> (backup is also "not really core" as in.. if you don't use
> backup
> >> >>> >> you'll
> >> >>> >>>> not end up running MR jobs, but this was probably true for MOB
> as
> >> in
> >> >>> >> "if
> >> >>> >>>> you don't enable MOB you don't need MR")
> >> >>> >>>>
> >> >>> >>>> any thoughts? do we a rule that says "we don't want to have
> hbase
> >> run
> >> >>> >> MR
> >> >>> >>>> jobs, only tool started manually by the user can do that". or
> can
> >> we
> >> >>> >>> start
> >> >>> >>>> adding MR calls around without problems?
> >> >>> >>>
> >> >>> >>
> >> >>>
> >> >>
> >> >>
> >>
>
Re: [DISCUSSION] MR jobs started by Master or RS
Posted by Heng Chen <he...@gmail.com>.
{quote}
If MR framework is not deployed in the cluster, hbase still functions
normally (post merge).
{quote}
If MR is not strong dependency for Master/RS, it is OK for me.
And if MR not deployed, Backup/Restore feature could not be used, right?
2016-09-23 10:49 GMT+08:00 Ted Yu <yu...@gmail.com>:
> If MR framework is not deployed in the cluster, hbase still functions
> normally (post merge).
>
> In terms of build time dependency, we have long been depending on
> mapreduce. Take a look at ExportSnapshot.
>
> Cheers
>
> On Thu, Sep 22, 2016 at 7:42 PM, Heng Chen <he...@gmail.com> wrote:
>
>> In our production cluster, it is a common case we just have HDFS and
>> HBase deployed.
>> If our Master/RS depend on MR framework (especially some features we
>> have not used at all), it introduced another cost for maintain. I
>> don't think it is a good idea.
>>
>> 2016-09-23 10:28 GMT+08:00 张铎 <pa...@gmail.com>:
>> > To be specific, for example, our nice Backup/Restore feature, if we think
>> > this is not a core feature of HBase, then we could make it depend on MR,
>> > and start a standalone BackupManager instance that submits MR jobs to do
>> > periodical maintenance job. And if we think this is a core feature that
>> > everyone should use it, then we'd better implement it without MR
>> > dependency, like DLS.
>> >
>> > Thanks.
>> >
>> > 2016-09-23 10:11 GMT+08:00 张铎 <pa...@gmail.com>:
>> >
>> >> I‘m -1 on let master or rs launch MR jobs. It is OK that some of our
>> >> features depend on MR but I think the bottom line is that we should
>> launch
>> >> the jobs from outside manually or by other services.
>> >>
>> >> 2016-09-23 9:47 GMT+08:00 Andrew Purtell <an...@gmail.com>:
>> >>
>> >>> Ok, got it. Well "shelling out" is on the line I think, so a fair
>> >>> question.
>> >>>
>> >>> Can this be driven by a utility derived from Tool like our other MR
>> apps?
>> >>> The issue is needing the AccessController to decide if allowed? But
>> nothing
>> >>> prevents the user from running the job manually/independently, right?
>> >>>
>> >>> > On Sep 22, 2016, at 3:44 PM, Matteo Bertozzi <
>> theo.bertozzi@gmail.com>
>> >>> wrote:
>> >>> >
>> >>> > just a remark. my query was not about tools using MR (everyone i
>> think
>> >>> is
>> >>> > ok with those).
>> >>> > the topic was about: "are we ok with running MR jobs from Master and
>> RSs
>> >>> > code?" since this will be the first time we do this
>> >>> >
>> >>> > Matteo
>> >>> >
>> >>> >
>> >>> >> On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das <dd...@hortonworks.com>
>> >>> wrote:
>> >>> >>
>> >>> >> Very much agree; for tools like ExportSnapshot / Backup / Restore,
>> it's
>> >>> >> fine to be dependent on MR. MR is the right framework for such. We
>> >>> should
>> >>> >> also do compactions using MR (just saying :) )
>> >>> >> ________________________________________
>> >>> >> From: Ted Yu <yu...@gmail.com>
>> >>> >> Sent: Thursday, September 22, 2016 2:00 PM
>> >>> >> To: dev@hbase.apache.org
>> >>> >> Subject: Re: [DISCUSSION] MR jobs started by Master or RS
>> >>> >>
>> >>> >> I agree - backup / restore is in the same category as import /
>> export.
>> >>> >>
>> >>> >> On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell <
>> >>> andrew.purtell@gmail.com>
>> >>> >> wrote:
>> >>> >>
>> >>> >>> Backup is extra tooling around core in my opinion. Like import or
>> >>> export.
>> >>> >>> Or the optional MOB tool. It's fine.
>> >>> >>>
>> >>> >>>> On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi <
>> mbertozzi@apache.org>
>> >>> >>> wrote:
>> >>> >>>>
>> >>> >>>> What's the latest opinion around running MR jobs from hbase
>> (Master
>> >>> or
>> >>> >>> RS)?
>> >>> >>>>
>> >>> >>>> I remember in the past that there was discussion about not having
>> MR
>> >>> >> has
>> >>> >>>> direct dependency of hbase.
>> >>> >>>>
>> >>> >>>> I think some of discussion where around MOB that had a MR job to
>> >>> >> compact,
>> >>> >>>> that later was transformed in a non-MR job to be merged, I think
>> we
>> >>> >> had a
>> >>> >>>> similar discussion for log split/replay.
>> >>> >>>>
>> >>> >>>> the latest is the new Backup feature (HBASE-7912), that runs a MR
>> job
>> >>> >>> from
>> >>> >>>> the master to copy data or restore data.
>> >>> >>>> (backup is also "not really core" as in.. if you don't use backup
>> >>> >> you'll
>> >>> >>>> not end up running MR jobs, but this was probably true for MOB as
>> in
>> >>> >> "if
>> >>> >>>> you don't enable MOB you don't need MR")
>> >>> >>>>
>> >>> >>>> any thoughts? do we a rule that says "we don't want to have hbase
>> run
>> >>> >> MR
>> >>> >>>> jobs, only tool started manually by the user can do that". or can
>> we
>> >>> >>> start
>> >>> >>>> adding MR calls around without problems?
>> >>> >>>
>> >>> >>
>> >>>
>> >>
>> >>
>>
Re: [DISCUSSION] MR jobs started by Master or RS
Posted by 张铎 <pa...@gmail.com>.
Stability is one thing, and another thing is the difficulty of
configuration and deployment.
For configuration, it is always a pain. I do not want to restart HMaster
many times to get thing right. A standalone service would be better.
For deployment, as chenheng said above, usually we do not deploy a YARN
along with the HBase cluster, which means we need to use an external YARN
cluster to run the job. The cluster may not be controlled by us which means
it may miss the HBase jars or the version is incorrect(as seems YARN's
timeline server depends on HBase?). This could cause the job to fail
because ClassNotFound or some other strange exceptions.
Thanks.
2016-09-23 11:47 GMT+08:00 Ted Yu <yu...@gmail.com>:
> You mean standalone service which runs Procedure V2 ?
> Not sure how much work is involved.
>
> Is this concerning the stability of Master where backup / restore
> procedures run ?
>
> To my understanding, errors in one procedure are isolated, not having
> adverse impact on Master's stability.
>
> On Thu, Sep 22, 2016 at 8:32 PM, 张铎 <pa...@gmail.com> wrote:
>
> > So what about a standalone service other than master? You can use your
> own
> > procedure store in that service?
> >
> > 2016-09-23 11:28 GMT+08:00 Ted Yu <yu...@gmail.com>:
> >
> > > An earlier implementation was client driven.
> > >
> > > But with that approach, it is hard to resume if there is error midway.
> > > Using Procedure V2 makes the backup / restore more robust.
> > >
> > > Another consideration is for security. It is hard to enforce security
> (to
> > > be implemented) for client driven actions.
> > >
> > > Cheers
> > >
> > > > On Sep 22, 2016, at 8:15 PM, Andrew Purtell <
> andrew.purtell@gmail.com>
> > > wrote:
> > > >
> > > > No, this misses Matteo's finer point, which is "shelling out" from
> the
> > > master directly to run MR is a first. Why not drive this with a utility
> > > derived from Tool?
> > > >
> > > > On Sep 22, 2016, at 7:57 PM, Vladimir Rodionov <
> vladrodionov@gmail.com
> > >
> > > wrote:
> > > >
> > > >>>> In our production cluster, it is a common case we just have HDFS
> > and
> > > >>>> HBase deployed.
> > > >>>> If our Master/RS depend on MR framework (especially some features
> we
> > > >>>> have not used at all), it introduced another cost for maintain.
> I
> > > >>>> don't think it is a good idea.
> > > >>
> > > >> So , you are not backup users in this case. Many our customers have
> > full
> > > >> stack deployed and
> > > >> want see backup to be a standard feature. Besides this, nothing will
> > > happen
> > > >> in your cluster
> > > >> if you won't be doing backups.
> > > >>
> > > >> This discussion (we do not want see M/R dependency) goes to nowhere.
> > We
> > > >> asked already, at least twice, to suggest another framework (other
> > than
> > > M/R)
> > > >> for bulk data copy with *conversion*. Still waiting for suggestions.
> > > >>
> > > >> -Vlad
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>> On Thu, Sep 22, 2016 at 7:49 PM, Ted Yu <yu...@gmail.com>
> wrote:
> > > >>>
> > > >>> If MR framework is not deployed in the cluster, hbase still
> functions
> > > >>> normally (post merge).
> > > >>>
> > > >>> In terms of build time dependency, we have long been depending on
> > > >>> mapreduce. Take a look at ExportSnapshot.
> > > >>>
> > > >>> Cheers
> > > >>>
> > > >>> On Thu, Sep 22, 2016 at 7:42 PM, Heng Chen <
> heng.chen.1986@gmail.com
> > >
> > > >>> wrote:
> > > >>>
> > > >>>> In our production cluster, it is a common case we just have HDFS
> > and
> > > >>>> HBase deployed.
> > > >>>> If our Master/RS depend on MR framework (especially some features
> we
> > > >>>> have not used at all), it introduced another cost for maintain.
> I
> > > >>>> don't think it is a good idea.
> > > >>>>
> > > >>>> 2016-09-23 10:28 GMT+08:00 张铎 <pa...@gmail.com>:
> > > >>>>> To be specific, for example, our nice Backup/Restore feature, if
> we
> > > >>> think
> > > >>>>> this is not a core feature of HBase, then we could make it depend
> > on
> > > >>> MR,
> > > >>>>> and start a standalone BackupManager instance that submits MR
> jobs
> > to
> > > >>> do
> > > >>>>> periodical maintenance job. And if we think this is a core
> feature
> > > that
> > > >>>>> everyone should use it, then we'd better implement it without MR
> > > >>>>> dependency, like DLS.
> > > >>>>>
> > > >>>>> Thanks.
> > > >>>>>
> > > >>>>> 2016-09-23 10:11 GMT+08:00 张铎 <pa...@gmail.com>:
> > > >>>>>
> > > >>>>>> I‘m -1 on let master or rs launch MR jobs. It is OK that some of
> > our
> > > >>>>>> features depend on MR but I think the bottom line is that we
> > should
> > > >>>> launch
> > > >>>>>> the jobs from outside manually or by other services.
> > > >>>>>>
> > > >>>>>> 2016-09-23 9:47 GMT+08:00 Andrew Purtell <
> > andrew.purtell@gmail.com
> > > >:
> > > >>>>>>
> > > >>>>>>> Ok, got it. Well "shelling out" is on the line I think, so a
> fair
> > > >>>>>>> question.
> > > >>>>>>>
> > > >>>>>>> Can this be driven by a utility derived from Tool like our
> other
> > MR
> > > >>>> apps?
> > > >>>>>>> The issue is needing the AccessController to decide if allowed?
> > But
> > > >>>> nothing
> > > >>>>>>> prevents the user from running the job manually/independently,
> > > right?
> > > >>>>>>>
> > > >>>>>>>> On Sep 22, 2016, at 3:44 PM, Matteo Bertozzi <
> > > >>>> theo.bertozzi@gmail.com>
> > > >>>>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>> just a remark. my query was not about tools using MR
> (everyone i
> > > >>>> think
> > > >>>>>>> is
> > > >>>>>>>> ok with those).
> > > >>>>>>>> the topic was about: "are we ok with running MR jobs from
> Master
> > > >>> and
> > > >>>> RSs
> > > >>>>>>>> code?" since this will be the first time we do this
> > > >>>>>>>>
> > > >>>>>>>> Matteo
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>> On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das <
> > > >>> ddas@hortonworks.com>
> > > >>>>>>> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>> Very much agree; for tools like ExportSnapshot / Backup /
> > > Restore,
> > > >>>> it's
> > > >>>>>>>>> fine to be dependent on MR. MR is the right framework for
> such.
> > > We
> > > >>>>>>> should
> > > >>>>>>>>> also do compactions using MR (just saying :) )
> > > >>>>>>>>> ________________________________________
> > > >>>>>>>>> From: Ted Yu <yu...@gmail.com>
> > > >>>>>>>>> Sent: Thursday, September 22, 2016 2:00 PM
> > > >>>>>>>>> To: dev@hbase.apache.org
> > > >>>>>>>>> Subject: Re: [DISCUSSION] MR jobs started by Master or RS
> > > >>>>>>>>>
> > > >>>>>>>>> I agree - backup / restore is in the same category as import
> /
> > > >>>> export.
> > > >>>>>>>>>
> > > >>>>>>>>> On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell <
> > > >>>>>>> andrew.purtell@gmail.com>
> > > >>>>>>>>> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>>> Backup is extra tooling around core in my opinion. Like
> import
> > > or
> > > >>>>>>> export.
> > > >>>>>>>>>> Or the optional MOB tool. It's fine.
> > > >>>>>>>>>>
> > > >>>>>>>>>>> On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi <
> > > >>>> mbertozzi@apache.org>
> > > >>>>>>>>>> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> What's the latest opinion around running MR jobs from hbase
> > > >>>> (Master
> > > >>>>>>> or
> > > >>>>>>>>>> RS)?
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I remember in the past that there was discussion about not
> > > >>> having
> > > >>>> MR
> > > >>>>>>>>> has
> > > >>>>>>>>>>> direct dependency of hbase.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I think some of discussion where around MOB that had a MR
> job
> > > to
> > > >>>>>>>>> compact,
> > > >>>>>>>>>>> that later was transformed in a non-MR job to be merged, I
> > > think
> > > >>>> we
> > > >>>>>>>>> had a
> > > >>>>>>>>>>> similar discussion for log split/replay.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> the latest is the new Backup feature (HBASE-7912), that
> runs
> > a
> > > >>> MR
> > > >>>> job
> > > >>>>>>>>>> from
> > > >>>>>>>>>>> the master to copy data or restore data.
> > > >>>>>>>>>>> (backup is also "not really core" as in.. if you don't use
> > > >>> backup
> > > >>>>>>>>> you'll
> > > >>>>>>>>>>> not end up running MR jobs, but this was probably true for
> > MOB
> > > >>> as
> > > >>>> in
> > > >>>>>>>>> "if
> > > >>>>>>>>>>> you don't enable MOB you don't need MR")
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> any thoughts? do we a rule that says "we don't want to have
> > > >>> hbase
> > > >>>> run
> > > >>>>>>>>> MR
> > > >>>>>>>>>>> jobs, only tool started manually by the user can do that".
> or
> > > >>> can
> > > >>>> we
> > > >>>>>>>>>> start
> > > >>>>>>>>>>> adding MR calls around without problems?
> > > >>>
> > >
> >
>
Re: [DISCUSSION] MR jobs started by Master or RS
Posted by Ted Yu <yu...@gmail.com>.
You mean standalone service which runs Procedure V2 ?
Not sure how much work is involved.
Is this concerning the stability of Master where backup / restore
procedures run ?
To my understanding, errors in one procedure are isolated, not having
adverse impact on Master's stability.
On Thu, Sep 22, 2016 at 8:32 PM, 张铎 <pa...@gmail.com> wrote:
> So what about a standalone service other than master? You can use your own
> procedure store in that service?
>
> 2016-09-23 11:28 GMT+08:00 Ted Yu <yu...@gmail.com>:
>
> > An earlier implementation was client driven.
> >
> > But with that approach, it is hard to resume if there is error midway.
> > Using Procedure V2 makes the backup / restore more robust.
> >
> > Another consideration is for security. It is hard to enforce security (to
> > be implemented) for client driven actions.
> >
> > Cheers
> >
> > > On Sep 22, 2016, at 8:15 PM, Andrew Purtell <an...@gmail.com>
> > wrote:
> > >
> > > No, this misses Matteo's finer point, which is "shelling out" from the
> > master directly to run MR is a first. Why not drive this with a utility
> > derived from Tool?
> > >
> > > On Sep 22, 2016, at 7:57 PM, Vladimir Rodionov <vladrodionov@gmail.com
> >
> > wrote:
> > >
> > >>>> In our production cluster, it is a common case we just have HDFS
> and
> > >>>> HBase deployed.
> > >>>> If our Master/RS depend on MR framework (especially some features we
> > >>>> have not used at all), it introduced another cost for maintain. I
> > >>>> don't think it is a good idea.
> > >>
> > >> So , you are not backup users in this case. Many our customers have
> full
> > >> stack deployed and
> > >> want see backup to be a standard feature. Besides this, nothing will
> > happen
> > >> in your cluster
> > >> if you won't be doing backups.
> > >>
> > >> This discussion (we do not want see M/R dependency) goes to nowhere.
> We
> > >> asked already, at least twice, to suggest another framework (other
> than
> > M/R)
> > >> for bulk data copy with *conversion*. Still waiting for suggestions.
> > >>
> > >> -Vlad
> > >>
> > >>
> > >>
> > >>
> > >>> On Thu, Sep 22, 2016 at 7:49 PM, Ted Yu <yu...@gmail.com> wrote:
> > >>>
> > >>> If MR framework is not deployed in the cluster, hbase still functions
> > >>> normally (post merge).
> > >>>
> > >>> In terms of build time dependency, we have long been depending on
> > >>> mapreduce. Take a look at ExportSnapshot.
> > >>>
> > >>> Cheers
> > >>>
> > >>> On Thu, Sep 22, 2016 at 7:42 PM, Heng Chen <heng.chen.1986@gmail.com
> >
> > >>> wrote:
> > >>>
> > >>>> In our production cluster, it is a common case we just have HDFS
> and
> > >>>> HBase deployed.
> > >>>> If our Master/RS depend on MR framework (especially some features we
> > >>>> have not used at all), it introduced another cost for maintain. I
> > >>>> don't think it is a good idea.
> > >>>>
> > >>>> 2016-09-23 10:28 GMT+08:00 张铎 <pa...@gmail.com>:
> > >>>>> To be specific, for example, our nice Backup/Restore feature, if we
> > >>> think
> > >>>>> this is not a core feature of HBase, then we could make it depend
> on
> > >>> MR,
> > >>>>> and start a standalone BackupManager instance that submits MR jobs
> to
> > >>> do
> > >>>>> periodical maintenance job. And if we think this is a core feature
> > that
> > >>>>> everyone should use it, then we'd better implement it without MR
> > >>>>> dependency, like DLS.
> > >>>>>
> > >>>>> Thanks.
> > >>>>>
> > >>>>> 2016-09-23 10:11 GMT+08:00 张铎 <pa...@gmail.com>:
> > >>>>>
> > >>>>>> I‘m -1 on let master or rs launch MR jobs. It is OK that some of
> our
> > >>>>>> features depend on MR but I think the bottom line is that we
> should
> > >>>> launch
> > >>>>>> the jobs from outside manually or by other services.
> > >>>>>>
> > >>>>>> 2016-09-23 9:47 GMT+08:00 Andrew Purtell <
> andrew.purtell@gmail.com
> > >:
> > >>>>>>
> > >>>>>>> Ok, got it. Well "shelling out" is on the line I think, so a fair
> > >>>>>>> question.
> > >>>>>>>
> > >>>>>>> Can this be driven by a utility derived from Tool like our other
> MR
> > >>>> apps?
> > >>>>>>> The issue is needing the AccessController to decide if allowed?
> But
> > >>>> nothing
> > >>>>>>> prevents the user from running the job manually/independently,
> > right?
> > >>>>>>>
> > >>>>>>>> On Sep 22, 2016, at 3:44 PM, Matteo Bertozzi <
> > >>>> theo.bertozzi@gmail.com>
> > >>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>> just a remark. my query was not about tools using MR (everyone i
> > >>>> think
> > >>>>>>> is
> > >>>>>>>> ok with those).
> > >>>>>>>> the topic was about: "are we ok with running MR jobs from Master
> > >>> and
> > >>>> RSs
> > >>>>>>>> code?" since this will be the first time we do this
> > >>>>>>>>
> > >>>>>>>> Matteo
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>> On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das <
> > >>> ddas@hortonworks.com>
> > >>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>> Very much agree; for tools like ExportSnapshot / Backup /
> > Restore,
> > >>>> it's
> > >>>>>>>>> fine to be dependent on MR. MR is the right framework for such.
> > We
> > >>>>>>> should
> > >>>>>>>>> also do compactions using MR (just saying :) )
> > >>>>>>>>> ________________________________________
> > >>>>>>>>> From: Ted Yu <yu...@gmail.com>
> > >>>>>>>>> Sent: Thursday, September 22, 2016 2:00 PM
> > >>>>>>>>> To: dev@hbase.apache.org
> > >>>>>>>>> Subject: Re: [DISCUSSION] MR jobs started by Master or RS
> > >>>>>>>>>
> > >>>>>>>>> I agree - backup / restore is in the same category as import /
> > >>>> export.
> > >>>>>>>>>
> > >>>>>>>>> On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell <
> > >>>>>>> andrew.purtell@gmail.com>
> > >>>>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> Backup is extra tooling around core in my opinion. Like import
> > or
> > >>>>>>> export.
> > >>>>>>>>>> Or the optional MOB tool. It's fine.
> > >>>>>>>>>>
> > >>>>>>>>>>> On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi <
> > >>>> mbertozzi@apache.org>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>> What's the latest opinion around running MR jobs from hbase
> > >>>> (Master
> > >>>>>>> or
> > >>>>>>>>>> RS)?
> > >>>>>>>>>>>
> > >>>>>>>>>>> I remember in the past that there was discussion about not
> > >>> having
> > >>>> MR
> > >>>>>>>>> has
> > >>>>>>>>>>> direct dependency of hbase.
> > >>>>>>>>>>>
> > >>>>>>>>>>> I think some of discussion where around MOB that had a MR job
> > to
> > >>>>>>>>> compact,
> > >>>>>>>>>>> that later was transformed in a non-MR job to be merged, I
> > think
> > >>>> we
> > >>>>>>>>> had a
> > >>>>>>>>>>> similar discussion for log split/replay.
> > >>>>>>>>>>>
> > >>>>>>>>>>> the latest is the new Backup feature (HBASE-7912), that runs
> a
> > >>> MR
> > >>>> job
> > >>>>>>>>>> from
> > >>>>>>>>>>> the master to copy data or restore data.
> > >>>>>>>>>>> (backup is also "not really core" as in.. if you don't use
> > >>> backup
> > >>>>>>>>> you'll
> > >>>>>>>>>>> not end up running MR jobs, but this was probably true for
> MOB
> > >>> as
> > >>>> in
> > >>>>>>>>> "if
> > >>>>>>>>>>> you don't enable MOB you don't need MR")
> > >>>>>>>>>>>
> > >>>>>>>>>>> any thoughts? do we a rule that says "we don't want to have
> > >>> hbase
> > >>>> run
> > >>>>>>>>> MR
> > >>>>>>>>>>> jobs, only tool started manually by the user can do that". or
> > >>> can
> > >>>> we
> > >>>>>>>>>> start
> > >>>>>>>>>>> adding MR calls around without problems?
> > >>>
> >
>
Re: [DISCUSSION] MR jobs started by Master or RS
Posted by Devaraj Das <dd...@hortonworks.com>.
Great points. I am getting a feeling that we have beaten this topic to death and have spent enough and more time on it, and it's time to move on. My takeaway is that it's fine to do MR for backup/restore - backup/restore is an optional feature - unless you use it, you don't need MR.
Fingers crossed,
Devaraj.
P.S. To Nick's point, if it makes sense to do MR/Spark even for core features like Compactions, we should be open to it. But that's a topic for another [Friday] DISCUSS thread.
________________________________________
From: Jerry He <je...@gmail.com>
Sent: Friday, September 23, 2016 8:40 AM
To: dev
Subject: Re: [DISCUSSION] MR jobs started by Master or RS
That is the point, Matteo.
Put it another way, there is nothing that stops a user from deploying
custom procedure, custom co-processor that calls out MR job.
The optional feature should satisfy some basic requirements. .e.g. No
impact if not deployed or used. Limited impact if used.
It can be made with isolated dynamic loading of extra configuration (Yarn),
non-blocking non-occupying on the server handlers, or separate handler.
The impact would mostly be on the overall cluster resources. In this sense,
there is no difference, using another standalone server or a command tool.
The exportEnapshot can then be moved to the server as well.
Also, thinking about in the higher level. It is probably beneficial if you
allow HBase to call out an external framework to do computation. It can be
think of as a UDF, a distributed UDF.
The execution of this UDF is totally in separate address spaces, and you
only need to poll the status. This would be like a dream in traditional
database.
My 2 cents.
Jerry
On Fri, Sep 23, 2016 at 6:46 AM, Matteo Bertozzi <th...@gmail.com>
wrote:
> let me try to go back to my original topic.
> this question was meant to be generic, and provide some rule for future
> code.
>
> from what I can gather, a rule that may satisfy everyone can be:
> - we don't want any core feature (e.g. compaction/log-split/log-reply)
> over MR, because some cluster may not want or may have an
> external/uncontrolled MR setup.
> - we allow non-core features (e.g. features enabled by a flag) to run MR
> jobs from hbase, because unless you use the feature, MR is not required.
>
> Matteo
>
>
> On Fri, Sep 23, 2016 at 5:39 AM, Ted Yu <yu...@gmail.com> wrote:
>
> > I suggest you look at Matteo's work for AssignmentManager which is to
> make
> > Master more stable.
> >
> > Cheers
> >
> > On Fri, Sep 23, 2016 at 5:32 AM, 张铎 <pa...@gmail.com> wrote:
> >
> > > No, not your fault, at lease, not this time:)
> > >
> > > Why I call the code ugly? Can you simply tell me the sequence of calls
> > when
> > > starting up the HMaster? HMaster is also a regionserver so it extends
> > > HRegionServer, and the initialization of HRegionServer sometimes needs
> to
> > > make rpc calls to HMaster. A simple change would cause probabilistic
> dead
> > > lock or some strange NPEs...
> > >
> > > That's why I'm very nervous when somebody wants to add new features or
> > add
> > > external dependencies to HMaster, especially add more works for the
> start
> > > up processing...
> > >
> > > Thanks.
> > >
> > > 2016-09-23 20:02 GMT+08:00 Ted Yu <yu...@gmail.com>:
> > >
> > > > I read through HADOOP-13433
> > > > <https://issues.apache.org/jira/browse/HADOOP-13433> - the cited
> race
> > > > condition is in jdk.
> > > >
> > > > Suggest pinging the reviewer on JIRA to get it moving.
> > > >
> > > > bq. But the ugly code in HMaster is readlly a problem...
> > > >
> > > > Can you be specific as to which code is ugly ? Is it in the backup /
> > > > restore mega patch ?
> > > >
> > > > Cheers
> > > >
> > > > On Thu, Sep 22, 2016 at 10:44 PM, 张铎 <pa...@gmail.com> wrote:
> > > >
> > > > > If you guys have already implemented the feature in the MR way and
> > the
> > > > > patch is ready for landing on master, I'm a -0 on it as I do not
> want
> > > to
> > > > > block the development progress.
> > > > >
> > > > > But I strongly suggest later we need to revisit the design and see
> if
> > > we
> > > > > can seperated the logic from HMaster as much as possible. HA is
> not a
> > > big
> > > > > problem if you do not store any metada locally. But the ugly code
> in
> > > > > HMaster is readlly a problem...
> > > > >
> > > > > And for security, I have a issue pending for a long time. Can
> someone
> > > > help
> > > > > taking a simple look at it? This is what I mean, ugly code...
> logout
> > > and
> > > > > destroy the credentials in a subject when it is still being used,
> and
> > > > > declared as LimitPrivacy so I can not change the behivor and the
> only
> > > way
> > > > > to fix it is to write another piece of ugly code...
> > > > >
> > > > > https://issues.apache.org/jira/browse/HADOOP-13433
> > > > >
> > > > > 2016-09-23 12:53 GMT+08:00 Vladimir Rodionov <
> vladrodionov@gmail.com
> > >:
> > > > >
> > > > > > >> If in the future, we find better ways of doing this without
> > using
> > > > MR,
> > > > > we
> > > > > > can certainly consider that
> > > > > >
> > > > > > Our framework for distributed operations is abstract and allows
> > > > > > different implementations. MR is just one implementation we
> > provide.
> > > > > >
> > > > > > -Vlad
> > > > > >
> > > > > > On Thu, Sep 22, 2016 at 9:38 PM, Devaraj Das <
> ddas@hortonworks.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Guys, first off apologies for bringing in the topic of MR-based
> > > > > > > compactions.. But I was thinking more about the SpliceMachine
> > > > approach
> > > > > of
> > > > > > > managing compactions in Spark where apparently they saw a lot
> of
> > > > > > benefits.
> > > > > > > Apologies for giving you that sore throat Andrew; I really
> didn't
> > > > mean
> > > > > to
> > > > > > > :-)
> > > > > > >
> > > > > > > So on this issue, we have these on the plate:
> > > > > > > 0. Somehow not use MR but something like that
> > > > > > > 1. Run a standalone service other than master
> > > > > > > 2. Shell out from the master
> > > > > > >
> > > > > > > I don't think we have a good answer to (0), and I don't think
> > it's
> > > > even
> > > > > > > worth the effort of trying to build something when MR is
> already
> > > > there,
> > > > > > and
> > > > > > > being used by HBase already for some operations.
> > > > > > >
> > > > > > > On (1), we have to deal with a myriad of issues - HA of the
> > server
> > > > not
> > > > > > > being the least of them all. Security (kerberos authentication,
> > > > another
> > > > > > > keytab to manage, etc. etc. etc.). IMO, that approach is DOA.
> > > Instead
> > > > > > let's
> > > > > > > substitute that (1) with the HBase Master. I haven't seen any
> > good
> > > > > reason
> > > > > > > why the HBase master shouldn't launch MR jobs if needed. It's
> not
> > > > > ideal;
> > > > > > > agreed.
> > > > > > >
> > > > > > > Now before going to (2), let's see what are the benefits of
> > running
> > > > the
> > > > > > > backup/restore jobs from the master. I think Ted has summarized
> > > some
> > > > of
> > > > > > the
> > > > > > > issues that we need to take care of - basically, the master can
> > > keep
> > > > > > track
> > > > > > > of running jobs, and should it fail, the backup master can
> > continue
> > > > > > keeping
> > > > > > > track of it (since the jobId would have been recorded in the
> proc
> > > > WAL).
> > > > > > The
> > > > > > > master can also do cleanup, etc. of failed backup/restore
> > > processes.
> > > > > > > Security is another issue - the job needs to run as 'hbase'
> since
> > > it
> > > > > owns
> > > > > > > the data. Having the master launch the job makes it get that
> > > > privilege.
> > > > > > In
> > > > > > > the (2) approach, it's hard to do some of the above management.
> > > > > > >
> > > > > > > Guys, just to reiterate, the patch as such is ready from the
> > > overall
> > > > > > > design/arch point of view (maybe code review is still pending
> > from
> > > > > > Matteo).
> > > > > > > If in the future, we find better ways of doing this without
> using
> > > MR,
> > > > > we
> > > > > > > can certainly consider that. But IMO don't think we should
> block
> > > this
> > > > > > patch
> > > > > > > from getting merged.
> > > > > > >
> > > > > > > ________________________________________
> > > > > > > From: 张铎 <pa...@gmail.com>
> > > > > > > Sent: Thursday, September 22, 2016 8:32 PM
> > > > > > > To: dev@hbase.apache.org
> > > > > > > Subject: Re: [DISCUSSION] MR jobs started by Master or RS
> > > > > > >
> > > > > > > So what about a standalone service other than master? You can
> use
> > > > your
> > > > > > own
> > > > > > > procedure store in that service?
> > > > > > >
> > > > > > > 2016-09-23 11:28 GMT+08:00 Ted Yu <yu...@gmail.com>:
> > > > > > >
> > > > > > > > An earlier implementation was client driven.
> > > > > > > >
> > > > > > > > But with that approach, it is hard to resume if there is
> error
> > > > > midway.
> > > > > > > > Using Procedure V2 makes the backup / restore more robust.
> > > > > > > >
> > > > > > > > Another consideration is for security. It is hard to enforce
> > > > security
> > > > > > (to
> > > > > > > > be implemented) for client driven actions.
> > > > > > > >
> > > > > > > > Cheers
> > > > > > > >
> > > > > > > > > On Sep 22, 2016, at 8:15 PM, Andrew Purtell <
> > > > > > andrew.purtell@gmail.com>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > No, this misses Matteo's finer point, which is "shelling
> out"
> > > > from
> > > > > > the
> > > > > > > > master directly to run MR is a first. Why not drive this
> with a
> > > > > utility
> > > > > > > > derived from Tool?
> > > > > > > > >
> > > > > > > > > On Sep 22, 2016, at 7:57 PM, Vladimir Rodionov <
> > > > > > vladrodionov@gmail.com
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >>>> In our production cluster, it is a common case we just
> > have
> > > > > HDFS
> > > > > > > and
> > > > > > > > >>>> HBase deployed.
> > > > > > > > >>>> If our Master/RS depend on MR framework (especially some
> > > > > features
> > > > > > we
> > > > > > > > >>>> have not used at all), it introduced another cost for
> > > > maintain.
> > > > > > I
> > > > > > > > >>>> don't think it is a good idea.
> > > > > > > > >>
> > > > > > > > >> So , you are not backup users in this case. Many our
> > customers
> > > > > have
> > > > > > > full
> > > > > > > > >> stack deployed and
> > > > > > > > >> want see backup to be a standard feature. Besides this,
> > > nothing
> > > > > will
> > > > > > > > happen
> > > > > > > > >> in your cluster
> > > > > > > > >> if you won't be doing backups.
> > > > > > > > >>
> > > > > > > > >> This discussion (we do not want see M/R dependency) goes
> to
> > > > > nowhere.
> > > > > > > We
> > > > > > > > >> asked already, at least twice, to suggest another
> framework
> > > > (other
> > > > > > > than
> > > > > > > > M/R)
> > > > > > > > >> for bulk data copy with *conversion*. Still waiting for
> > > > > suggestions.
> > > > > > > > >>
> > > > > > > > >> -Vlad
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>> On Thu, Sep 22, 2016 at 7:49 PM, Ted Yu <
> > yuzhihong@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > > >>>
> > > > > > > > >>> If MR framework is not deployed in the cluster, hbase
> still
> > > > > > functions
> > > > > > > > >>> normally (post merge).
> > > > > > > > >>>
> > > > > > > > >>> In terms of build time dependency, we have long been
> > > depending
> > > > on
> > > > > > > > >>> mapreduce. Take a look at ExportSnapshot.
> > > > > > > > >>>
> > > > > > > > >>> Cheers
> > > > > > > > >>>
> > > > > > > > >>> On Thu, Sep 22, 2016 at 7:42 PM, Heng Chen <
> > > > > > heng.chen.1986@gmail.com
> > > > > > > >
> > > > > > > > >>> wrote:
> > > > > > > > >>>
> > > > > > > > >>>> In our production cluster, it is a common case we just
> > have
> > > > > HDFS
> > > > > > > and
> > > > > > > > >>>> HBase deployed.
> > > > > > > > >>>> If our Master/RS depend on MR framework (especially some
> > > > > features
> > > > > > we
> > > > > > > > >>>> have not used at all), it introduced another cost for
> > > > maintain.
> > > > > > I
> > > > > > > > >>>> don't think it is a good idea.
> > > > > > > > >>>>
> > > > > > > > >>>> 2016-09-23 10:28 GMT+08:00 张铎 <pa...@gmail.com>:
> > > > > > > > >>>>> To be specific, for example, our nice Backup/Restore
> > > feature,
> > > > > if
> > > > > > we
> > > > > > > > >>> think
> > > > > > > > >>>>> this is not a core feature of HBase, then we could make
> > it
> > > > > depend
> > > > > > > on
> > > > > > > > >>> MR,
> > > > > > > > >>>>> and start a standalone BackupManager instance that
> > submits
> > > MR
> > > > > > jobs
> > > > > > > to
> > > > > > > > >>> do
> > > > > > > > >>>>> periodical maintenance job. And if we think this is a
> > core
> > > > > > feature
> > > > > > > > that
> > > > > > > > >>>>> everyone should use it, then we'd better implement it
> > > without
> > > > > MR
> > > > > > > > >>>>> dependency, like DLS.
> > > > > > > > >>>>>
> > > > > > > > >>>>> Thanks.
> > > > > > > > >>>>>
> > > > > > > > >>>>> 2016-09-23 10:11 GMT+08:00 张铎 <pa...@gmail.com>:
> > > > > > > > >>>>>
> > > > > > > > >>>>>> I‘m -1 on let master or rs launch MR jobs. It is OK
> that
> > > > some
> > > > > of
> > > > > > > our
> > > > > > > > >>>>>> features depend on MR but I think the bottom line is
> > that
> > > we
> > > > > > > should
> > > > > > > > >>>> launch
> > > > > > > > >>>>>> the jobs from outside manually or by other services.
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> 2016-09-23 9:47 GMT+08:00 Andrew Purtell <
> > > > > > > andrew.purtell@gmail.com
> > > > > > > > >:
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>> Ok, got it. Well "shelling out" is on the line I
> think,
> > > so
> > > > a
> > > > > > fair
> > > > > > > > >>>>>>> question.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> Can this be driven by a utility derived from Tool
> like
> > > our
> > > > > > other
> > > > > > > MR
> > > > > > > > >>>> apps?
> > > > > > > > >>>>>>> The issue is needing the AccessController to decide
> if
> > > > > allowed?
> > > > > > > But
> > > > > > > > >>>> nothing
> > > > > > > > >>>>>>> prevents the user from running the job
> > > > > manually/independently,
> > > > > > > > right?
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>>> On Sep 22, 2016, at 3:44 PM, Matteo Bertozzi <
> > > > > > > > >>>> theo.bertozzi@gmail.com>
> > > > > > > > >>>>>>> wrote:
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> just a remark. my query was not about tools using MR
> > > > > > (everyone i
> > > > > > > > >>>> think
> > > > > > > > >>>>>>> is
> > > > > > > > >>>>>>>> ok with those).
> > > > > > > > >>>>>>>> the topic was about: "are we ok with running MR jobs
> > > from
> > > > > > Master
> > > > > > > > >>> and
> > > > > > > > >>>> RSs
> > > > > > > > >>>>>>>> code?" since this will be the first time we do this
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> Matteo
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>>> On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das <
> > > > > > > > >>> ddas@hortonworks.com>
> > > > > > > > >>>>>>> wrote:
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> Very much agree; for tools like ExportSnapshot /
> > > Backup /
> > > > > > > > Restore,
> > > > > > > > >>>> it's
> > > > > > > > >>>>>>>>> fine to be dependent on MR. MR is the right
> framework
> > > for
> > > > > > such.
> > > > > > > > We
> > > > > > > > >>>>>>> should
> > > > > > > > >>>>>>>>> also do compactions using MR (just saying :) )
> > > > > > > > >>>>>>>>> ________________________________________
> > > > > > > > >>>>>>>>> From: Ted Yu <yu...@gmail.com>
> > > > > > > > >>>>>>>>> Sent: Thursday, September 22, 2016 2:00 PM
> > > > > > > > >>>>>>>>> To: dev@hbase.apache.org
> > > > > > > > >>>>>>>>> Subject: Re: [DISCUSSION] MR jobs started by Master
> > or
> > > RS
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> I agree - backup / restore is in the same category
> as
> > > > > import
> > > > > > /
> > > > > > > > >>>> export.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell <
> > > > > > > > >>>>>>> andrew.purtell@gmail.com>
> > > > > > > > >>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>>> Backup is extra tooling around core in my opinion.
> > > Like
> > > > > > import
> > > > > > > > or
> > > > > > > > >>>>>>> export.
> > > > > > > > >>>>>>>>>> Or the optional MOB tool. It's fine.
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>>> On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi <
> > > > > > > > >>>> mbertozzi@apache.org>
> > > > > > > > >>>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> What's the latest opinion around running MR jobs
> > from
> > > > > hbase
> > > > > > > > >>>> (Master
> > > > > > > > >>>>>>> or
> > > > > > > > >>>>>>>>>> RS)?
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> I remember in the past that there was discussion
> > > about
> > > > > not
> > > > > > > > >>> having
> > > > > > > > >>>> MR
> > > > > > > > >>>>>>>>> has
> > > > > > > > >>>>>>>>>>> direct dependency of hbase.
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> I think some of discussion where around MOB that
> > had
> > > a
> > > > MR
> > > > > > job
> > > > > > > > to
> > > > > > > > >>>>>>>>> compact,
> > > > > > > > >>>>>>>>>>> that later was transformed in a non-MR job to be
> > > > merged,
> > > > > I
> > > > > > > > think
> > > > > > > > >>>> we
> > > > > > > > >>>>>>>>> had a
> > > > > > > > >>>>>>>>>>> similar discussion for log split/replay.
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> the latest is the new Backup feature
> (HBASE-7912),
> > > that
> > > > > > runs
> > > > > > > a
> > > > > > > > >>> MR
> > > > > > > > >>>> job
> > > > > > > > >>>>>>>>>> from
> > > > > > > > >>>>>>>>>>> the master to copy data or restore data.
> > > > > > > > >>>>>>>>>>> (backup is also "not really core" as in.. if you
> > > don't
> > > > > use
> > > > > > > > >>> backup
> > > > > > > > >>>>>>>>> you'll
> > > > > > > > >>>>>>>>>>> not end up running MR jobs, but this was probably
> > > true
> > > > > for
> > > > > > > MOB
> > > > > > > > >>> as
> > > > > > > > >>>> in
> > > > > > > > >>>>>>>>> "if
> > > > > > > > >>>>>>>>>>> you don't enable MOB you don't need MR")
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> any thoughts? do we a rule that says "we don't
> want
> > > to
> > > > > have
> > > > > > > > >>> hbase
> > > > > > > > >>>> run
> > > > > > > > >>>>>>>>> MR
> > > > > > > > >>>>>>>>>>> jobs, only tool started manually by the user can
> do
> > > > > that".
> > > > > > or
> > > > > > > > >>> can
> > > > > > > > >>>> we
> > > > > > > > >>>>>>>>>> start
> > > > > > > > >>>>>>>>>>> adding MR calls around without problems?
> > > > > > > > >>>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSSION] MR jobs started by Master or RS
Posted by Andrew Purtell <ap...@apache.org>.
It's simple: Use a tool (something derived from Tool, like our other
utilities for import, export, etc.), or use a separate process, but don't
"shell out" to MR from the Master.
On Fri, Sep 23, 2016 at 12:24 PM, Vladimir Rodionov <vl...@gmail.com>
wrote:
> >> -1 on that backup be in core hbase
>
> Not sure I understand what it means.
>
> 1. We are not allowed to use Master to orchestrate the whole process? We
> have already brought up all advantages of using
> Master and distributed procedures for backup and restore.
>
>
> Downside of moving this to client tool is lack of fault tolerance:
> 1.1 Client won't be allowed to do any operations, that can, potentially
> affect
> cluster, such as disabling splits/merges, balancer.
> 1.2 In case of client failure who will be doing the whole rollback stuff?
> We are trying to make it atomic.
>
> Security is not clear.
>
> 2. We are not allowed to modify code of existing HBase core classes (what
> does core mean anyway)?
>
> 3. We are not allowed to create backup system table (hbase:backup) in a
> system space? Only in user space? The table is global.
>
> 2. is critical. Despite the fact, that 95% of code is new, we have touched,
> of course some existing HBase code.
> 3. is not that critical, of course we can move backup system into user
> space.
>
> And finally, will moving backup into external tool give us +1 from stack?
>
> -Vlad
>
>
>
>
>
> On Fri, Sep 23, 2016 at 11:26 AM, Stack <st...@duboce.net> wrote:
>
> > On Fri, Sep 23, 2016 at 11:22 AM, Vladimir Rodionov <
> > vladrodionov@gmail.com>
> > wrote:
> >
> > > >> + MR is dead
> > >
> > > Does MR know that? :)
> > >
> > > Again. With all due respect, stack - still no suggestions what should
> we
> > > use for "bulk data move and transformation" instead of MR?
> > >
> >
> > Use whatever distributed engine suits your fancy -- MR, Spark,
> distributed
> > shell -- just don't have HBase core depend on it, even optionally.
> >
> >
> > > I suggest voting first on "do we need backup in HBase"? In my opinion,
> > some
> > > group members still not sure about that and some will give -1
> > > in any case. Just because ...
> > >
> > >
> > We could run a vote, sure. -1 on that backup be in core hbase (+1 on
> adding
> > all the API any such external tool might need to run).
> >
> > St.Ack
> >
> >
> >
> > > -Vlad
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Fri, Sep 23, 2016 at 10:57 AM, Stack <st...@duboce.net> wrote:
> > >
> > > > On Fri, Sep 23, 2016 at 6:46 AM, Matteo Bertozzi <
> > > theo.bertozzi@gmail.com>
> > > > wrote:
> > > >
> > > > > let me try to go back to my original topic.
> > > > > this question was meant to be generic, and provide some rule for
> > future
> > > > > code.
> > > > >
> > > > > from what I can gather, a rule that may satisfy everyone can be:
> > > > > - we don't want any core feature (e.g. compaction/log-split/log-
> > > reply)
> > > > > over MR, because some cluster may not want or may have an
> > > > > external/uncontrolled MR setup.
> > > > >
> > > >
> > > > +1
> > > >
> > > >
> > > > > - we allow non-core features (e.g. features enabled by a flag) to
> > run
> > > MR
> > > > > jobs from hbase, because unless you use the feature, MR is not
> > > required.
> > > > >
> > > > >
> > > > -1 to hbase core depending on MR or core -- whether behind a flag or
> > not
> > > --
> > > > ever being able to launch MR jobs.
> > > >
> > > > + MR is dead. We should be busy working hard to undo it from
> > hbase-server
> > > > moving it out to be an optional module (Spark would be its peer).
> > > > + Master is a rats nest of state. Matteo, Stephen, and Appy are busy
> > > > working hard on moving it up on to a new foundation. Lets not clutter
> > > task
> > > > harder by piling on more moving parts.
> > > >
> > > > St.Ack
> > > >
> > > >
> > > > > Matteo
> > > > >
> > > > >
> > > > > On Fri, Sep 23, 2016 at 5:39 AM, Ted Yu <yu...@gmail.com>
> wrote:
> > > > >
> > > > > > I suggest you look at Matteo's work for AssignmentManager which
> is
> > to
> > > > > make
> > > > > > Master more stable.
> > > > > >
> > > > > > Cheers
> > > > > >
> > > > > > On Fri, Sep 23, 2016 at 5:32 AM, 张铎 <pa...@gmail.com>
> wrote:
> > > > > >
> > > > > > > No, not your fault, at lease, not this time:)
> > > > > > >
> > > > > > > Why I call the code ugly? Can you simply tell me the sequence
> of
> > > > calls
> > > > > > when
> > > > > > > starting up the HMaster? HMaster is also a regionserver so it
> > > extends
> > > > > > > HRegionServer, and the initialization of HRegionServer
> sometimes
> > > > needs
> > > > > to
> > > > > > > make rpc calls to HMaster. A simple change would cause
> > > probabilistic
> > > > > dead
> > > > > > > lock or some strange NPEs...
> > > > > > >
> > > > > > > That's why I'm very nervous when somebody wants to add new
> > features
> > > > or
> > > > > > add
> > > > > > > external dependencies to HMaster, especially add more works for
> > the
> > > > > start
> > > > > > > up processing...
> > > > > > >
> > > > > > > Thanks.
> > > > > > >
> > > > > > > 2016-09-23 20:02 GMT+08:00 Ted Yu <yu...@gmail.com>:
> > > > > > >
> > > > > > > > I read through HADOOP-13433
> > > > > > > > <https://issues.apache.org/jira/browse/HADOOP-13433> - the
> > cited
> > > > > race
> > > > > > > > condition is in jdk.
> > > > > > > >
> > > > > > > > Suggest pinging the reviewer on JIRA to get it moving.
> > > > > > > >
> > > > > > > > bq. But the ugly code in HMaster is readlly a problem...
> > > > > > > >
> > > > > > > > Can you be specific as to which code is ugly ? Is it in the
> > > backup
> > > > /
> > > > > > > > restore mega patch ?
> > > > > > > >
> > > > > > > > Cheers
> > > > > > > >
> > > > > > > > On Thu, Sep 22, 2016 at 10:44 PM, 张铎 <pa...@gmail.com>
> > > > wrote:
> > > > > > > >
> > > > > > > > > If you guys have already implemented the feature in the MR
> > way
> > > > and
> > > > > > the
> > > > > > > > > patch is ready for landing on master, I'm a -0 on it as I
> do
> > > not
> > > > > want
> > > > > > > to
> > > > > > > > > block the development progress.
> > > > > > > > >
> > > > > > > > > But I strongly suggest later we need to revisit the design
> > and
> > > > see
> > > > > if
> > > > > > > we
> > > > > > > > > can seperated the logic from HMaster as much as possible.
> HA
> > is
> > > > > not a
> > > > > > > big
> > > > > > > > > problem if you do not store any metada locally. But the
> ugly
> > > code
> > > > > in
> > > > > > > > > HMaster is readlly a problem...
> > > > > > > > >
> > > > > > > > > And for security, I have a issue pending for a long time.
> Can
> > > > > someone
> > > > > > > > help
> > > > > > > > > taking a simple look at it? This is what I mean, ugly
> code...
> > > > > logout
> > > > > > > and
> > > > > > > > > destroy the credentials in a subject when it is still being
> > > used,
> > > > > and
> > > > > > > > > declared as LimitPrivacy so I can not change the behivor
> and
> > > the
> > > > > only
> > > > > > > way
> > > > > > > > > to fix it is to write another piece of ugly code...
> > > > > > > > >
> > > > > > > > > https://issues.apache.org/jira/browse/HADOOP-13433
> > > > > > > > >
> > > > > > > > > 2016-09-23 12:53 GMT+08:00 Vladimir Rodionov <
> > > > > vladrodionov@gmail.com
> > > > > > >:
> > > > > > > > >
> > > > > > > > > > >> If in the future, we find better ways of doing this
> > > without
> > > > > > using
> > > > > > > > MR,
> > > > > > > > > we
> > > > > > > > > > can certainly consider that
> > > > > > > > > >
> > > > > > > > > > Our framework for distributed operations is abstract and
> > > allows
> > > > > > > > > > different implementations. MR is just one implementation
> we
> > > > > > provide.
> > > > > > > > > >
> > > > > > > > > > -Vlad
> > > > > > > > > >
> > > > > > > > > > On Thu, Sep 22, 2016 at 9:38 PM, Devaraj Das <
> > > > > ddas@hortonworks.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Guys, first off apologies for bringing in the topic of
> > > > MR-based
> > > > > > > > > > > compactions.. But I was thinking more about the
> > > SpliceMachine
> > > > > > > > approach
> > > > > > > > > of
> > > > > > > > > > > managing compactions in Spark where apparently they
> saw a
> > > lot
> > > > > of
> > > > > > > > > > benefits.
> > > > > > > > > > > Apologies for giving you that sore throat Andrew; I
> > really
> > > > > didn't
> > > > > > > > mean
> > > > > > > > > to
> > > > > > > > > > > :-)
> > > > > > > > > > >
> > > > > > > > > > > So on this issue, we have these on the plate:
> > > > > > > > > > > 0. Somehow not use MR but something like that
> > > > > > > > > > > 1. Run a standalone service other than master
> > > > > > > > > > > 2. Shell out from the master
> > > > > > > > > > >
> > > > > > > > > > > I don't think we have a good answer to (0), and I don't
> > > think
> > > > > > it's
> > > > > > > > even
> > > > > > > > > > > worth the effort of trying to build something when MR
> is
> > > > > already
> > > > > > > > there,
> > > > > > > > > > and
> > > > > > > > > > > being used by HBase already for some operations.
> > > > > > > > > > >
> > > > > > > > > > > On (1), we have to deal with a myriad of issues - HA of
> > the
> > > > > > server
> > > > > > > > not
> > > > > > > > > > > being the least of them all. Security (kerberos
> > > > authentication,
> > > > > > > > another
> > > > > > > > > > > keytab to manage, etc. etc. etc.). IMO, that approach
> is
> > > DOA.
> > > > > > > Instead
> > > > > > > > > > let's
> > > > > > > > > > > substitute that (1) with the HBase Master. I haven't
> seen
> > > any
> > > > > > good
> > > > > > > > > reason
> > > > > > > > > > > why the HBase master shouldn't launch MR jobs if
> needed.
> > > It's
> > > > > not
> > > > > > > > > ideal;
> > > > > > > > > > > agreed.
> > > > > > > > > > >
> > > > > > > > > > > Now before going to (2), let's see what are the
> benefits
> > of
> > > > > > running
> > > > > > > > the
> > > > > > > > > > > backup/restore jobs from the master. I think Ted has
> > > > summarized
> > > > > > > some
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > > issues that we need to take care of - basically, the
> > master
> > > > can
> > > > > > > keep
> > > > > > > > > > track
> > > > > > > > > > > of running jobs, and should it fail, the backup master
> > can
> > > > > > continue
> > > > > > > > > > keeping
> > > > > > > > > > > track of it (since the jobId would have been recorded
> in
> > > the
> > > > > proc
> > > > > > > > WAL).
> > > > > > > > > > The
> > > > > > > > > > > master can also do cleanup, etc. of failed
> backup/restore
> > > > > > > processes.
> > > > > > > > > > > Security is another issue - the job needs to run as
> > 'hbase'
> > > > > since
> > > > > > > it
> > > > > > > > > owns
> > > > > > > > > > > the data. Having the master launch the job makes it get
> > > that
> > > > > > > > privilege.
> > > > > > > > > > In
> > > > > > > > > > > the (2) approach, it's hard to do some of the above
> > > > management.
> > > > > > > > > > >
> > > > > > > > > > > Guys, just to reiterate, the patch as such is ready
> from
> > > the
> > > > > > > overall
> > > > > > > > > > > design/arch point of view (maybe code review is still
> > > pending
> > > > > > from
> > > > > > > > > > Matteo).
> > > > > > > > > > > If in the future, we find better ways of doing this
> > without
> > > > > using
> > > > > > > MR,
> > > > > > > > > we
> > > > > > > > > > > can certainly consider that. But IMO don't think we
> > should
> > > > > block
> > > > > > > this
> > > > > > > > > > patch
> > > > > > > > > > > from getting merged.
> > > > > > > > > > >
> > > > > > > > > > > ________________________________________
> > > > > > > > > > > From: 张铎 <pa...@gmail.com>
> > > > > > > > > > > Sent: Thursday, September 22, 2016 8:32 PM
> > > > > > > > > > > To: dev@hbase.apache.org
> > > > > > > > > > > Subject: Re: [DISCUSSION] MR jobs started by Master or
> RS
> > > > > > > > > > >
> > > > > > > > > > > So what about a standalone service other than master?
> You
> > > can
> > > > > use
> > > > > > > > your
> > > > > > > > > > own
> > > > > > > > > > > procedure store in that service?
> > > > > > > > > > >
> > > > > > > > > > > 2016-09-23 11:28 GMT+08:00 Ted Yu <yuzhihong@gmail.com
> >:
> > > > > > > > > > >
> > > > > > > > > > > > An earlier implementation was client driven.
> > > > > > > > > > > >
> > > > > > > > > > > > But with that approach, it is hard to resume if there
> > is
> > > > > error
> > > > > > > > > midway.
> > > > > > > > > > > > Using Procedure V2 makes the backup / restore more
> > > robust.
> > > > > > > > > > > >
> > > > > > > > > > > > Another consideration is for security. It is hard to
> > > > enforce
> > > > > > > > security
> > > > > > > > > > (to
> > > > > > > > > > > > be implemented) for client driven actions.
> > > > > > > > > > > >
> > > > > > > > > > > > Cheers
> > > > > > > > > > > >
> > > > > > > > > > > > > On Sep 22, 2016, at 8:15 PM, Andrew Purtell <
> > > > > > > > > > andrew.purtell@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > No, this misses Matteo's finer point, which is
> > > "shelling
> > > > > out"
> > > > > > > > from
> > > > > > > > > > the
> > > > > > > > > > > > master directly to run MR is a first. Why not drive
> > this
> > > > > with a
> > > > > > > > > utility
> > > > > > > > > > > > derived from Tool?
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Sep 22, 2016, at 7:57 PM, Vladimir Rodionov <
> > > > > > > > > > vladrodionov@gmail.com
> > > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > >>>> In our production cluster, it is a common case
> we
> > > > just
> > > > > > have
> > > > > > > > > HDFS
> > > > > > > > > > > and
> > > > > > > > > > > > >>>> HBase deployed.
> > > > > > > > > > > > >>>> If our Master/RS depend on MR framework
> > (especially
> > > > some
> > > > > > > > > features
> > > > > > > > > > we
> > > > > > > > > > > > >>>> have not used at all), it introduced another
> cost
> > > for
> > > > > > > > maintain.
> > > > > > > > > > I
> > > > > > > > > > > > >>>> don't think it is a good idea.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> So , you are not backup users in this case. Many
> our
> > > > > > customers
> > > > > > > > > have
> > > > > > > > > > > full
> > > > > > > > > > > > >> stack deployed and
> > > > > > > > > > > > >> want see backup to be a standard feature. Besides
> > > this,
> > > > > > > nothing
> > > > > > > > > will
> > > > > > > > > > > > happen
> > > > > > > > > > > > >> in your cluster
> > > > > > > > > > > > >> if you won't be doing backups.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> This discussion (we do not want see M/R
> dependency)
> > > goes
> > > > > to
> > > > > > > > > nowhere.
> > > > > > > > > > > We
> > > > > > > > > > > > >> asked already, at least twice, to suggest another
> > > > > framework
> > > > > > > > (other
> > > > > > > > > > > than
> > > > > > > > > > > > M/R)
> > > > > > > > > > > > >> for bulk data copy with *conversion*. Still
> waiting
> > > for
> > > > > > > > > suggestions.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> -Vlad
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>> On Thu, Sep 22, 2016 at 7:49 PM, Ted Yu <
> > > > > > yuzhihong@gmail.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>> If MR framework is not deployed in the cluster,
> > hbase
> > > > > still
> > > > > > > > > > functions
> > > > > > > > > > > > >>> normally (post merge).
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>> In terms of build time dependency, we have long
> > been
> > > > > > > depending
> > > > > > > > on
> > > > > > > > > > > > >>> mapreduce. Take a look at ExportSnapshot.
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>> Cheers
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>> On Thu, Sep 22, 2016 at 7:42 PM, Heng Chen <
> > > > > > > > > > heng.chen.1986@gmail.com
> > > > > > > > > > > >
> > > > > > > > > > > > >>> wrote:
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>>> In our production cluster, it is a common case
> we
> > > > just
> > > > > > have
> > > > > > > > > HDFS
> > > > > > > > > > > and
> > > > > > > > > > > > >>>> HBase deployed.
> > > > > > > > > > > > >>>> If our Master/RS depend on MR framework
> > (especially
> > > > some
> > > > > > > > > features
> > > > > > > > > > we
> > > > > > > > > > > > >>>> have not used at all), it introduced another
> cost
> > > for
> > > > > > > > maintain.
> > > > > > > > > > I
> > > > > > > > > > > > >>>> don't think it is a good idea.
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>> 2016-09-23 10:28 GMT+08:00 张铎 <
> > > palomino219@gmail.com
> > > > >:
> > > > > > > > > > > > >>>>> To be specific, for example, our nice
> > > Backup/Restore
> > > > > > > feature,
> > > > > > > > > if
> > > > > > > > > > we
> > > > > > > > > > > > >>> think
> > > > > > > > > > > > >>>>> this is not a core feature of HBase, then we
> > could
> > > > make
> > > > > > it
> > > > > > > > > depend
> > > > > > > > > > > on
> > > > > > > > > > > > >>> MR,
> > > > > > > > > > > > >>>>> and start a standalone BackupManager instance
> > that
> > > > > > submits
> > > > > > > MR
> > > > > > > > > > jobs
> > > > > > > > > > > to
> > > > > > > > > > > > >>> do
> > > > > > > > > > > > >>>>> periodical maintenance job. And if we think
> this
> > > is a
> > > > > > core
> > > > > > > > > > feature
> > > > > > > > > > > > that
> > > > > > > > > > > > >>>>> everyone should use it, then we'd better
> > implement
> > > it
> > > > > > > without
> > > > > > > > > MR
> > > > > > > > > > > > >>>>> dependency, like DLS.
> > > > > > > > > > > > >>>>>
> > > > > > > > > > > > >>>>> Thanks.
> > > > > > > > > > > > >>>>>
> > > > > > > > > > > > >>>>> 2016-09-23 10:11 GMT+08:00 张铎 <
> > > palomino219@gmail.com
> > > > >:
> > > > > > > > > > > > >>>>>
> > > > > > > > > > > > >>>>>> I‘m -1 on let master or rs launch MR jobs. It
> is
> > > OK
> > > > > that
> > > > > > > > some
> > > > > > > > > of
> > > > > > > > > > > our
> > > > > > > > > > > > >>>>>> features depend on MR but I think the bottom
> > line
> > > is
> > > > > > that
> > > > > > > we
> > > > > > > > > > > should
> > > > > > > > > > > > >>>> launch
> > > > > > > > > > > > >>>>>> the jobs from outside manually or by other
> > > services.
> > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > >>>>>> 2016-09-23 9:47 GMT+08:00 Andrew Purtell <
> > > > > > > > > > > andrew.purtell@gmail.com
> > > > > > > > > > > > >:
> > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > >>>>>>> Ok, got it. Well "shelling out" is on the
> line
> > I
> > > > > think,
> > > > > > > so
> > > > > > > > a
> > > > > > > > > > fair
> > > > > > > > > > > > >>>>>>> question.
> > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > >>>>>>> Can this be driven by a utility derived from
> > Tool
> > > > > like
> > > > > > > our
> > > > > > > > > > other
> > > > > > > > > > > MR
> > > > > > > > > > > > >>>> apps?
> > > > > > > > > > > > >>>>>>> The issue is needing the AccessController to
> > > decide
> > > > > if
> > > > > > > > > allowed?
> > > > > > > > > > > But
> > > > > > > > > > > > >>>> nothing
> > > > > > > > > > > > >>>>>>> prevents the user from running the job
> > > > > > > > > manually/independently,
> > > > > > > > > > > > right?
> > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > >>>>>>>> On Sep 22, 2016, at 3:44 PM, Matteo
> Bertozzi <
> > > > > > > > > > > > >>>> theo.bertozzi@gmail.com>
> > > > > > > > > > > > >>>>>>> wrote:
> > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > >>>>>>>> just a remark. my query was not about tools
> > > using
> > > > MR
> > > > > > > > > > (everyone i
> > > > > > > > > > > > >>>> think
> > > > > > > > > > > > >>>>>>> is
> > > > > > > > > > > > >>>>>>>> ok with those).
> > > > > > > > > > > > >>>>>>>> the topic was about: "are we ok with running
> > MR
> > > > jobs
> > > > > > > from
> > > > > > > > > > Master
> > > > > > > > > > > > >>> and
> > > > > > > > > > > > >>>> RSs
> > > > > > > > > > > > >>>>>>>> code?" since this will be the first time we
> do
> > > > this
> > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > >>>>>>>> Matteo
> > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > >>>>>>>>> On Thu, Sep 22, 2016 at 2:49 PM, Devaraj
> Das
> > <
> > > > > > > > > > > > >>> ddas@hortonworks.com>
> > > > > > > > > > > > >>>>>>> wrote:
> > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > >>>>>>>>> Very much agree; for tools like
> > ExportSnapshot
> > > /
> > > > > > > Backup /
> > > > > > > > > > > > Restore,
> > > > > > > > > > > > >>>> it's
> > > > > > > > > > > > >>>>>>>>> fine to be dependent on MR. MR is the right
> > > > > framework
> > > > > > > for
> > > > > > > > > > such.
> > > > > > > > > > > > We
> > > > > > > > > > > > >>>>>>> should
> > > > > > > > > > > > >>>>>>>>> also do compactions using MR (just saying
> :)
> > )
> > > > > > > > > > > > >>>>>>>>> ________________________________________
> > > > > > > > > > > > >>>>>>>>> From: Ted Yu <yu...@gmail.com>
> > > > > > > > > > > > >>>>>>>>> Sent: Thursday, September 22, 2016 2:00 PM
> > > > > > > > > > > > >>>>>>>>> To: dev@hbase.apache.org
> > > > > > > > > > > > >>>>>>>>> Subject: Re: [DISCUSSION] MR jobs started
> by
> > > > Master
> > > > > > or
> > > > > > > RS
> > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > >>>>>>>>> I agree - backup / restore is in the same
> > > > category
> > > > > as
> > > > > > > > > import
> > > > > > > > > > /
> > > > > > > > > > > > >>>> export.
> > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > >>>>>>>>> On Thu, Sep 22, 2016 at 1:58 PM, Andrew
> > > Purtell <
> > > > > > > > > > > > >>>>>>> andrew.purtell@gmail.com>
> > > > > > > > > > > > >>>>>>>>> wrote:
> > > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>> Backup is extra tooling around core in my
> > > > opinion.
> > > > > > > Like
> > > > > > > > > > import
> > > > > > > > > > > > or
> > > > > > > > > > > > >>>>>>> export.
> > > > > > > > > > > > >>>>>>>>>> Or the optional MOB tool. It's fine.
> > > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>> On Sep 22, 2016, at 1:50 PM, Matteo
> > Bertozzi
> > > <
> > > > > > > > > > > > >>>> mbertozzi@apache.org>
> > > > > > > > > > > > >>>>>>>>>> wrote:
> > > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>> What's the latest opinion around running
> MR
> > > > jobs
> > > > > > from
> > > > > > > > > hbase
> > > > > > > > > > > > >>>> (Master
> > > > > > > > > > > > >>>>>>> or
> > > > > > > > > > > > >>>>>>>>>> RS)?
> > > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>> I remember in the past that there was
> > > > discussion
> > > > > > > about
> > > > > > > > > not
> > > > > > > > > > > > >>> having
> > > > > > > > > > > > >>>> MR
> > > > > > > > > > > > >>>>>>>>> has
> > > > > > > > > > > > >>>>>>>>>>> direct dependency of hbase.
> > > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>> I think some of discussion where around
> MOB
> > > > that
> > > > > > had
> > > > > > > a
> > > > > > > > MR
> > > > > > > > > > job
> > > > > > > > > > > > to
> > > > > > > > > > > > >>>>>>>>> compact,
> > > > > > > > > > > > >>>>>>>>>>> that later was transformed in a non-MR
> job
> > to
> > > > be
> > > > > > > > merged,
> > > > > > > > > I
> > > > > > > > > > > > think
> > > > > > > > > > > > >>>> we
> > > > > > > > > > > > >>>>>>>>> had a
> > > > > > > > > > > > >>>>>>>>>>> similar discussion for log split/replay.
> > > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>> the latest is the new Backup feature
> > > > > (HBASE-7912),
> > > > > > > that
> > > > > > > > > > runs
> > > > > > > > > > > a
> > > > > > > > > > > > >>> MR
> > > > > > > > > > > > >>>> job
> > > > > > > > > > > > >>>>>>>>>> from
> > > > > > > > > > > > >>>>>>>>>>> the master to copy data or restore data.
> > > > > > > > > > > > >>>>>>>>>>> (backup is also "not really core" as in..
> > if
> > > > you
> > > > > > > don't
> > > > > > > > > use
> > > > > > > > > > > > >>> backup
> > > > > > > > > > > > >>>>>>>>> you'll
> > > > > > > > > > > > >>>>>>>>>>> not end up running MR jobs, but this was
> > > > probably
> > > > > > > true
> > > > > > > > > for
> > > > > > > > > > > MOB
> > > > > > > > > > > > >>> as
> > > > > > > > > > > > >>>> in
> > > > > > > > > > > > >>>>>>>>> "if
> > > > > > > > > > > > >>>>>>>>>>> you don't enable MOB you don't need MR")
> > > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > > >>>>>>>>>>> any thoughts? do we a rule that says "we
> > > don't
> > > > > want
> > > > > > > to
> > > > > > > > > have
> > > > > > > > > > > > >>> hbase
> > > > > > > > > > > > >>>> run
> > > > > > > > > > > > >>>>>>>>> MR
> > > > > > > > > > > > >>>>>>>>>>> jobs, only tool started manually by the
> > user
> > > > can
> > > > > do
> > > > > > > > > that".
> > > > > > > > > > or
> > > > > > > > > > > > >>> can
> > > > > > > > > > > > >>>> we
> > > > > > > > > > > > >>>>>>>>>> start
> > > > > > > > > > > > >>>>>>>>>>> adding MR calls around without problems?
> > > > > > > > > > > > >>>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: [DISCUSSION] MR jobs started by Master or RS
Posted by Vladimir Rodionov <vl...@gmail.com>.
>> -1 on that backup be in core hbase
Not sure I understand what it means.
1. We are not allowed to use Master to orchestrate the whole process? We
have already brought up all advantages of using
Master and distributed procedures for backup and restore.
Downside of moving this to client tool is lack of fault tolerance:
1.1 Client won't be allowed to do any operations, that can, potentially affect
cluster, such as disabling splits/merges, balancer.
1.2 In case of client failure who will be doing the whole rollback stuff?
We are trying to make it atomic.
Security is not clear.
2. We are not allowed to modify code of existing HBase core classes (what
does core mean anyway)?
3. We are not allowed to create backup system table (hbase:backup) in a
system space? Only in user space? The table is global.
2. is critical. Despite the fact, that 95% of code is new, we have touched,
of course some existing HBase code.
3. is not that critical, of course we can move backup system into user
space.
And finally, will moving backup into external tool give us +1 from stack?
-Vlad
On Fri, Sep 23, 2016 at 11:26 AM, Stack <st...@duboce.net> wrote:
> On Fri, Sep 23, 2016 at 11:22 AM, Vladimir Rodionov <
> vladrodionov@gmail.com>
> wrote:
>
> > >> + MR is dead
> >
> > Does MR know that? :)
> >
> > Again. With all due respect, stack - still no suggestions what should we
> > use for "bulk data move and transformation" instead of MR?
> >
>
> Use whatever distributed engine suits your fancy -- MR, Spark, distributed
> shell -- just don't have HBase core depend on it, even optionally.
>
>
> > I suggest voting first on "do we need backup in HBase"? In my opinion,
> some
> > group members still not sure about that and some will give -1
> > in any case. Just because ...
> >
> >
> We could run a vote, sure. -1 on that backup be in core hbase (+1 on adding
> all the API any such external tool might need to run).
>
> St.Ack
>
>
>
> > -Vlad
> >
> >
> >
> >
> >
> >
> > On Fri, Sep 23, 2016 at 10:57 AM, Stack <st...@duboce.net> wrote:
> >
> > > On Fri, Sep 23, 2016 at 6:46 AM, Matteo Bertozzi <
> > theo.bertozzi@gmail.com>
> > > wrote:
> > >
> > > > let me try to go back to my original topic.
> > > > this question was meant to be generic, and provide some rule for
> future
> > > > code.
> > > >
> > > > from what I can gather, a rule that may satisfy everyone can be:
> > > > - we don't want any core feature (e.g. compaction/log-split/log-
> > reply)
> > > > over MR, because some cluster may not want or may have an
> > > > external/uncontrolled MR setup.
> > > >
> > >
> > > +1
> > >
> > >
> > > > - we allow non-core features (e.g. features enabled by a flag) to
> run
> > MR
> > > > jobs from hbase, because unless you use the feature, MR is not
> > required.
> > > >
> > > >
> > > -1 to hbase core depending on MR or core -- whether behind a flag or
> not
> > --
> > > ever being able to launch MR jobs.
> > >
> > > + MR is dead. We should be busy working hard to undo it from
> hbase-server
> > > moving it out to be an optional module (Spark would be its peer).
> > > + Master is a rats nest of state. Matteo, Stephen, and Appy are busy
> > > working hard on moving it up on to a new foundation. Lets not clutter
> > task
> > > harder by piling on more moving parts.
> > >
> > > St.Ack
> > >
> > >
> > > > Matteo
> > > >
> > > >
> > > > On Fri, Sep 23, 2016 at 5:39 AM, Ted Yu <yu...@gmail.com> wrote:
> > > >
> > > > > I suggest you look at Matteo's work for AssignmentManager which is
> to
> > > > make
> > > > > Master more stable.
> > > > >
> > > > > Cheers
> > > > >
> > > > > On Fri, Sep 23, 2016 at 5:32 AM, 张铎 <pa...@gmail.com> wrote:
> > > > >
> > > > > > No, not your fault, at lease, not this time:)
> > > > > >
> > > > > > Why I call the code ugly? Can you simply tell me the sequence of
> > > calls
> > > > > when
> > > > > > starting up the HMaster? HMaster is also a regionserver so it
> > extends
> > > > > > HRegionServer, and the initialization of HRegionServer sometimes
> > > needs
> > > > to
> > > > > > make rpc calls to HMaster. A simple change would cause
> > probabilistic
> > > > dead
> > > > > > lock or some strange NPEs...
> > > > > >
> > > > > > That's why I'm very nervous when somebody wants to add new
> features
> > > or
> > > > > add
> > > > > > external dependencies to HMaster, especially add more works for
> the
> > > > start
> > > > > > up processing...
> > > > > >
> > > > > > Thanks.
> > > > > >
> > > > > > 2016-09-23 20:02 GMT+08:00 Ted Yu <yu...@gmail.com>:
> > > > > >
> > > > > > > I read through HADOOP-13433
> > > > > > > <https://issues.apache.org/jira/browse/HADOOP-13433> - the
> cited
> > > > race
> > > > > > > condition is in jdk.
> > > > > > >
> > > > > > > Suggest pinging the reviewer on JIRA to get it moving.
> > > > > > >
> > > > > > > bq. But the ugly code in HMaster is readlly a problem...
> > > > > > >
> > > > > > > Can you be specific as to which code is ugly ? Is it in the
> > backup
> > > /
> > > > > > > restore mega patch ?
> > > > > > >
> > > > > > > Cheers
> > > > > > >
> > > > > > > On Thu, Sep 22, 2016 at 10:44 PM, 张铎 <pa...@gmail.com>
> > > wrote:
> > > > > > >
> > > > > > > > If you guys have already implemented the feature in the MR
> way
> > > and
> > > > > the
> > > > > > > > patch is ready for landing on master, I'm a -0 on it as I do
> > not
> > > > want
> > > > > > to
> > > > > > > > block the development progress.
> > > > > > > >
> > > > > > > > But I strongly suggest later we need to revisit the design
> and
> > > see
> > > > if
> > > > > > we
> > > > > > > > can seperated the logic from HMaster as much as possible. HA
> is
> > > > not a
> > > > > > big
> > > > > > > > problem if you do not store any metada locally. But the ugly
> > code
> > > > in
> > > > > > > > HMaster is readlly a problem...
> > > > > > > >
> > > > > > > > And for security, I have a issue pending for a long time. Can
> > > > someone
> > > > > > > help
> > > > > > > > taking a simple look at it? This is what I mean, ugly code...
> > > > logout
> > > > > > and
> > > > > > > > destroy the credentials in a subject when it is still being
> > used,
> > > > and
> > > > > > > > declared as LimitPrivacy so I can not change the behivor and
> > the
> > > > only
> > > > > > way
> > > > > > > > to fix it is to write another piece of ugly code...
> > > > > > > >
> > > > > > > > https://issues.apache.org/jira/browse/HADOOP-13433
> > > > > > > >
> > > > > > > > 2016-09-23 12:53 GMT+08:00 Vladimir Rodionov <
> > > > vladrodionov@gmail.com
> > > > > >:
> > > > > > > >
> > > > > > > > > >> If in the future, we find better ways of doing this
> > without
> > > > > using
> > > > > > > MR,
> > > > > > > > we
> > > > > > > > > can certainly consider that
> > > > > > > > >
> > > > > > > > > Our framework for distributed operations is abstract and
> > allows
> > > > > > > > > different implementations. MR is just one implementation we
> > > > > provide.
> > > > > > > > >
> > > > > > > > > -Vlad
> > > > > > > > >
> > > > > > > > > On Thu, Sep 22, 2016 at 9:38 PM, Devaraj Das <
> > > > ddas@hortonworks.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Guys, first off apologies for bringing in the topic of
> > > MR-based
> > > > > > > > > > compactions.. But I was thinking more about the
> > SpliceMachine
> > > > > > > approach
> > > > > > > > of
> > > > > > > > > > managing compactions in Spark where apparently they saw a
> > lot
> > > > of
> > > > > > > > > benefits.
> > > > > > > > > > Apologies for giving you that sore throat Andrew; I
> really
> > > > didn't
> > > > > > > mean
> > > > > > > > to
> > > > > > > > > > :-)
> > > > > > > > > >
> > > > > > > > > > So on this issue, we have these on the plate:
> > > > > > > > > > 0. Somehow not use MR but something like that
> > > > > > > > > > 1. Run a standalone service other than master
> > > > > > > > > > 2. Shell out from the master
> > > > > > > > > >
> > > > > > > > > > I don't think we have a good answer to (0), and I don't
> > think
> > > > > it's
> > > > > > > even
> > > > > > > > > > worth the effort of trying to build something when MR is
> > > > already
> > > > > > > there,
> > > > > > > > > and
> > > > > > > > > > being used by HBase already for some operations.
> > > > > > > > > >
> > > > > > > > > > On (1), we have to deal with a myriad of issues - HA of
> the
> > > > > server
> > > > > > > not
> > > > > > > > > > being the least of them all. Security (kerberos
> > > authentication,
> > > > > > > another
> > > > > > > > > > keytab to manage, etc. etc. etc.). IMO, that approach is
> > DOA.
> > > > > > Instead
> > > > > > > > > let's
> > > > > > > > > > substitute that (1) with the HBase Master. I haven't seen
> > any
> > > > > good
> > > > > > > > reason
> > > > > > > > > > why the HBase master shouldn't launch MR jobs if needed.
> > It's
> > > > not
> > > > > > > > ideal;
> > > > > > > > > > agreed.
> > > > > > > > > >
> > > > > > > > > > Now before going to (2), let's see what are the benefits
> of
> > > > > running
> > > > > > > the
> > > > > > > > > > backup/restore jobs from the master. I think Ted has
> > > summarized
> > > > > > some
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > issues that we need to take care of - basically, the
> master
> > > can
> > > > > > keep
> > > > > > > > > track
> > > > > > > > > > of running jobs, and should it fail, the backup master
> can
> > > > > continue
> > > > > > > > > keeping
> > > > > > > > > > track of it (since the jobId would have been recorded in
> > the
> > > > proc
> > > > > > > WAL).
> > > > > > > > > The
> > > > > > > > > > master can also do cleanup, etc. of failed backup/restore
> > > > > > processes.
> > > > > > > > > > Security is another issue - the job needs to run as
> 'hbase'
> > > > since
> > > > > > it
> > > > > > > > owns
> > > > > > > > > > the data. Having the master launch the job makes it get
> > that
> > > > > > > privilege.
> > > > > > > > > In
> > > > > > > > > > the (2) approach, it's hard to do some of the above
> > > management.
> > > > > > > > > >
> > > > > > > > > > Guys, just to reiterate, the patch as such is ready from
> > the
> > > > > > overall
> > > > > > > > > > design/arch point of view (maybe code review is still
> > pending
> > > > > from
> > > > > > > > > Matteo).
> > > > > > > > > > If in the future, we find better ways of doing this
> without
> > > > using
> > > > > > MR,
> > > > > > > > we
> > > > > > > > > > can certainly consider that. But IMO don't think we
> should
> > > > block
> > > > > > this
> > > > > > > > > patch
> > > > > > > > > > from getting merged.
> > > > > > > > > >
> > > > > > > > > > ________________________________________
> > > > > > > > > > From: 张铎 <pa...@gmail.com>
> > > > > > > > > > Sent: Thursday, September 22, 2016 8:32 PM
> > > > > > > > > > To: dev@hbase.apache.org
> > > > > > > > > > Subject: Re: [DISCUSSION] MR jobs started by Master or RS
> > > > > > > > > >
> > > > > > > > > > So what about a standalone service other than master? You
> > can
> > > > use
> > > > > > > your
> > > > > > > > > own
> > > > > > > > > > procedure store in that service?
> > > > > > > > > >
> > > > > > > > > > 2016-09-23 11:28 GMT+08:00 Ted Yu <yu...@gmail.com>:
> > > > > > > > > >
> > > > > > > > > > > An earlier implementation was client driven.
> > > > > > > > > > >
> > > > > > > > > > > But with that approach, it is hard to resume if there
> is
> > > > error
> > > > > > > > midway.
> > > > > > > > > > > Using Procedure V2 makes the backup / restore more
> > robust.
> > > > > > > > > > >
> > > > > > > > > > > Another consideration is for security. It is hard to
> > > enforce
> > > > > > > security
> > > > > > > > > (to
> > > > > > > > > > > be implemented) for client driven actions.
> > > > > > > > > > >
> > > > > > > > > > > Cheers
> > > > > > > > > > >
> > > > > > > > > > > > On Sep 22, 2016, at 8:15 PM, Andrew Purtell <
> > > > > > > > > andrew.purtell@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > No, this misses Matteo's finer point, which is
> > "shelling
> > > > out"
> > > > > > > from
> > > > > > > > > the
> > > > > > > > > > > master directly to run MR is a first. Why not drive
> this
> > > > with a
> > > > > > > > utility
> > > > > > > > > > > derived from Tool?
> > > > > > > > > > > >
> > > > > > > > > > > > On Sep 22, 2016, at 7:57 PM, Vladimir Rodionov <
> > > > > > > > > vladrodionov@gmail.com
> > > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > >>>> In our production cluster, it is a common case we
> > > just
> > > > > have
> > > > > > > > HDFS
> > > > > > > > > > and
> > > > > > > > > > > >>>> HBase deployed.
> > > > > > > > > > > >>>> If our Master/RS depend on MR framework
> (especially
> > > some
> > > > > > > > features
> > > > > > > > > we
> > > > > > > > > > > >>>> have not used at all), it introduced another cost
> > for
> > > > > > > maintain.
> > > > > > > > > I
> > > > > > > > > > > >>>> don't think it is a good idea.
> > > > > > > > > > > >>
> > > > > > > > > > > >> So , you are not backup users in this case. Many our
> > > > > customers
> > > > > > > > have
> > > > > > > > > > full
> > > > > > > > > > > >> stack deployed and
> > > > > > > > > > > >> want see backup to be a standard feature. Besides
> > this,
> > > > > > nothing
> > > > > > > > will
> > > > > > > > > > > happen
> > > > > > > > > > > >> in your cluster
> > > > > > > > > > > >> if you won't be doing backups.
> > > > > > > > > > > >>
> > > > > > > > > > > >> This discussion (we do not want see M/R dependency)
> > goes
> > > > to
> > > > > > > > nowhere.
> > > > > > > > > > We
> > > > > > > > > > > >> asked already, at least twice, to suggest another
> > > > framework
> > > > > > > (other
> > > > > > > > > > than
> > > > > > > > > > > M/R)
> > > > > > > > > > > >> for bulk data copy with *conversion*. Still waiting
> > for
> > > > > > > > suggestions.
> > > > > > > > > > > >>
> > > > > > > > > > > >> -Vlad
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > > >>> On Thu, Sep 22, 2016 at 7:49 PM, Ted Yu <
> > > > > yuzhihong@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> If MR framework is not deployed in the cluster,
> hbase
> > > > still
> > > > > > > > > functions
> > > > > > > > > > > >>> normally (post merge).
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> In terms of build time dependency, we have long
> been
> > > > > > depending
> > > > > > > on
> > > > > > > > > > > >>> mapreduce. Take a look at ExportSnapshot.
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> Cheers
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> On Thu, Sep 22, 2016 at 7:42 PM, Heng Chen <
> > > > > > > > > heng.chen.1986@gmail.com
> > > > > > > > > > >
> > > > > > > > > > > >>> wrote:
> > > > > > > > > > > >>>
> > > > > > > > > > > >>>> In our production cluster, it is a common case we
> > > just
> > > > > have
> > > > > > > > HDFS
> > > > > > > > > > and
> > > > > > > > > > > >>>> HBase deployed.
> > > > > > > > > > > >>>> If our Master/RS depend on MR framework
> (especially
> > > some
> > > > > > > > features
> > > > > > > > > we
> > > > > > > > > > > >>>> have not used at all), it introduced another cost
> > for
> > > > > > > maintain.
> > > > > > > > > I
> > > > > > > > > > > >>>> don't think it is a good idea.
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> 2016-09-23 10:28 GMT+08:00 张铎 <
> > palomino219@gmail.com
> > > >:
> > > > > > > > > > > >>>>> To be specific, for example, our nice
> > Backup/Restore
> > > > > > feature,
> > > > > > > > if
> > > > > > > > > we
> > > > > > > > > > > >>> think
> > > > > > > > > > > >>>>> this is not a core feature of HBase, then we
> could
> > > make
> > > > > it
> > > > > > > > depend
> > > > > > > > > > on
> > > > > > > > > > > >>> MR,
> > > > > > > > > > > >>>>> and start a standalone BackupManager instance
> that
> > > > > submits
> > > > > > MR
> > > > > > > > > jobs
> > > > > > > > > > to
> > > > > > > > > > > >>> do
> > > > > > > > > > > >>>>> periodical maintenance job. And if we think this
> > is a
> > > > > core
> > > > > > > > > feature
> > > > > > > > > > > that
> > > > > > > > > > > >>>>> everyone should use it, then we'd better
> implement
> > it
> > > > > > without
> > > > > > > > MR
> > > > > > > > > > > >>>>> dependency, like DLS.
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>> Thanks.
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>> 2016-09-23 10:11 GMT+08:00 张铎 <
> > palomino219@gmail.com
> > > >:
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>>> I‘m -1 on let master or rs launch MR jobs. It is
> > OK
> > > > that
> > > > > > > some
> > > > > > > > of
> > > > > > > > > > our
> > > > > > > > > > > >>>>>> features depend on MR but I think the bottom
> line
> > is
> > > > > that
> > > > > > we
> > > > > > > > > > should
> > > > > > > > > > > >>>> launch
> > > > > > > > > > > >>>>>> the jobs from outside manually or by other
> > services.
> > > > > > > > > > > >>>>>>
> > > > > > > > > > > >>>>>> 2016-09-23 9:47 GMT+08:00 Andrew Purtell <
> > > > > > > > > > andrew.purtell@gmail.com
> > > > > > > > > > > >:
> > > > > > > > > > > >>>>>>
> > > > > > > > > > > >>>>>>> Ok, got it. Well "shelling out" is on the line
> I
> > > > think,
> > > > > > so
> > > > > > > a
> > > > > > > > > fair
> > > > > > > > > > > >>>>>>> question.
> > > > > > > > > > > >>>>>>>
> > > > > > > > > > > >>>>>>> Can this be driven by a utility derived from
> Tool
> > > > like
> > > > > > our
> > > > > > > > > other
> > > > > > > > > > MR
> > > > > > > > > > > >>>> apps?
> > > > > > > > > > > >>>>>>> The issue is needing the AccessController to
> > decide
> > > > if
> > > > > > > > allowed?
> > > > > > > > > > But
> > > > > > > > > > > >>>> nothing
> > > > > > > > > > > >>>>>>> prevents the user from running the job
> > > > > > > > manually/independently,
> > > > > > > > > > > right?
> > > > > > > > > > > >>>>>>>
> > > > > > > > > > > >>>>>>>> On Sep 22, 2016, at 3:44 PM, Matteo Bertozzi <
> > > > > > > > > > > >>>> theo.bertozzi@gmail.com>
> > > > > > > > > > > >>>>>>> wrote:
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>>> just a remark. my query was not about tools
> > using
> > > MR
> > > > > > > > > (everyone i
> > > > > > > > > > > >>>> think
> > > > > > > > > > > >>>>>>> is
> > > > > > > > > > > >>>>>>>> ok with those).
> > > > > > > > > > > >>>>>>>> the topic was about: "are we ok with running
> MR
> > > jobs
> > > > > > from
> > > > > > > > > Master
> > > > > > > > > > > >>> and
> > > > > > > > > > > >>>> RSs
> > > > > > > > > > > >>>>>>>> code?" since this will be the first time we do
> > > this
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>>> Matteo
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > >>>>>>>>> On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das
> <
> > > > > > > > > > > >>> ddas@hortonworks.com>
> > > > > > > > > > > >>>>>>> wrote:
> > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > >>>>>>>>> Very much agree; for tools like
> ExportSnapshot
> > /
> > > > > > Backup /
> > > > > > > > > > > Restore,
> > > > > > > > > > > >>>> it's
> > > > > > > > > > > >>>>>>>>> fine to be dependent on MR. MR is the right
> > > > framework
> > > > > > for
> > > > > > > > > such.
> > > > > > > > > > > We
> > > > > > > > > > > >>>>>>> should
> > > > > > > > > > > >>>>>>>>> also do compactions using MR (just saying :)
> )
> > > > > > > > > > > >>>>>>>>> ________________________________________
> > > > > > > > > > > >>>>>>>>> From: Ted Yu <yu...@gmail.com>
> > > > > > > > > > > >>>>>>>>> Sent: Thursday, September 22, 2016 2:00 PM
> > > > > > > > > > > >>>>>>>>> To: dev@hbase.apache.org
> > > > > > > > > > > >>>>>>>>> Subject: Re: [DISCUSSION] MR jobs started by
> > > Master
> > > > > or
> > > > > > RS
> > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > >>>>>>>>> I agree - backup / restore is in the same
> > > category
> > > > as
> > > > > > > > import
> > > > > > > > > /
> > > > > > > > > > > >>>> export.
> > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > >>>>>>>>> On Thu, Sep 22, 2016 at 1:58 PM, Andrew
> > Purtell <
> > > > > > > > > > > >>>>>>> andrew.purtell@gmail.com>
> > > > > > > > > > > >>>>>>>>> wrote:
> > > > > > > > > > > >>>>>>>>>
> > > > > > > > > > > >>>>>>>>>> Backup is extra tooling around core in my
> > > opinion.
> > > > > > Like
> > > > > > > > > import
> > > > > > > > > > > or
> > > > > > > > > > > >>>>>>> export.
> > > > > > > > > > > >>>>>>>>>> Or the optional MOB tool. It's fine.
> > > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>> On Sep 22, 2016, at 1:50 PM, Matteo
> Bertozzi
> > <
> > > > > > > > > > > >>>> mbertozzi@apache.org>
> > > > > > > > > > > >>>>>>>>>> wrote:
> > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>> What's the latest opinion around running MR
> > > jobs
> > > > > from
> > > > > > > > hbase
> > > > > > > > > > > >>>> (Master
> > > > > > > > > > > >>>>>>> or
> > > > > > > > > > > >>>>>>>>>> RS)?
> > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>> I remember in the past that there was
> > > discussion
> > > > > > about
> > > > > > > > not
> > > > > > > > > > > >>> having
> > > > > > > > > > > >>>> MR
> > > > > > > > > > > >>>>>>>>> has
> > > > > > > > > > > >>>>>>>>>>> direct dependency of hbase.
> > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>> I think some of discussion where around MOB
> > > that
> > > > > had
> > > > > > a
> > > > > > > MR
> > > > > > > > > job
> > > > > > > > > > > to
> > > > > > > > > > > >>>>>>>>> compact,
> > > > > > > > > > > >>>>>>>>>>> that later was transformed in a non-MR job
> to
> > > be
> > > > > > > merged,
> > > > > > > > I
> > > > > > > > > > > think
> > > > > > > > > > > >>>> we
> > > > > > > > > > > >>>>>>>>> had a
> > > > > > > > > > > >>>>>>>>>>> similar discussion for log split/replay.
> > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>> the latest is the new Backup feature
> > > > (HBASE-7912),
> > > > > > that
> > > > > > > > > runs
> > > > > > > > > > a
> > > > > > > > > > > >>> MR
> > > > > > > > > > > >>>> job
> > > > > > > > > > > >>>>>>>>>> from
> > > > > > > > > > > >>>>>>>>>>> the master to copy data or restore data.
> > > > > > > > > > > >>>>>>>>>>> (backup is also "not really core" as in..
> if
> > > you
> > > > > > don't
> > > > > > > > use
> > > > > > > > > > > >>> backup
> > > > > > > > > > > >>>>>>>>> you'll
> > > > > > > > > > > >>>>>>>>>>> not end up running MR jobs, but this was
> > > probably
> > > > > > true
> > > > > > > > for
> > > > > > > > > > MOB
> > > > > > > > > > > >>> as
> > > > > > > > > > > >>>> in
> > > > > > > > > > > >>>>>>>>> "if
> > > > > > > > > > > >>>>>>>>>>> you don't enable MOB you don't need MR")
> > > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > > >>>>>>>>>>> any thoughts? do we a rule that says "we
> > don't
> > > > want
> > > > > > to
> > > > > > > > have
> > > > > > > > > > > >>> hbase
> > > > > > > > > > > >>>> run
> > > > > > > > > > > >>>>>>>>> MR
> > > > > > > > > > > >>>>>>>>>>> jobs, only tool started manually by the
> user
> > > can
> > > > do
> > > > > > > > that".
> > > > > > > > > or
> > > > > > > > > > > >>> can
> > > > > > > > > > > >>>> we
> > > > > > > > > > > >>>>>>>>>> start
> > > > > > > > > > > >>>>>>>>>>> adding MR calls around without problems?
> > > > > > > > > > > >>>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSSION] MR jobs started by Master or RS
Posted by Stack <st...@duboce.net>.
On Fri, Sep 23, 2016 at 11:22 AM, Vladimir Rodionov <vl...@gmail.com>
wrote:
> >> + MR is dead
>
> Does MR know that? :)
>
> Again. With all due respect, stack - still no suggestions what should we
> use for "bulk data move and transformation" instead of MR?
>
Use whatever distributed engine suits your fancy -- MR, Spark, distributed
shell -- just don't have HBase core depend on it, even optionally.
> I suggest voting first on "do we need backup in HBase"? In my opinion, some
> group members still not sure about that and some will give -1
> in any case. Just because ...
>
>
We could run a vote, sure. -1 on that backup be in core hbase (+1 on adding
all the API any such external tool might need to run).
St.Ack
> -Vlad
>
>
>
>
>
>
> On Fri, Sep 23, 2016 at 10:57 AM, Stack <st...@duboce.net> wrote:
>
> > On Fri, Sep 23, 2016 at 6:46 AM, Matteo Bertozzi <
> theo.bertozzi@gmail.com>
> > wrote:
> >
> > > let me try to go back to my original topic.
> > > this question was meant to be generic, and provide some rule for future
> > > code.
> > >
> > > from what I can gather, a rule that may satisfy everyone can be:
> > > - we don't want any core feature (e.g. compaction/log-split/log-
> reply)
> > > over MR, because some cluster may not want or may have an
> > > external/uncontrolled MR setup.
> > >
> >
> > +1
> >
> >
> > > - we allow non-core features (e.g. features enabled by a flag) to run
> MR
> > > jobs from hbase, because unless you use the feature, MR is not
> required.
> > >
> > >
> > -1 to hbase core depending on MR or core -- whether behind a flag or not
> --
> > ever being able to launch MR jobs.
> >
> > + MR is dead. We should be busy working hard to undo it from hbase-server
> > moving it out to be an optional module (Spark would be its peer).
> > + Master is a rats nest of state. Matteo, Stephen, and Appy are busy
> > working hard on moving it up on to a new foundation. Lets not clutter
> task
> > harder by piling on more moving parts.
> >
> > St.Ack
> >
> >
> > > Matteo
> > >
> > >
> > > On Fri, Sep 23, 2016 at 5:39 AM, Ted Yu <yu...@gmail.com> wrote:
> > >
> > > > I suggest you look at Matteo's work for AssignmentManager which is to
> > > make
> > > > Master more stable.
> > > >
> > > > Cheers
> > > >
> > > > On Fri, Sep 23, 2016 at 5:32 AM, 张铎 <pa...@gmail.com> wrote:
> > > >
> > > > > No, not your fault, at lease, not this time:)
> > > > >
> > > > > Why I call the code ugly? Can you simply tell me the sequence of
> > calls
> > > > when
> > > > > starting up the HMaster? HMaster is also a regionserver so it
> extends
> > > > > HRegionServer, and the initialization of HRegionServer sometimes
> > needs
> > > to
> > > > > make rpc calls to HMaster. A simple change would cause
> probabilistic
> > > dead
> > > > > lock or some strange NPEs...
> > > > >
> > > > > That's why I'm very nervous when somebody wants to add new features
> > or
> > > > add
> > > > > external dependencies to HMaster, especially add more works for the
> > > start
> > > > > up processing...
> > > > >
> > > > > Thanks.
> > > > >
> > > > > 2016-09-23 20:02 GMT+08:00 Ted Yu <yu...@gmail.com>:
> > > > >
> > > > > > I read through HADOOP-13433
> > > > > > <https://issues.apache.org/jira/browse/HADOOP-13433> - the cited
> > > race
> > > > > > condition is in jdk.
> > > > > >
> > > > > > Suggest pinging the reviewer on JIRA to get it moving.
> > > > > >
> > > > > > bq. But the ugly code in HMaster is readlly a problem...
> > > > > >
> > > > > > Can you be specific as to which code is ugly ? Is it in the
> backup
> > /
> > > > > > restore mega patch ?
> > > > > >
> > > > > > Cheers
> > > > > >
> > > > > > On Thu, Sep 22, 2016 at 10:44 PM, 张铎 <pa...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > If you guys have already implemented the feature in the MR way
> > and
> > > > the
> > > > > > > patch is ready for landing on master, I'm a -0 on it as I do
> not
> > > want
> > > > > to
> > > > > > > block the development progress.
> > > > > > >
> > > > > > > But I strongly suggest later we need to revisit the design and
> > see
> > > if
> > > > > we
> > > > > > > can seperated the logic from HMaster as much as possible. HA is
> > > not a
> > > > > big
> > > > > > > problem if you do not store any metada locally. But the ugly
> code
> > > in
> > > > > > > HMaster is readlly a problem...
> > > > > > >
> > > > > > > And for security, I have a issue pending for a long time. Can
> > > someone
> > > > > > help
> > > > > > > taking a simple look at it? This is what I mean, ugly code...
> > > logout
> > > > > and
> > > > > > > destroy the credentials in a subject when it is still being
> used,
> > > and
> > > > > > > declared as LimitPrivacy so I can not change the behivor and
> the
> > > only
> > > > > way
> > > > > > > to fix it is to write another piece of ugly code...
> > > > > > >
> > > > > > > https://issues.apache.org/jira/browse/HADOOP-13433
> > > > > > >
> > > > > > > 2016-09-23 12:53 GMT+08:00 Vladimir Rodionov <
> > > vladrodionov@gmail.com
> > > > >:
> > > > > > >
> > > > > > > > >> If in the future, we find better ways of doing this
> without
> > > > using
> > > > > > MR,
> > > > > > > we
> > > > > > > > can certainly consider that
> > > > > > > >
> > > > > > > > Our framework for distributed operations is abstract and
> allows
> > > > > > > > different implementations. MR is just one implementation we
> > > > provide.
> > > > > > > >
> > > > > > > > -Vlad
> > > > > > > >
> > > > > > > > On Thu, Sep 22, 2016 at 9:38 PM, Devaraj Das <
> > > ddas@hortonworks.com
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Guys, first off apologies for bringing in the topic of
> > MR-based
> > > > > > > > > compactions.. But I was thinking more about the
> SpliceMachine
> > > > > > approach
> > > > > > > of
> > > > > > > > > managing compactions in Spark where apparently they saw a
> lot
> > > of
> > > > > > > > benefits.
> > > > > > > > > Apologies for giving you that sore throat Andrew; I really
> > > didn't
> > > > > > mean
> > > > > > > to
> > > > > > > > > :-)
> > > > > > > > >
> > > > > > > > > So on this issue, we have these on the plate:
> > > > > > > > > 0. Somehow not use MR but something like that
> > > > > > > > > 1. Run a standalone service other than master
> > > > > > > > > 2. Shell out from the master
> > > > > > > > >
> > > > > > > > > I don't think we have a good answer to (0), and I don't
> think
> > > > it's
> > > > > > even
> > > > > > > > > worth the effort of trying to build something when MR is
> > > already
> > > > > > there,
> > > > > > > > and
> > > > > > > > > being used by HBase already for some operations.
> > > > > > > > >
> > > > > > > > > On (1), we have to deal with a myriad of issues - HA of the
> > > > server
> > > > > > not
> > > > > > > > > being the least of them all. Security (kerberos
> > authentication,
> > > > > > another
> > > > > > > > > keytab to manage, etc. etc. etc.). IMO, that approach is
> DOA.
> > > > > Instead
> > > > > > > > let's
> > > > > > > > > substitute that (1) with the HBase Master. I haven't seen
> any
> > > > good
> > > > > > > reason
> > > > > > > > > why the HBase master shouldn't launch MR jobs if needed.
> It's
> > > not
> > > > > > > ideal;
> > > > > > > > > agreed.
> > > > > > > > >
> > > > > > > > > Now before going to (2), let's see what are the benefits of
> > > > running
> > > > > > the
> > > > > > > > > backup/restore jobs from the master. I think Ted has
> > summarized
> > > > > some
> > > > > > of
> > > > > > > > the
> > > > > > > > > issues that we need to take care of - basically, the master
> > can
> > > > > keep
> > > > > > > > track
> > > > > > > > > of running jobs, and should it fail, the backup master can
> > > > continue
> > > > > > > > keeping
> > > > > > > > > track of it (since the jobId would have been recorded in
> the
> > > proc
> > > > > > WAL).
> > > > > > > > The
> > > > > > > > > master can also do cleanup, etc. of failed backup/restore
> > > > > processes.
> > > > > > > > > Security is another issue - the job needs to run as 'hbase'
> > > since
> > > > > it
> > > > > > > owns
> > > > > > > > > the data. Having the master launch the job makes it get
> that
> > > > > > privilege.
> > > > > > > > In
> > > > > > > > > the (2) approach, it's hard to do some of the above
> > management.
> > > > > > > > >
> > > > > > > > > Guys, just to reiterate, the patch as such is ready from
> the
> > > > > overall
> > > > > > > > > design/arch point of view (maybe code review is still
> pending
> > > > from
> > > > > > > > Matteo).
> > > > > > > > > If in the future, we find better ways of doing this without
> > > using
> > > > > MR,
> > > > > > > we
> > > > > > > > > can certainly consider that. But IMO don't think we should
> > > block
> > > > > this
> > > > > > > > patch
> > > > > > > > > from getting merged.
> > > > > > > > >
> > > > > > > > > ________________________________________
> > > > > > > > > From: 张铎 <pa...@gmail.com>
> > > > > > > > > Sent: Thursday, September 22, 2016 8:32 PM
> > > > > > > > > To: dev@hbase.apache.org
> > > > > > > > > Subject: Re: [DISCUSSION] MR jobs started by Master or RS
> > > > > > > > >
> > > > > > > > > So what about a standalone service other than master? You
> can
> > > use
> > > > > > your
> > > > > > > > own
> > > > > > > > > procedure store in that service?
> > > > > > > > >
> > > > > > > > > 2016-09-23 11:28 GMT+08:00 Ted Yu <yu...@gmail.com>:
> > > > > > > > >
> > > > > > > > > > An earlier implementation was client driven.
> > > > > > > > > >
> > > > > > > > > > But with that approach, it is hard to resume if there is
> > > error
> > > > > > > midway.
> > > > > > > > > > Using Procedure V2 makes the backup / restore more
> robust.
> > > > > > > > > >
> > > > > > > > > > Another consideration is for security. It is hard to
> > enforce
> > > > > > security
> > > > > > > > (to
> > > > > > > > > > be implemented) for client driven actions.
> > > > > > > > > >
> > > > > > > > > > Cheers
> > > > > > > > > >
> > > > > > > > > > > On Sep 22, 2016, at 8:15 PM, Andrew Purtell <
> > > > > > > > andrew.purtell@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > No, this misses Matteo's finer point, which is
> "shelling
> > > out"
> > > > > > from
> > > > > > > > the
> > > > > > > > > > master directly to run MR is a first. Why not drive this
> > > with a
> > > > > > > utility
> > > > > > > > > > derived from Tool?
> > > > > > > > > > >
> > > > > > > > > > > On Sep 22, 2016, at 7:57 PM, Vladimir Rodionov <
> > > > > > > > vladrodionov@gmail.com
> > > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > >>>> In our production cluster, it is a common case we
> > just
> > > > have
> > > > > > > HDFS
> > > > > > > > > and
> > > > > > > > > > >>>> HBase deployed.
> > > > > > > > > > >>>> If our Master/RS depend on MR framework (especially
> > some
> > > > > > > features
> > > > > > > > we
> > > > > > > > > > >>>> have not used at all), it introduced another cost
> for
> > > > > > maintain.
> > > > > > > > I
> > > > > > > > > > >>>> don't think it is a good idea.
> > > > > > > > > > >>
> > > > > > > > > > >> So , you are not backup users in this case. Many our
> > > > customers
> > > > > > > have
> > > > > > > > > full
> > > > > > > > > > >> stack deployed and
> > > > > > > > > > >> want see backup to be a standard feature. Besides
> this,
> > > > > nothing
> > > > > > > will
> > > > > > > > > > happen
> > > > > > > > > > >> in your cluster
> > > > > > > > > > >> if you won't be doing backups.
> > > > > > > > > > >>
> > > > > > > > > > >> This discussion (we do not want see M/R dependency)
> goes
> > > to
> > > > > > > nowhere.
> > > > > > > > > We
> > > > > > > > > > >> asked already, at least twice, to suggest another
> > > framework
> > > > > > (other
> > > > > > > > > than
> > > > > > > > > > M/R)
> > > > > > > > > > >> for bulk data copy with *conversion*. Still waiting
> for
> > > > > > > suggestions.
> > > > > > > > > > >>
> > > > > > > > > > >> -Vlad
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >>> On Thu, Sep 22, 2016 at 7:49 PM, Ted Yu <
> > > > yuzhihong@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > > >>>
> > > > > > > > > > >>> If MR framework is not deployed in the cluster, hbase
> > > still
> > > > > > > > functions
> > > > > > > > > > >>> normally (post merge).
> > > > > > > > > > >>>
> > > > > > > > > > >>> In terms of build time dependency, we have long been
> > > > > depending
> > > > > > on
> > > > > > > > > > >>> mapreduce. Take a look at ExportSnapshot.
> > > > > > > > > > >>>
> > > > > > > > > > >>> Cheers
> > > > > > > > > > >>>
> > > > > > > > > > >>> On Thu, Sep 22, 2016 at 7:42 PM, Heng Chen <
> > > > > > > > heng.chen.1986@gmail.com
> > > > > > > > > >
> > > > > > > > > > >>> wrote:
> > > > > > > > > > >>>
> > > > > > > > > > >>>> In our production cluster, it is a common case we
> > just
> > > > have
> > > > > > > HDFS
> > > > > > > > > and
> > > > > > > > > > >>>> HBase deployed.
> > > > > > > > > > >>>> If our Master/RS depend on MR framework (especially
> > some
> > > > > > > features
> > > > > > > > we
> > > > > > > > > > >>>> have not used at all), it introduced another cost
> for
> > > > > > maintain.
> > > > > > > > I
> > > > > > > > > > >>>> don't think it is a good idea.
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> 2016-09-23 10:28 GMT+08:00 张铎 <
> palomino219@gmail.com
> > >:
> > > > > > > > > > >>>>> To be specific, for example, our nice
> Backup/Restore
> > > > > feature,
> > > > > > > if
> > > > > > > > we
> > > > > > > > > > >>> think
> > > > > > > > > > >>>>> this is not a core feature of HBase, then we could
> > make
> > > > it
> > > > > > > depend
> > > > > > > > > on
> > > > > > > > > > >>> MR,
> > > > > > > > > > >>>>> and start a standalone BackupManager instance that
> > > > submits
> > > > > MR
> > > > > > > > jobs
> > > > > > > > > to
> > > > > > > > > > >>> do
> > > > > > > > > > >>>>> periodical maintenance job. And if we think this
> is a
> > > > core
> > > > > > > > feature
> > > > > > > > > > that
> > > > > > > > > > >>>>> everyone should use it, then we'd better implement
> it
> > > > > without
> > > > > > > MR
> > > > > > > > > > >>>>> dependency, like DLS.
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> Thanks.
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> 2016-09-23 10:11 GMT+08:00 张铎 <
> palomino219@gmail.com
> > >:
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>>> I‘m -1 on let master or rs launch MR jobs. It is
> OK
> > > that
> > > > > > some
> > > > > > > of
> > > > > > > > > our
> > > > > > > > > > >>>>>> features depend on MR but I think the bottom line
> is
> > > > that
> > > > > we
> > > > > > > > > should
> > > > > > > > > > >>>> launch
> > > > > > > > > > >>>>>> the jobs from outside manually or by other
> services.
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>> 2016-09-23 9:47 GMT+08:00 Andrew Purtell <
> > > > > > > > > andrew.purtell@gmail.com
> > > > > > > > > > >:
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>>> Ok, got it. Well "shelling out" is on the line I
> > > think,
> > > > > so
> > > > > > a
> > > > > > > > fair
> > > > > > > > > > >>>>>>> question.
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>> Can this be driven by a utility derived from Tool
> > > like
> > > > > our
> > > > > > > > other
> > > > > > > > > MR
> > > > > > > > > > >>>> apps?
> > > > > > > > > > >>>>>>> The issue is needing the AccessController to
> decide
> > > if
> > > > > > > allowed?
> > > > > > > > > But
> > > > > > > > > > >>>> nothing
> > > > > > > > > > >>>>>>> prevents the user from running the job
> > > > > > > manually/independently,
> > > > > > > > > > right?
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>>> On Sep 22, 2016, at 3:44 PM, Matteo Bertozzi <
> > > > > > > > > > >>>> theo.bertozzi@gmail.com>
> > > > > > > > > > >>>>>>> wrote:
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>> just a remark. my query was not about tools
> using
> > MR
> > > > > > > > (everyone i
> > > > > > > > > > >>>> think
> > > > > > > > > > >>>>>>> is
> > > > > > > > > > >>>>>>>> ok with those).
> > > > > > > > > > >>>>>>>> the topic was about: "are we ok with running MR
> > jobs
> > > > > from
> > > > > > > > Master
> > > > > > > > > > >>> and
> > > > > > > > > > >>>> RSs
> > > > > > > > > > >>>>>>>> code?" since this will be the first time we do
> > this
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>> Matteo
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>>> On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das <
> > > > > > > > > > >>> ddas@hortonworks.com>
> > > > > > > > > > >>>>>>> wrote:
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>>> Very much agree; for tools like ExportSnapshot
> /
> > > > > Backup /
> > > > > > > > > > Restore,
> > > > > > > > > > >>>> it's
> > > > > > > > > > >>>>>>>>> fine to be dependent on MR. MR is the right
> > > framework
> > > > > for
> > > > > > > > such.
> > > > > > > > > > We
> > > > > > > > > > >>>>>>> should
> > > > > > > > > > >>>>>>>>> also do compactions using MR (just saying :) )
> > > > > > > > > > >>>>>>>>> ________________________________________
> > > > > > > > > > >>>>>>>>> From: Ted Yu <yu...@gmail.com>
> > > > > > > > > > >>>>>>>>> Sent: Thursday, September 22, 2016 2:00 PM
> > > > > > > > > > >>>>>>>>> To: dev@hbase.apache.org
> > > > > > > > > > >>>>>>>>> Subject: Re: [DISCUSSION] MR jobs started by
> > Master
> > > > or
> > > > > RS
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>>> I agree - backup / restore is in the same
> > category
> > > as
> > > > > > > import
> > > > > > > > /
> > > > > > > > > > >>>> export.
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>>> On Thu, Sep 22, 2016 at 1:58 PM, Andrew
> Purtell <
> > > > > > > > > > >>>>>>> andrew.purtell@gmail.com>
> > > > > > > > > > >>>>>>>>> wrote:
> > > > > > > > > > >>>>>>>>>
> > > > > > > > > > >>>>>>>>>> Backup is extra tooling around core in my
> > opinion.
> > > > > Like
> > > > > > > > import
> > > > > > > > > > or
> > > > > > > > > > >>>>>>> export.
> > > > > > > > > > >>>>>>>>>> Or the optional MOB tool. It's fine.
> > > > > > > > > > >>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>> On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi
> <
> > > > > > > > > > >>>> mbertozzi@apache.org>
> > > > > > > > > > >>>>>>>>>> wrote:
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>> What's the latest opinion around running MR
> > jobs
> > > > from
> > > > > > > hbase
> > > > > > > > > > >>>> (Master
> > > > > > > > > > >>>>>>> or
> > > > > > > > > > >>>>>>>>>> RS)?
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>> I remember in the past that there was
> > discussion
> > > > > about
> > > > > > > not
> > > > > > > > > > >>> having
> > > > > > > > > > >>>> MR
> > > > > > > > > > >>>>>>>>> has
> > > > > > > > > > >>>>>>>>>>> direct dependency of hbase.
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>> I think some of discussion where around MOB
> > that
> > > > had
> > > > > a
> > > > > > MR
> > > > > > > > job
> > > > > > > > > > to
> > > > > > > > > > >>>>>>>>> compact,
> > > > > > > > > > >>>>>>>>>>> that later was transformed in a non-MR job to
> > be
> > > > > > merged,
> > > > > > > I
> > > > > > > > > > think
> > > > > > > > > > >>>> we
> > > > > > > > > > >>>>>>>>> had a
> > > > > > > > > > >>>>>>>>>>> similar discussion for log split/replay.
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>> the latest is the new Backup feature
> > > (HBASE-7912),
> > > > > that
> > > > > > > > runs
> > > > > > > > > a
> > > > > > > > > > >>> MR
> > > > > > > > > > >>>> job
> > > > > > > > > > >>>>>>>>>> from
> > > > > > > > > > >>>>>>>>>>> the master to copy data or restore data.
> > > > > > > > > > >>>>>>>>>>> (backup is also "not really core" as in.. if
> > you
> > > > > don't
> > > > > > > use
> > > > > > > > > > >>> backup
> > > > > > > > > > >>>>>>>>> you'll
> > > > > > > > > > >>>>>>>>>>> not end up running MR jobs, but this was
> > probably
> > > > > true
> > > > > > > for
> > > > > > > > > MOB
> > > > > > > > > > >>> as
> > > > > > > > > > >>>> in
> > > > > > > > > > >>>>>>>>> "if
> > > > > > > > > > >>>>>>>>>>> you don't enable MOB you don't need MR")
> > > > > > > > > > >>>>>>>>>>>
> > > > > > > > > > >>>>>>>>>>> any thoughts? do we a rule that says "we
> don't
> > > want
> > > > > to
> > > > > > > have
> > > > > > > > > > >>> hbase
> > > > > > > > > > >>>> run
> > > > > > > > > > >>>>>>>>> MR
> > > > > > > > > > >>>>>>>>>>> jobs, only tool started manually by the user
> > can
> > > do
> > > > > > > that".
> > > > > > > > or
> > > > > > > > > > >>> can
> > > > > > > > > > >>>> we
> > > > > > > > > > >>>>>>>>>> start
> > > > > > > > > > >>>>>>>>>>> adding MR calls around without problems?
> > > > > > > > > > >>>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSSION] MR jobs started by Master or RS
Posted by Vladimir Rodionov <vl...@gmail.com>.
>> + MR is dead
Does MR know that? :)
Again. With all due respect, stack - still no suggestions what should we
use for "bulk data move and transformation" instead of MR?
Backup needs this, by default. We are talking about TBs of data in a single
session. We not only copy data between src and dest,
but transform it as well on-the-fly (WALs into HFiles)
That can be Spark in a future, of course, but not right now. Framework is
open.
I suggest voting first on "do we need backup in HBase"? In my opinion, some
group members still not sure about that and some will give -1
in any case. Just because ...
-Vlad
On Fri, Sep 23, 2016 at 10:57 AM, Stack <st...@duboce.net> wrote:
> On Fri, Sep 23, 2016 at 6:46 AM, Matteo Bertozzi <th...@gmail.com>
> wrote:
>
> > let me try to go back to my original topic.
> > this question was meant to be generic, and provide some rule for future
> > code.
> >
> > from what I can gather, a rule that may satisfy everyone can be:
> > - we don't want any core feature (e.g. compaction/log-split/log-reply)
> > over MR, because some cluster may not want or may have an
> > external/uncontrolled MR setup.
> >
>
> +1
>
>
> > - we allow non-core features (e.g. features enabled by a flag) to run MR
> > jobs from hbase, because unless you use the feature, MR is not required.
> >
> >
> -1 to hbase core depending on MR or core -- whether behind a flag or not --
> ever being able to launch MR jobs.
>
> + MR is dead. We should be busy working hard to undo it from hbase-server
> moving it out to be an optional module (Spark would be its peer).
> + Master is a rats nest of state. Matteo, Stephen, and Appy are busy
> working hard on moving it up on to a new foundation. Lets not clutter task
> harder by piling on more moving parts.
>
> St.Ack
>
>
> > Matteo
> >
> >
> > On Fri, Sep 23, 2016 at 5:39 AM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > I suggest you look at Matteo's work for AssignmentManager which is to
> > make
> > > Master more stable.
> > >
> > > Cheers
> > >
> > > On Fri, Sep 23, 2016 at 5:32 AM, 张铎 <pa...@gmail.com> wrote:
> > >
> > > > No, not your fault, at lease, not this time:)
> > > >
> > > > Why I call the code ugly? Can you simply tell me the sequence of
> calls
> > > when
> > > > starting up the HMaster? HMaster is also a regionserver so it extends
> > > > HRegionServer, and the initialization of HRegionServer sometimes
> needs
> > to
> > > > make rpc calls to HMaster. A simple change would cause probabilistic
> > dead
> > > > lock or some strange NPEs...
> > > >
> > > > That's why I'm very nervous when somebody wants to add new features
> or
> > > add
> > > > external dependencies to HMaster, especially add more works for the
> > start
> > > > up processing...
> > > >
> > > > Thanks.
> > > >
> > > > 2016-09-23 20:02 GMT+08:00 Ted Yu <yu...@gmail.com>:
> > > >
> > > > > I read through HADOOP-13433
> > > > > <https://issues.apache.org/jira/browse/HADOOP-13433> - the cited
> > race
> > > > > condition is in jdk.
> > > > >
> > > > > Suggest pinging the reviewer on JIRA to get it moving.
> > > > >
> > > > > bq. But the ugly code in HMaster is readlly a problem...
> > > > >
> > > > > Can you be specific as to which code is ugly ? Is it in the backup
> /
> > > > > restore mega patch ?
> > > > >
> > > > > Cheers
> > > > >
> > > > > On Thu, Sep 22, 2016 at 10:44 PM, 张铎 <pa...@gmail.com>
> wrote:
> > > > >
> > > > > > If you guys have already implemented the feature in the MR way
> and
> > > the
> > > > > > patch is ready for landing on master, I'm a -0 on it as I do not
> > want
> > > > to
> > > > > > block the development progress.
> > > > > >
> > > > > > But I strongly suggest later we need to revisit the design and
> see
> > if
> > > > we
> > > > > > can seperated the logic from HMaster as much as possible. HA is
> > not a
> > > > big
> > > > > > problem if you do not store any metada locally. But the ugly code
> > in
> > > > > > HMaster is readlly a problem...
> > > > > >
> > > > > > And for security, I have a issue pending for a long time. Can
> > someone
> > > > > help
> > > > > > taking a simple look at it? This is what I mean, ugly code...
> > logout
> > > > and
> > > > > > destroy the credentials in a subject when it is still being used,
> > and
> > > > > > declared as LimitPrivacy so I can not change the behivor and the
> > only
> > > > way
> > > > > > to fix it is to write another piece of ugly code...
> > > > > >
> > > > > > https://issues.apache.org/jira/browse/HADOOP-13433
> > > > > >
> > > > > > 2016-09-23 12:53 GMT+08:00 Vladimir Rodionov <
> > vladrodionov@gmail.com
> > > >:
> > > > > >
> > > > > > > >> If in the future, we find better ways of doing this without
> > > using
> > > > > MR,
> > > > > > we
> > > > > > > can certainly consider that
> > > > > > >
> > > > > > > Our framework for distributed operations is abstract and allows
> > > > > > > different implementations. MR is just one implementation we
> > > provide.
> > > > > > >
> > > > > > > -Vlad
> > > > > > >
> > > > > > > On Thu, Sep 22, 2016 at 9:38 PM, Devaraj Das <
> > ddas@hortonworks.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Guys, first off apologies for bringing in the topic of
> MR-based
> > > > > > > > compactions.. But I was thinking more about the SpliceMachine
> > > > > approach
> > > > > > of
> > > > > > > > managing compactions in Spark where apparently they saw a lot
> > of
> > > > > > > benefits.
> > > > > > > > Apologies for giving you that sore throat Andrew; I really
> > didn't
> > > > > mean
> > > > > > to
> > > > > > > > :-)
> > > > > > > >
> > > > > > > > So on this issue, we have these on the plate:
> > > > > > > > 0. Somehow not use MR but something like that
> > > > > > > > 1. Run a standalone service other than master
> > > > > > > > 2. Shell out from the master
> > > > > > > >
> > > > > > > > I don't think we have a good answer to (0), and I don't think
> > > it's
> > > > > even
> > > > > > > > worth the effort of trying to build something when MR is
> > already
> > > > > there,
> > > > > > > and
> > > > > > > > being used by HBase already for some operations.
> > > > > > > >
> > > > > > > > On (1), we have to deal with a myriad of issues - HA of the
> > > server
> > > > > not
> > > > > > > > being the least of them all. Security (kerberos
> authentication,
> > > > > another
> > > > > > > > keytab to manage, etc. etc. etc.). IMO, that approach is DOA.
> > > > Instead
> > > > > > > let's
> > > > > > > > substitute that (1) with the HBase Master. I haven't seen any
> > > good
> > > > > > reason
> > > > > > > > why the HBase master shouldn't launch MR jobs if needed. It's
> > not
> > > > > > ideal;
> > > > > > > > agreed.
> > > > > > > >
> > > > > > > > Now before going to (2), let's see what are the benefits of
> > > running
> > > > > the
> > > > > > > > backup/restore jobs from the master. I think Ted has
> summarized
> > > > some
> > > > > of
> > > > > > > the
> > > > > > > > issues that we need to take care of - basically, the master
> can
> > > > keep
> > > > > > > track
> > > > > > > > of running jobs, and should it fail, the backup master can
> > > continue
> > > > > > > keeping
> > > > > > > > track of it (since the jobId would have been recorded in the
> > proc
> > > > > WAL).
> > > > > > > The
> > > > > > > > master can also do cleanup, etc. of failed backup/restore
> > > > processes.
> > > > > > > > Security is another issue - the job needs to run as 'hbase'
> > since
> > > > it
> > > > > > owns
> > > > > > > > the data. Having the master launch the job makes it get that
> > > > > privilege.
> > > > > > > In
> > > > > > > > the (2) approach, it's hard to do some of the above
> management.
> > > > > > > >
> > > > > > > > Guys, just to reiterate, the patch as such is ready from the
> > > > overall
> > > > > > > > design/arch point of view (maybe code review is still pending
> > > from
> > > > > > > Matteo).
> > > > > > > > If in the future, we find better ways of doing this without
> > using
> > > > MR,
> > > > > > we
> > > > > > > > can certainly consider that. But IMO don't think we should
> > block
> > > > this
> > > > > > > patch
> > > > > > > > from getting merged.
> > > > > > > >
> > > > > > > > ________________________________________
> > > > > > > > From: 张铎 <pa...@gmail.com>
> > > > > > > > Sent: Thursday, September 22, 2016 8:32 PM
> > > > > > > > To: dev@hbase.apache.org
> > > > > > > > Subject: Re: [DISCUSSION] MR jobs started by Master or RS
> > > > > > > >
> > > > > > > > So what about a standalone service other than master? You can
> > use
> > > > > your
> > > > > > > own
> > > > > > > > procedure store in that service?
> > > > > > > >
> > > > > > > > 2016-09-23 11:28 GMT+08:00 Ted Yu <yu...@gmail.com>:
> > > > > > > >
> > > > > > > > > An earlier implementation was client driven.
> > > > > > > > >
> > > > > > > > > But with that approach, it is hard to resume if there is
> > error
> > > > > > midway.
> > > > > > > > > Using Procedure V2 makes the backup / restore more robust.
> > > > > > > > >
> > > > > > > > > Another consideration is for security. It is hard to
> enforce
> > > > > security
> > > > > > > (to
> > > > > > > > > be implemented) for client driven actions.
> > > > > > > > >
> > > > > > > > > Cheers
> > > > > > > > >
> > > > > > > > > > On Sep 22, 2016, at 8:15 PM, Andrew Purtell <
> > > > > > > andrew.purtell@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > No, this misses Matteo's finer point, which is "shelling
> > out"
> > > > > from
> > > > > > > the
> > > > > > > > > master directly to run MR is a first. Why not drive this
> > with a
> > > > > > utility
> > > > > > > > > derived from Tool?
> > > > > > > > > >
> > > > > > > > > > On Sep 22, 2016, at 7:57 PM, Vladimir Rodionov <
> > > > > > > vladrodionov@gmail.com
> > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >>>> In our production cluster, it is a common case we
> just
> > > have
> > > > > > HDFS
> > > > > > > > and
> > > > > > > > > >>>> HBase deployed.
> > > > > > > > > >>>> If our Master/RS depend on MR framework (especially
> some
> > > > > > features
> > > > > > > we
> > > > > > > > > >>>> have not used at all), it introduced another cost for
> > > > > maintain.
> > > > > > > I
> > > > > > > > > >>>> don't think it is a good idea.
> > > > > > > > > >>
> > > > > > > > > >> So , you are not backup users in this case. Many our
> > > customers
> > > > > > have
> > > > > > > > full
> > > > > > > > > >> stack deployed and
> > > > > > > > > >> want see backup to be a standard feature. Besides this,
> > > > nothing
> > > > > > will
> > > > > > > > > happen
> > > > > > > > > >> in your cluster
> > > > > > > > > >> if you won't be doing backups.
> > > > > > > > > >>
> > > > > > > > > >> This discussion (we do not want see M/R dependency) goes
> > to
> > > > > > nowhere.
> > > > > > > > We
> > > > > > > > > >> asked already, at least twice, to suggest another
> > framework
> > > > > (other
> > > > > > > > than
> > > > > > > > > M/R)
> > > > > > > > > >> for bulk data copy with *conversion*. Still waiting for
> > > > > > suggestions.
> > > > > > > > > >>
> > > > > > > > > >> -Vlad
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>> On Thu, Sep 22, 2016 at 7:49 PM, Ted Yu <
> > > yuzhihong@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > > > >>>
> > > > > > > > > >>> If MR framework is not deployed in the cluster, hbase
> > still
> > > > > > > functions
> > > > > > > > > >>> normally (post merge).
> > > > > > > > > >>>
> > > > > > > > > >>> In terms of build time dependency, we have long been
> > > > depending
> > > > > on
> > > > > > > > > >>> mapreduce. Take a look at ExportSnapshot.
> > > > > > > > > >>>
> > > > > > > > > >>> Cheers
> > > > > > > > > >>>
> > > > > > > > > >>> On Thu, Sep 22, 2016 at 7:42 PM, Heng Chen <
> > > > > > > heng.chen.1986@gmail.com
> > > > > > > > >
> > > > > > > > > >>> wrote:
> > > > > > > > > >>>
> > > > > > > > > >>>> In our production cluster, it is a common case we
> just
> > > have
> > > > > > HDFS
> > > > > > > > and
> > > > > > > > > >>>> HBase deployed.
> > > > > > > > > >>>> If our Master/RS depend on MR framework (especially
> some
> > > > > > features
> > > > > > > we
> > > > > > > > > >>>> have not used at all), it introduced another cost for
> > > > > maintain.
> > > > > > > I
> > > > > > > > > >>>> don't think it is a good idea.
> > > > > > > > > >>>>
> > > > > > > > > >>>> 2016-09-23 10:28 GMT+08:00 张铎 <palomino219@gmail.com
> >:
> > > > > > > > > >>>>> To be specific, for example, our nice Backup/Restore
> > > > feature,
> > > > > > if
> > > > > > > we
> > > > > > > > > >>> think
> > > > > > > > > >>>>> this is not a core feature of HBase, then we could
> make
> > > it
> > > > > > depend
> > > > > > > > on
> > > > > > > > > >>> MR,
> > > > > > > > > >>>>> and start a standalone BackupManager instance that
> > > submits
> > > > MR
> > > > > > > jobs
> > > > > > > > to
> > > > > > > > > >>> do
> > > > > > > > > >>>>> periodical maintenance job. And if we think this is a
> > > core
> > > > > > > feature
> > > > > > > > > that
> > > > > > > > > >>>>> everyone should use it, then we'd better implement it
> > > > without
> > > > > > MR
> > > > > > > > > >>>>> dependency, like DLS.
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> Thanks.
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> 2016-09-23 10:11 GMT+08:00 张铎 <palomino219@gmail.com
> >:
> > > > > > > > > >>>>>
> > > > > > > > > >>>>>> I‘m -1 on let master or rs launch MR jobs. It is OK
> > that
> > > > > some
> > > > > > of
> > > > > > > > our
> > > > > > > > > >>>>>> features depend on MR but I think the bottom line is
> > > that
> > > > we
> > > > > > > > should
> > > > > > > > > >>>> launch
> > > > > > > > > >>>>>> the jobs from outside manually or by other services.
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> 2016-09-23 9:47 GMT+08:00 Andrew Purtell <
> > > > > > > > andrew.purtell@gmail.com
> > > > > > > > > >:
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>>> Ok, got it. Well "shelling out" is on the line I
> > think,
> > > > so
> > > > > a
> > > > > > > fair
> > > > > > > > > >>>>>>> question.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> Can this be driven by a utility derived from Tool
> > like
> > > > our
> > > > > > > other
> > > > > > > > MR
> > > > > > > > > >>>> apps?
> > > > > > > > > >>>>>>> The issue is needing the AccessController to decide
> > if
> > > > > > allowed?
> > > > > > > > But
> > > > > > > > > >>>> nothing
> > > > > > > > > >>>>>>> prevents the user from running the job
> > > > > > manually/independently,
> > > > > > > > > right?
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>> On Sep 22, 2016, at 3:44 PM, Matteo Bertozzi <
> > > > > > > > > >>>> theo.bertozzi@gmail.com>
> > > > > > > > > >>>>>>> wrote:
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> just a remark. my query was not about tools using
> MR
> > > > > > > (everyone i
> > > > > > > > > >>>> think
> > > > > > > > > >>>>>>> is
> > > > > > > > > >>>>>>>> ok with those).
> > > > > > > > > >>>>>>>> the topic was about: "are we ok with running MR
> jobs
> > > > from
> > > > > > > Master
> > > > > > > > > >>> and
> > > > > > > > > >>>> RSs
> > > > > > > > > >>>>>>>> code?" since this will be the first time we do
> this
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> Matteo
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>>> On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das <
> > > > > > > > > >>> ddas@hortonworks.com>
> > > > > > > > > >>>>>>> wrote:
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> Very much agree; for tools like ExportSnapshot /
> > > > Backup /
> > > > > > > > > Restore,
> > > > > > > > > >>>> it's
> > > > > > > > > >>>>>>>>> fine to be dependent on MR. MR is the right
> > framework
> > > > for
> > > > > > > such.
> > > > > > > > > We
> > > > > > > > > >>>>>>> should
> > > > > > > > > >>>>>>>>> also do compactions using MR (just saying :) )
> > > > > > > > > >>>>>>>>> ________________________________________
> > > > > > > > > >>>>>>>>> From: Ted Yu <yu...@gmail.com>
> > > > > > > > > >>>>>>>>> Sent: Thursday, September 22, 2016 2:00 PM
> > > > > > > > > >>>>>>>>> To: dev@hbase.apache.org
> > > > > > > > > >>>>>>>>> Subject: Re: [DISCUSSION] MR jobs started by
> Master
> > > or
> > > > RS
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> I agree - backup / restore is in the same
> category
> > as
> > > > > > import
> > > > > > > /
> > > > > > > > > >>>> export.
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>> On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell <
> > > > > > > > > >>>>>>> andrew.purtell@gmail.com>
> > > > > > > > > >>>>>>>>> wrote:
> > > > > > > > > >>>>>>>>>
> > > > > > > > > >>>>>>>>>> Backup is extra tooling around core in my
> opinion.
> > > > Like
> > > > > > > import
> > > > > > > > > or
> > > > > > > > > >>>>>>> export.
> > > > > > > > > >>>>>>>>>> Or the optional MOB tool. It's fine.
> > > > > > > > > >>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi <
> > > > > > > > > >>>> mbertozzi@apache.org>
> > > > > > > > > >>>>>>>>>> wrote:
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> What's the latest opinion around running MR
> jobs
> > > from
> > > > > > hbase
> > > > > > > > > >>>> (Master
> > > > > > > > > >>>>>>> or
> > > > > > > > > >>>>>>>>>> RS)?
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> I remember in the past that there was
> discussion
> > > > about
> > > > > > not
> > > > > > > > > >>> having
> > > > > > > > > >>>> MR
> > > > > > > > > >>>>>>>>> has
> > > > > > > > > >>>>>>>>>>> direct dependency of hbase.
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> I think some of discussion where around MOB
> that
> > > had
> > > > a
> > > > > MR
> > > > > > > job
> > > > > > > > > to
> > > > > > > > > >>>>>>>>> compact,
> > > > > > > > > >>>>>>>>>>> that later was transformed in a non-MR job to
> be
> > > > > merged,
> > > > > > I
> > > > > > > > > think
> > > > > > > > > >>>> we
> > > > > > > > > >>>>>>>>> had a
> > > > > > > > > >>>>>>>>>>> similar discussion for log split/replay.
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> the latest is the new Backup feature
> > (HBASE-7912),
> > > > that
> > > > > > > runs
> > > > > > > > a
> > > > > > > > > >>> MR
> > > > > > > > > >>>> job
> > > > > > > > > >>>>>>>>>> from
> > > > > > > > > >>>>>>>>>>> the master to copy data or restore data.
> > > > > > > > > >>>>>>>>>>> (backup is also "not really core" as in.. if
> you
> > > > don't
> > > > > > use
> > > > > > > > > >>> backup
> > > > > > > > > >>>>>>>>> you'll
> > > > > > > > > >>>>>>>>>>> not end up running MR jobs, but this was
> probably
> > > > true
> > > > > > for
> > > > > > > > MOB
> > > > > > > > > >>> as
> > > > > > > > > >>>> in
> > > > > > > > > >>>>>>>>> "if
> > > > > > > > > >>>>>>>>>>> you don't enable MOB you don't need MR")
> > > > > > > > > >>>>>>>>>>>
> > > > > > > > > >>>>>>>>>>> any thoughts? do we a rule that says "we don't
> > want
> > > > to
> > > > > > have
> > > > > > > > > >>> hbase
> > > > > > > > > >>>> run
> > > > > > > > > >>>>>>>>> MR
> > > > > > > > > >>>>>>>>>>> jobs, only tool started manually by the user
> can
> > do
> > > > > > that".
> > > > > > > or
> > > > > > > > > >>> can
> > > > > > > > > >>>> we
> > > > > > > > > >>>>>>>>>> start
> > > > > > > > > >>>>>>>>>>> adding MR calls around without problems?
> > > > > > > > > >>>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSSION] MR jobs started by Master or RS
Posted by Stack <st...@duboce.net>.
On Fri, Sep 23, 2016 at 6:46 AM, Matteo Bertozzi <th...@gmail.com>
wrote:
> let me try to go back to my original topic.
> this question was meant to be generic, and provide some rule for future
> code.
>
> from what I can gather, a rule that may satisfy everyone can be:
> - we don't want any core feature (e.g. compaction/log-split/log-reply)
> over MR, because some cluster may not want or may have an
> external/uncontrolled MR setup.
>
+1
> - we allow non-core features (e.g. features enabled by a flag) to run MR
> jobs from hbase, because unless you use the feature, MR is not required.
>
>
-1 to hbase core depending on MR or core -- whether behind a flag or not --
ever being able to launch MR jobs.
+ MR is dead. We should be busy working hard to undo it from hbase-server
moving it out to be an optional module (Spark would be its peer).
+ Master is a rats nest of state. Matteo, Stephen, and Appy are busy
working hard on moving it up on to a new foundation. Lets not clutter task
harder by piling on more moving parts.
St.Ack
> Matteo
>
>
> On Fri, Sep 23, 2016 at 5:39 AM, Ted Yu <yu...@gmail.com> wrote:
>
> > I suggest you look at Matteo's work for AssignmentManager which is to
> make
> > Master more stable.
> >
> > Cheers
> >
> > On Fri, Sep 23, 2016 at 5:32 AM, 张铎 <pa...@gmail.com> wrote:
> >
> > > No, not your fault, at lease, not this time:)
> > >
> > > Why I call the code ugly? Can you simply tell me the sequence of calls
> > when
> > > starting up the HMaster? HMaster is also a regionserver so it extends
> > > HRegionServer, and the initialization of HRegionServer sometimes needs
> to
> > > make rpc calls to HMaster. A simple change would cause probabilistic
> dead
> > > lock or some strange NPEs...
> > >
> > > That's why I'm very nervous when somebody wants to add new features or
> > add
> > > external dependencies to HMaster, especially add more works for the
> start
> > > up processing...
> > >
> > > Thanks.
> > >
> > > 2016-09-23 20:02 GMT+08:00 Ted Yu <yu...@gmail.com>:
> > >
> > > > I read through HADOOP-13433
> > > > <https://issues.apache.org/jira/browse/HADOOP-13433> - the cited
> race
> > > > condition is in jdk.
> > > >
> > > > Suggest pinging the reviewer on JIRA to get it moving.
> > > >
> > > > bq. But the ugly code in HMaster is readlly a problem...
> > > >
> > > > Can you be specific as to which code is ugly ? Is it in the backup /
> > > > restore mega patch ?
> > > >
> > > > Cheers
> > > >
> > > > On Thu, Sep 22, 2016 at 10:44 PM, 张铎 <pa...@gmail.com> wrote:
> > > >
> > > > > If you guys have already implemented the feature in the MR way and
> > the
> > > > > patch is ready for landing on master, I'm a -0 on it as I do not
> want
> > > to
> > > > > block the development progress.
> > > > >
> > > > > But I strongly suggest later we need to revisit the design and see
> if
> > > we
> > > > > can seperated the logic from HMaster as much as possible. HA is
> not a
> > > big
> > > > > problem if you do not store any metada locally. But the ugly code
> in
> > > > > HMaster is readlly a problem...
> > > > >
> > > > > And for security, I have a issue pending for a long time. Can
> someone
> > > > help
> > > > > taking a simple look at it? This is what I mean, ugly code...
> logout
> > > and
> > > > > destroy the credentials in a subject when it is still being used,
> and
> > > > > declared as LimitPrivacy so I can not change the behivor and the
> only
> > > way
> > > > > to fix it is to write another piece of ugly code...
> > > > >
> > > > > https://issues.apache.org/jira/browse/HADOOP-13433
> > > > >
> > > > > 2016-09-23 12:53 GMT+08:00 Vladimir Rodionov <
> vladrodionov@gmail.com
> > >:
> > > > >
> > > > > > >> If in the future, we find better ways of doing this without
> > using
> > > > MR,
> > > > > we
> > > > > > can certainly consider that
> > > > > >
> > > > > > Our framework for distributed operations is abstract and allows
> > > > > > different implementations. MR is just one implementation we
> > provide.
> > > > > >
> > > > > > -Vlad
> > > > > >
> > > > > > On Thu, Sep 22, 2016 at 9:38 PM, Devaraj Das <
> ddas@hortonworks.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Guys, first off apologies for bringing in the topic of MR-based
> > > > > > > compactions.. But I was thinking more about the SpliceMachine
> > > > approach
> > > > > of
> > > > > > > managing compactions in Spark where apparently they saw a lot
> of
> > > > > > benefits.
> > > > > > > Apologies for giving you that sore throat Andrew; I really
> didn't
> > > > mean
> > > > > to
> > > > > > > :-)
> > > > > > >
> > > > > > > So on this issue, we have these on the plate:
> > > > > > > 0. Somehow not use MR but something like that
> > > > > > > 1. Run a standalone service other than master
> > > > > > > 2. Shell out from the master
> > > > > > >
> > > > > > > I don't think we have a good answer to (0), and I don't think
> > it's
> > > > even
> > > > > > > worth the effort of trying to build something when MR is
> already
> > > > there,
> > > > > > and
> > > > > > > being used by HBase already for some operations.
> > > > > > >
> > > > > > > On (1), we have to deal with a myriad of issues - HA of the
> > server
> > > > not
> > > > > > > being the least of them all. Security (kerberos authentication,
> > > > another
> > > > > > > keytab to manage, etc. etc. etc.). IMO, that approach is DOA.
> > > Instead
> > > > > > let's
> > > > > > > substitute that (1) with the HBase Master. I haven't seen any
> > good
> > > > > reason
> > > > > > > why the HBase master shouldn't launch MR jobs if needed. It's
> not
> > > > > ideal;
> > > > > > > agreed.
> > > > > > >
> > > > > > > Now before going to (2), let's see what are the benefits of
> > running
> > > > the
> > > > > > > backup/restore jobs from the master. I think Ted has summarized
> > > some
> > > > of
> > > > > > the
> > > > > > > issues that we need to take care of - basically, the master can
> > > keep
> > > > > > track
> > > > > > > of running jobs, and should it fail, the backup master can
> > continue
> > > > > > keeping
> > > > > > > track of it (since the jobId would have been recorded in the
> proc
> > > > WAL).
> > > > > > The
> > > > > > > master can also do cleanup, etc. of failed backup/restore
> > > processes.
> > > > > > > Security is another issue - the job needs to run as 'hbase'
> since
> > > it
> > > > > owns
> > > > > > > the data. Having the master launch the job makes it get that
> > > > privilege.
> > > > > > In
> > > > > > > the (2) approach, it's hard to do some of the above management.
> > > > > > >
> > > > > > > Guys, just to reiterate, the patch as such is ready from the
> > > overall
> > > > > > > design/arch point of view (maybe code review is still pending
> > from
> > > > > > Matteo).
> > > > > > > If in the future, we find better ways of doing this without
> using
> > > MR,
> > > > > we
> > > > > > > can certainly consider that. But IMO don't think we should
> block
> > > this
> > > > > > patch
> > > > > > > from getting merged.
> > > > > > >
> > > > > > > ________________________________________
> > > > > > > From: 张铎 <pa...@gmail.com>
> > > > > > > Sent: Thursday, September 22, 2016 8:32 PM
> > > > > > > To: dev@hbase.apache.org
> > > > > > > Subject: Re: [DISCUSSION] MR jobs started by Master or RS
> > > > > > >
> > > > > > > So what about a standalone service other than master? You can
> use
> > > > your
> > > > > > own
> > > > > > > procedure store in that service?
> > > > > > >
> > > > > > > 2016-09-23 11:28 GMT+08:00 Ted Yu <yu...@gmail.com>:
> > > > > > >
> > > > > > > > An earlier implementation was client driven.
> > > > > > > >
> > > > > > > > But with that approach, it is hard to resume if there is
> error
> > > > > midway.
> > > > > > > > Using Procedure V2 makes the backup / restore more robust.
> > > > > > > >
> > > > > > > > Another consideration is for security. It is hard to enforce
> > > > security
> > > > > > (to
> > > > > > > > be implemented) for client driven actions.
> > > > > > > >
> > > > > > > > Cheers
> > > > > > > >
> > > > > > > > > On Sep 22, 2016, at 8:15 PM, Andrew Purtell <
> > > > > > andrew.purtell@gmail.com>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > No, this misses Matteo's finer point, which is "shelling
> out"
> > > > from
> > > > > > the
> > > > > > > > master directly to run MR is a first. Why not drive this
> with a
> > > > > utility
> > > > > > > > derived from Tool?
> > > > > > > > >
> > > > > > > > > On Sep 22, 2016, at 7:57 PM, Vladimir Rodionov <
> > > > > > vladrodionov@gmail.com
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >>>> In our production cluster, it is a common case we just
> > have
> > > > > HDFS
> > > > > > > and
> > > > > > > > >>>> HBase deployed.
> > > > > > > > >>>> If our Master/RS depend on MR framework (especially some
> > > > > features
> > > > > > we
> > > > > > > > >>>> have not used at all), it introduced another cost for
> > > > maintain.
> > > > > > I
> > > > > > > > >>>> don't think it is a good idea.
> > > > > > > > >>
> > > > > > > > >> So , you are not backup users in this case. Many our
> > customers
> > > > > have
> > > > > > > full
> > > > > > > > >> stack deployed and
> > > > > > > > >> want see backup to be a standard feature. Besides this,
> > > nothing
> > > > > will
> > > > > > > > happen
> > > > > > > > >> in your cluster
> > > > > > > > >> if you won't be doing backups.
> > > > > > > > >>
> > > > > > > > >> This discussion (we do not want see M/R dependency) goes
> to
> > > > > nowhere.
> > > > > > > We
> > > > > > > > >> asked already, at least twice, to suggest another
> framework
> > > > (other
> > > > > > > than
> > > > > > > > M/R)
> > > > > > > > >> for bulk data copy with *conversion*. Still waiting for
> > > > > suggestions.
> > > > > > > > >>
> > > > > > > > >> -Vlad
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>> On Thu, Sep 22, 2016 at 7:49 PM, Ted Yu <
> > yuzhihong@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > > >>>
> > > > > > > > >>> If MR framework is not deployed in the cluster, hbase
> still
> > > > > > functions
> > > > > > > > >>> normally (post merge).
> > > > > > > > >>>
> > > > > > > > >>> In terms of build time dependency, we have long been
> > > depending
> > > > on
> > > > > > > > >>> mapreduce. Take a look at ExportSnapshot.
> > > > > > > > >>>
> > > > > > > > >>> Cheers
> > > > > > > > >>>
> > > > > > > > >>> On Thu, Sep 22, 2016 at 7:42 PM, Heng Chen <
> > > > > > heng.chen.1986@gmail.com
> > > > > > > >
> > > > > > > > >>> wrote:
> > > > > > > > >>>
> > > > > > > > >>>> In our production cluster, it is a common case we just
> > have
> > > > > HDFS
> > > > > > > and
> > > > > > > > >>>> HBase deployed.
> > > > > > > > >>>> If our Master/RS depend on MR framework (especially some
> > > > > features
> > > > > > we
> > > > > > > > >>>> have not used at all), it introduced another cost for
> > > > maintain.
> > > > > > I
> > > > > > > > >>>> don't think it is a good idea.
> > > > > > > > >>>>
> > > > > > > > >>>> 2016-09-23 10:28 GMT+08:00 张铎 <pa...@gmail.com>:
> > > > > > > > >>>>> To be specific, for example, our nice Backup/Restore
> > > feature,
> > > > > if
> > > > > > we
> > > > > > > > >>> think
> > > > > > > > >>>>> this is not a core feature of HBase, then we could make
> > it
> > > > > depend
> > > > > > > on
> > > > > > > > >>> MR,
> > > > > > > > >>>>> and start a standalone BackupManager instance that
> > submits
> > > MR
> > > > > > jobs
> > > > > > > to
> > > > > > > > >>> do
> > > > > > > > >>>>> periodical maintenance job. And if we think this is a
> > core
> > > > > > feature
> > > > > > > > that
> > > > > > > > >>>>> everyone should use it, then we'd better implement it
> > > without
> > > > > MR
> > > > > > > > >>>>> dependency, like DLS.
> > > > > > > > >>>>>
> > > > > > > > >>>>> Thanks.
> > > > > > > > >>>>>
> > > > > > > > >>>>> 2016-09-23 10:11 GMT+08:00 张铎 <pa...@gmail.com>:
> > > > > > > > >>>>>
> > > > > > > > >>>>>> I‘m -1 on let master or rs launch MR jobs. It is OK
> that
> > > > some
> > > > > of
> > > > > > > our
> > > > > > > > >>>>>> features depend on MR but I think the bottom line is
> > that
> > > we
> > > > > > > should
> > > > > > > > >>>> launch
> > > > > > > > >>>>>> the jobs from outside manually or by other services.
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> 2016-09-23 9:47 GMT+08:00 Andrew Purtell <
> > > > > > > andrew.purtell@gmail.com
> > > > > > > > >:
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>> Ok, got it. Well "shelling out" is on the line I
> think,
> > > so
> > > > a
> > > > > > fair
> > > > > > > > >>>>>>> question.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> Can this be driven by a utility derived from Tool
> like
> > > our
> > > > > > other
> > > > > > > MR
> > > > > > > > >>>> apps?
> > > > > > > > >>>>>>> The issue is needing the AccessController to decide
> if
> > > > > allowed?
> > > > > > > But
> > > > > > > > >>>> nothing
> > > > > > > > >>>>>>> prevents the user from running the job
> > > > > manually/independently,
> > > > > > > > right?
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>>> On Sep 22, 2016, at 3:44 PM, Matteo Bertozzi <
> > > > > > > > >>>> theo.bertozzi@gmail.com>
> > > > > > > > >>>>>>> wrote:
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> just a remark. my query was not about tools using MR
> > > > > > (everyone i
> > > > > > > > >>>> think
> > > > > > > > >>>>>>> is
> > > > > > > > >>>>>>>> ok with those).
> > > > > > > > >>>>>>>> the topic was about: "are we ok with running MR jobs
> > > from
> > > > > > Master
> > > > > > > > >>> and
> > > > > > > > >>>> RSs
> > > > > > > > >>>>>>>> code?" since this will be the first time we do this
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> Matteo
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>>> On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das <
> > > > > > > > >>> ddas@hortonworks.com>
> > > > > > > > >>>>>>> wrote:
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> Very much agree; for tools like ExportSnapshot /
> > > Backup /
> > > > > > > > Restore,
> > > > > > > > >>>> it's
> > > > > > > > >>>>>>>>> fine to be dependent on MR. MR is the right
> framework
> > > for
> > > > > > such.
> > > > > > > > We
> > > > > > > > >>>>>>> should
> > > > > > > > >>>>>>>>> also do compactions using MR (just saying :) )
> > > > > > > > >>>>>>>>> ________________________________________
> > > > > > > > >>>>>>>>> From: Ted Yu <yu...@gmail.com>
> > > > > > > > >>>>>>>>> Sent: Thursday, September 22, 2016 2:00 PM
> > > > > > > > >>>>>>>>> To: dev@hbase.apache.org
> > > > > > > > >>>>>>>>> Subject: Re: [DISCUSSION] MR jobs started by Master
> > or
> > > RS
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> I agree - backup / restore is in the same category
> as
> > > > > import
> > > > > > /
> > > > > > > > >>>> export.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell <
> > > > > > > > >>>>>>> andrew.purtell@gmail.com>
> > > > > > > > >>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>>> Backup is extra tooling around core in my opinion.
> > > Like
> > > > > > import
> > > > > > > > or
> > > > > > > > >>>>>>> export.
> > > > > > > > >>>>>>>>>> Or the optional MOB tool. It's fine.
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>>> On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi <
> > > > > > > > >>>> mbertozzi@apache.org>
> > > > > > > > >>>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> What's the latest opinion around running MR jobs
> > from
> > > > > hbase
> > > > > > > > >>>> (Master
> > > > > > > > >>>>>>> or
> > > > > > > > >>>>>>>>>> RS)?
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> I remember in the past that there was discussion
> > > about
> > > > > not
> > > > > > > > >>> having
> > > > > > > > >>>> MR
> > > > > > > > >>>>>>>>> has
> > > > > > > > >>>>>>>>>>> direct dependency of hbase.
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> I think some of discussion where around MOB that
> > had
> > > a
> > > > MR
> > > > > > job
> > > > > > > > to
> > > > > > > > >>>>>>>>> compact,
> > > > > > > > >>>>>>>>>>> that later was transformed in a non-MR job to be
> > > > merged,
> > > > > I
> > > > > > > > think
> > > > > > > > >>>> we
> > > > > > > > >>>>>>>>> had a
> > > > > > > > >>>>>>>>>>> similar discussion for log split/replay.
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> the latest is the new Backup feature
> (HBASE-7912),
> > > that
> > > > > > runs
> > > > > > > a
> > > > > > > > >>> MR
> > > > > > > > >>>> job
> > > > > > > > >>>>>>>>>> from
> > > > > > > > >>>>>>>>>>> the master to copy data or restore data.
> > > > > > > > >>>>>>>>>>> (backup is also "not really core" as in.. if you
> > > don't
> > > > > use
> > > > > > > > >>> backup
> > > > > > > > >>>>>>>>> you'll
> > > > > > > > >>>>>>>>>>> not end up running MR jobs, but this was probably
> > > true
> > > > > for
> > > > > > > MOB
> > > > > > > > >>> as
> > > > > > > > >>>> in
> > > > > > > > >>>>>>>>> "if
> > > > > > > > >>>>>>>>>>> you don't enable MOB you don't need MR")
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> any thoughts? do we a rule that says "we don't
> want
> > > to
> > > > > have
> > > > > > > > >>> hbase
> > > > > > > > >>>> run
> > > > > > > > >>>>>>>>> MR
> > > > > > > > >>>>>>>>>>> jobs, only tool started manually by the user can
> do
> > > > > that".
> > > > > > or
> > > > > > > > >>> can
> > > > > > > > >>>> we
> > > > > > > > >>>>>>>>>> start
> > > > > > > > >>>>>>>>>>> adding MR calls around without problems?
> > > > > > > > >>>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSSION] MR jobs started by Master or RS
Posted by Nick Dimiduk <nd...@gmail.com>.
So we support MR/YARN anyway through these optional features, thus
burdening is with a baseline of maintenance. Since we have that anyway, do
we accept _optional_ implementations of "core" features based on MR it even
YARN? I'd guess HBase is co-deployed with a YARN or Spark cluster often
enough that shipping alternative implementations that can take advantage of
this infrastructure is quite reasonable -- even thoughtful!
The trouble with multiple implementations is the risk of one path or
another left to languish. We have a means to deal with this already, which
is to say if there's a champion up for maintaining it, and it makes good
sense, we accept it.
On Friday, September 23, 2016, Matteo Bertozzi <th...@gmail.com>
wrote:
> let me try to go back to my original topic.
> this question was meant to be generic, and provide some rule for future
> code.
>
> from what I can gather, a rule that may satisfy everyone can be:
> - we don't want any core feature (e.g. compaction/log-split/log-reply)
> over MR, because some cluster may not want or may have an
> external/uncontrolled MR setup.
> - we allow non-core features (e.g. features enabled by a flag) to run MR
> jobs from hbase, because unless you use the feature, MR is not required.
>
> Matteo
>
>
> On Fri, Sep 23, 2016 at 5:39 AM, Ted Yu <yuzhihong@gmail.com
> <javascript:;>> wrote:
>
> > I suggest you look at Matteo's work for AssignmentManager which is to
> make
> > Master more stable.
> >
> > Cheers
> >
> > On Fri, Sep 23, 2016 at 5:32 AM, 张铎 <palomino219@gmail.com
> <javascript:;>> wrote:
> >
> > > No, not your fault, at lease, not this time:)
> > >
> > > Why I call the code ugly? Can you simply tell me the sequence of calls
> > when
> > > starting up the HMaster? HMaster is also a regionserver so it extends
> > > HRegionServer, and the initialization of HRegionServer sometimes needs
> to
> > > make rpc calls to HMaster. A simple change would cause probabilistic
> dead
> > > lock or some strange NPEs...
> > >
> > > That's why I'm very nervous when somebody wants to add new features or
> > add
> > > external dependencies to HMaster, especially add more works for the
> start
> > > up processing...
> > >
> > > Thanks.
> > >
> > > 2016-09-23 20:02 GMT+08:00 Ted Yu <yuzhihong@gmail.com <javascript:;>
> >:
> > >
> > > > I read through HADOOP-13433
> > > > <https://issues.apache.org/jira/browse/HADOOP-13433> - the cited
> race
> > > > condition is in jdk.
> > > >
> > > > Suggest pinging the reviewer on JIRA to get it moving.
> > > >
> > > > bq. But the ugly code in HMaster is readlly a problem...
> > > >
> > > > Can you be specific as to which code is ugly ? Is it in the backup /
> > > > restore mega patch ?
> > > >
> > > > Cheers
> > > >
> > > > On Thu, Sep 22, 2016 at 10:44 PM, 张铎 <palomino219@gmail.com
> <javascript:;>> wrote:
> > > >
> > > > > If you guys have already implemented the feature in the MR way and
> > the
> > > > > patch is ready for landing on master, I'm a -0 on it as I do not
> want
> > > to
> > > > > block the development progress.
> > > > >
> > > > > But I strongly suggest later we need to revisit the design and see
> if
> > > we
> > > > > can seperated the logic from HMaster as much as possible. HA is
> not a
> > > big
> > > > > problem if you do not store any metada locally. But the ugly code
> in
> > > > > HMaster is readlly a problem...
> > > > >
> > > > > And for security, I have a issue pending for a long time. Can
> someone
> > > > help
> > > > > taking a simple look at it? This is what I mean, ugly code...
> logout
> > > and
> > > > > destroy the credentials in a subject when it is still being used,
> and
> > > > > declared as LimitPrivacy so I can not change the behivor and the
> only
> > > way
> > > > > to fix it is to write another piece of ugly code...
> > > > >
> > > > > https://issues.apache.org/jira/browse/HADOOP-13433
> > > > >
> > > > > 2016-09-23 12:53 GMT+08:00 Vladimir Rodionov <
> vladrodionov@gmail.com <javascript:;>
> > >:
> > > > >
> > > > > > >> If in the future, we find better ways of doing this without
> > using
> > > > MR,
> > > > > we
> > > > > > can certainly consider that
> > > > > >
> > > > > > Our framework for distributed operations is abstract and allows
> > > > > > different implementations. MR is just one implementation we
> > provide.
> > > > > >
> > > > > > -Vlad
> > > > > >
> > > > > > On Thu, Sep 22, 2016 at 9:38 PM, Devaraj Das <
> ddas@hortonworks.com <javascript:;>
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Guys, first off apologies for bringing in the topic of MR-based
> > > > > > > compactions.. But I was thinking more about the SpliceMachine
> > > > approach
> > > > > of
> > > > > > > managing compactions in Spark where apparently they saw a lot
> of
> > > > > > benefits.
> > > > > > > Apologies for giving you that sore throat Andrew; I really
> didn't
> > > > mean
> > > > > to
> > > > > > > :-)
> > > > > > >
> > > > > > > So on this issue, we have these on the plate:
> > > > > > > 0. Somehow not use MR but something like that
> > > > > > > 1. Run a standalone service other than master
> > > > > > > 2. Shell out from the master
> > > > > > >
> > > > > > > I don't think we have a good answer to (0), and I don't think
> > it's
> > > > even
> > > > > > > worth the effort of trying to build something when MR is
> already
> > > > there,
> > > > > > and
> > > > > > > being used by HBase already for some operations.
> > > > > > >
> > > > > > > On (1), we have to deal with a myriad of issues - HA of the
> > server
> > > > not
> > > > > > > being the least of them all. Security (kerberos authentication,
> > > > another
> > > > > > > keytab to manage, etc. etc. etc.). IMO, that approach is DOA.
> > > Instead
> > > > > > let's
> > > > > > > substitute that (1) with the HBase Master. I haven't seen any
> > good
> > > > > reason
> > > > > > > why the HBase master shouldn't launch MR jobs if needed. It's
> not
> > > > > ideal;
> > > > > > > agreed.
> > > > > > >
> > > > > > > Now before going to (2), let's see what are the benefits of
> > running
> > > > the
> > > > > > > backup/restore jobs from the master. I think Ted has summarized
> > > some
> > > > of
> > > > > > the
> > > > > > > issues that we need to take care of - basically, the master can
> > > keep
> > > > > > track
> > > > > > > of running jobs, and should it fail, the backup master can
> > continue
> > > > > > keeping
> > > > > > > track of it (since the jobId would have been recorded in the
> proc
> > > > WAL).
> > > > > > The
> > > > > > > master can also do cleanup, etc. of failed backup/restore
> > > processes.
> > > > > > > Security is another issue - the job needs to run as 'hbase'
> since
> > > it
> > > > > owns
> > > > > > > the data. Having the master launch the job makes it get that
> > > > privilege.
> > > > > > In
> > > > > > > the (2) approach, it's hard to do some of the above management.
> > > > > > >
> > > > > > > Guys, just to reiterate, the patch as such is ready from the
> > > overall
> > > > > > > design/arch point of view (maybe code review is still pending
> > from
> > > > > > Matteo).
> > > > > > > If in the future, we find better ways of doing this without
> using
> > > MR,
> > > > > we
> > > > > > > can certainly consider that. But IMO don't think we should
> block
> > > this
> > > > > > patch
> > > > > > > from getting merged.
> > > > > > >
> > > > > > > ________________________________________
> > > > > > > From: 张铎 <palomino219@gmail.com <javascript:;>>
> > > > > > > Sent: Thursday, September 22, 2016 8:32 PM
> > > > > > > To: dev@hbase.apache.org <javascript:;>
> > > > > > > Subject: Re: [DISCUSSION] MR jobs started by Master or RS
> > > > > > >
> > > > > > > So what about a standalone service other than master? You can
> use
> > > > your
> > > > > > own
> > > > > > > procedure store in that service?
> > > > > > >
> > > > > > > 2016-09-23 11:28 GMT+08:00 Ted Yu <yuzhihong@gmail.com
> <javascript:;>>:
> > > > > > >
> > > > > > > > An earlier implementation was client driven.
> > > > > > > >
> > > > > > > > But with that approach, it is hard to resume if there is
> error
> > > > > midway.
> > > > > > > > Using Procedure V2 makes the backup / restore more robust.
> > > > > > > >
> > > > > > > > Another consideration is for security. It is hard to enforce
> > > > security
> > > > > > (to
> > > > > > > > be implemented) for client driven actions.
> > > > > > > >
> > > > > > > > Cheers
> > > > > > > >
> > > > > > > > > On Sep 22, 2016, at 8:15 PM, Andrew Purtell <
> > > > > > andrew.purtell@gmail.com <javascript:;>>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > No, this misses Matteo's finer point, which is "shelling
> out"
> > > > from
> > > > > > the
> > > > > > > > master directly to run MR is a first. Why not drive this
> with a
> > > > > utility
> > > > > > > > derived from Tool?
> > > > > > > > >
> > > > > > > > > On Sep 22, 2016, at 7:57 PM, Vladimir Rodionov <
> > > > > > vladrodionov@gmail.com <javascript:;>
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >>>> In our production cluster, it is a common case we just
> > have
> > > > > HDFS
> > > > > > > and
> > > > > > > > >>>> HBase deployed.
> > > > > > > > >>>> If our Master/RS depend on MR framework (especially some
> > > > > features
> > > > > > we
> > > > > > > > >>>> have not used at all), it introduced another cost for
> > > > maintain.
> > > > > > I
> > > > > > > > >>>> don't think it is a good idea.
> > > > > > > > >>
> > > > > > > > >> So , you are not backup users in this case. Many our
> > customers
> > > > > have
> > > > > > > full
> > > > > > > > >> stack deployed and
> > > > > > > > >> want see backup to be a standard feature. Besides this,
> > > nothing
> > > > > will
> > > > > > > > happen
> > > > > > > > >> in your cluster
> > > > > > > > >> if you won't be doing backups.
> > > > > > > > >>
> > > > > > > > >> This discussion (we do not want see M/R dependency) goes
> to
> > > > > nowhere.
> > > > > > > We
> > > > > > > > >> asked already, at least twice, to suggest another
> framework
> > > > (other
> > > > > > > than
> > > > > > > > M/R)
> > > > > > > > >> for bulk data copy with *conversion*. Still waiting for
> > > > > suggestions.
> > > > > > > > >>
> > > > > > > > >> -Vlad
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>> On Thu, Sep 22, 2016 at 7:49 PM, Ted Yu <
> > yuzhihong@gmail.com <javascript:;>
> > > >
> > > > > > wrote:
> > > > > > > > >>>
> > > > > > > > >>> If MR framework is not deployed in the cluster, hbase
> still
> > > > > > functions
> > > > > > > > >>> normally (post merge).
> > > > > > > > >>>
> > > > > > > > >>> In terms of build time dependency, we have long been
> > > depending
> > > > on
> > > > > > > > >>> mapreduce. Take a look at ExportSnapshot.
> > > > > > > > >>>
> > > > > > > > >>> Cheers
> > > > > > > > >>>
> > > > > > > > >>> On Thu, Sep 22, 2016 at 7:42 PM, Heng Chen <
> > > > > > heng.chen.1986@gmail.com <javascript:;>
> > > > > > > >
> > > > > > > > >>> wrote:
> > > > > > > > >>>
> > > > > > > > >>>> In our production cluster, it is a common case we just
> > have
> > > > > HDFS
> > > > > > > and
> > > > > > > > >>>> HBase deployed.
> > > > > > > > >>>> If our Master/RS depend on MR framework (especially some
> > > > > features
> > > > > > we
> > > > > > > > >>>> have not used at all), it introduced another cost for
> > > > maintain.
> > > > > > I
> > > > > > > > >>>> don't think it is a good idea.
> > > > > > > > >>>>
> > > > > > > > >>>> 2016-09-23 10:28 GMT+08:00 张铎 <palomino219@gmail.com
> <javascript:;>>:
> > > > > > > > >>>>> To be specific, for example, our nice Backup/Restore
> > > feature,
> > > > > if
> > > > > > we
> > > > > > > > >>> think
> > > > > > > > >>>>> this is not a core feature of HBase, then we could make
> > it
> > > > > depend
> > > > > > > on
> > > > > > > > >>> MR,
> > > > > > > > >>>>> and start a standalone BackupManager instance that
> > submits
> > > MR
> > > > > > jobs
> > > > > > > to
> > > > > > > > >>> do
> > > > > > > > >>>>> periodical maintenance job. And if we think this is a
> > core
> > > > > > feature
> > > > > > > > that
> > > > > > > > >>>>> everyone should use it, then we'd better implement it
> > > without
> > > > > MR
> > > > > > > > >>>>> dependency, like DLS.
> > > > > > > > >>>>>
> > > > > > > > >>>>> Thanks.
> > > > > > > > >>>>>
> > > > > > > > >>>>> 2016-09-23 10:11 GMT+08:00 张铎 <palomino219@gmail.com
> <javascript:;>>:
> > > > > > > > >>>>>
> > > > > > > > >>>>>> I‘m -1 on let master or rs launch MR jobs. It is OK
> that
> > > > some
> > > > > of
> > > > > > > our
> > > > > > > > >>>>>> features depend on MR but I think the bottom line is
> > that
> > > we
> > > > > > > should
> > > > > > > > >>>> launch
> > > > > > > > >>>>>> the jobs from outside manually or by other services.
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> 2016-09-23 9:47 GMT+08:00 Andrew Purtell <
> > > > > > > andrew.purtell@gmail.com <javascript:;>
> > > > > > > > >:
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>> Ok, got it. Well "shelling out" is on the line I
> think,
> > > so
> > > > a
> > > > > > fair
> > > > > > > > >>>>>>> question.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> Can this be driven by a utility derived from Tool
> like
> > > our
> > > > > > other
> > > > > > > MR
> > > > > > > > >>>> apps?
> > > > > > > > >>>>>>> The issue is needing the AccessController to decide
> if
> > > > > allowed?
> > > > > > > But
> > > > > > > > >>>> nothing
> > > > > > > > >>>>>>> prevents the user from running the job
> > > > > manually/independently,
> > > > > > > > right?
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>>> On Sep 22, 2016, at 3:44 PM, Matteo Bertozzi <
> > > > > > > > >>>> theo.bertozzi@gmail.com <javascript:;>>
> > > > > > > > >>>>>>> wrote:
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> just a remark. my query was not about tools using MR
> > > > > > (everyone i
> > > > > > > > >>>> think
> > > > > > > > >>>>>>> is
> > > > > > > > >>>>>>>> ok with those).
> > > > > > > > >>>>>>>> the topic was about: "are we ok with running MR jobs
> > > from
> > > > > > Master
> > > > > > > > >>> and
> > > > > > > > >>>> RSs
> > > > > > > > >>>>>>>> code?" since this will be the first time we do this
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> Matteo
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>>> On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das <
> > > > > > > > >>> ddas@hortonworks.com <javascript:;>>
> > > > > > > > >>>>>>> wrote:
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> Very much agree; for tools like ExportSnapshot /
> > > Backup /
> > > > > > > > Restore,
> > > > > > > > >>>> it's
> > > > > > > > >>>>>>>>> fine to be dependent on MR. MR is the right
> framework
> > > for
> > > > > > such.
> > > > > > > > We
> > > > > > > > >>>>>>> should
> > > > > > > > >>>>>>>>> also do compactions using MR (just saying :) )
> > > > > > > > >>>>>>>>> ________________________________________
> > > > > > > > >>>>>>>>> From: Ted Yu <yuzhihong@gmail.com <javascript:;>>
> > > > > > > > >>>>>>>>> Sent: Thursday, September 22, 2016 2:00 PM
> > > > > > > > >>>>>>>>> To: dev@hbase.apache.org <javascript:;>
> > > > > > > > >>>>>>>>> Subject: Re: [DISCUSSION] MR jobs started by Master
> > or
> > > RS
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> I agree - backup / restore is in the same category
> as
> > > > > import
> > > > > > /
> > > > > > > > >>>> export.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell <
> > > > > > > > >>>>>>> andrew.purtell@gmail.com <javascript:;>>
> > > > > > > > >>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>>> Backup is extra tooling around core in my opinion.
> > > Like
> > > > > > import
> > > > > > > > or
> > > > > > > > >>>>>>> export.
> > > > > > > > >>>>>>>>>> Or the optional MOB tool. It's fine.
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>>> On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi <
> > > > > > > > >>>> mbertozzi@apache.org <javascript:;>>
> > > > > > > > >>>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> What's the latest opinion around running MR jobs
> > from
> > > > > hbase
> > > > > > > > >>>> (Master
> > > > > > > > >>>>>>> or
> > > > > > > > >>>>>>>>>> RS)?
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> I remember in the past that there was discussion
> > > about
> > > > > not
> > > > > > > > >>> having
> > > > > > > > >>>> MR
> > > > > > > > >>>>>>>>> has
> > > > > > > > >>>>>>>>>>> direct dependency of hbase.
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> I think some of discussion where around MOB that
> > had
> > > a
> > > > MR
> > > > > > job
> > > > > > > > to
> > > > > > > > >>>>>>>>> compact,
> > > > > > > > >>>>>>>>>>> that later was transformed in a non-MR job to be
> > > > merged,
> > > > > I
> > > > > > > > think
> > > > > > > > >>>> we
> > > > > > > > >>>>>>>>> had a
> > > > > > > > >>>>>>>>>>> similar discussion for log split/replay.
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> the latest is the new Backup feature
> (HBASE-7912),
> > > that
> > > > > > runs
> > > > > > > a
> > > > > > > > >>> MR
> > > > > > > > >>>> job
> > > > > > > > >>>>>>>>>> from
> > > > > > > > >>>>>>>>>>> the master to copy data or restore data.
> > > > > > > > >>>>>>>>>>> (backup is also "not really core" as in.. if you
> > > don't
> > > > > use
> > > > > > > > >>> backup
> > > > > > > > >>>>>>>>> you'll
> > > > > > > > >>>>>>>>>>> not end up running MR jobs, but this was probably
> > > true
> > > > > for
> > > > > > > MOB
> > > > > > > > >>> as
> > > > > > > > >>>> in
> > > > > > > > >>>>>>>>> "if
> > > > > > > > >>>>>>>>>>> you don't enable MOB you don't need MR")
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> any thoughts? do we a rule that says "we don't
> want
> > > to
> > > > > have
> > > > > > > > >>> hbase
> > > > > > > > >>>> run
> > > > > > > > >>>>>>>>> MR
> > > > > > > > >>>>>>>>>>> jobs, only tool started manually by the user can
> do
> > > > > that".
> > > > > > or
> > > > > > > > >>> can
> > > > > > > > >>>> we
> > > > > > > > >>>>>>>>>> start
> > > > > > > > >>>>>>>>>>> adding MR calls around without problems?
> > > > > > > > >>>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSSION] MR jobs started by Master or RS
Posted by Jerry He <je...@gmail.com>.
That is the point, Matteo.
Put it another way, there is nothing that stops a user from deploying
custom procedure, custom co-processor that calls out MR job.
The optional feature should satisfy some basic requirements. .e.g. No
impact if not deployed or used. Limited impact if used.
It can be made with isolated dynamic loading of extra configuration (Yarn),
non-blocking non-occupying on the server handlers, or separate handler.
The impact would mostly be on the overall cluster resources. In this sense,
there is no difference, using another standalone server or a command tool.
The exportEnapshot can then be moved to the server as well.
Also, thinking about in the higher level. It is probably beneficial if you
allow HBase to call out an external framework to do computation. It can be
think of as a UDF, a distributed UDF.
The execution of this UDF is totally in separate address spaces, and you
only need to poll the status. This would be like a dream in traditional
database.
My 2 cents.
Jerry
On Fri, Sep 23, 2016 at 6:46 AM, Matteo Bertozzi <th...@gmail.com>
wrote:
> let me try to go back to my original topic.
> this question was meant to be generic, and provide some rule for future
> code.
>
> from what I can gather, a rule that may satisfy everyone can be:
> - we don't want any core feature (e.g. compaction/log-split/log-reply)
> over MR, because some cluster may not want or may have an
> external/uncontrolled MR setup.
> - we allow non-core features (e.g. features enabled by a flag) to run MR
> jobs from hbase, because unless you use the feature, MR is not required.
>
> Matteo
>
>
> On Fri, Sep 23, 2016 at 5:39 AM, Ted Yu <yu...@gmail.com> wrote:
>
> > I suggest you look at Matteo's work for AssignmentManager which is to
> make
> > Master more stable.
> >
> > Cheers
> >
> > On Fri, Sep 23, 2016 at 5:32 AM, 张铎 <pa...@gmail.com> wrote:
> >
> > > No, not your fault, at lease, not this time:)
> > >
> > > Why I call the code ugly? Can you simply tell me the sequence of calls
> > when
> > > starting up the HMaster? HMaster is also a regionserver so it extends
> > > HRegionServer, and the initialization of HRegionServer sometimes needs
> to
> > > make rpc calls to HMaster. A simple change would cause probabilistic
> dead
> > > lock or some strange NPEs...
> > >
> > > That's why I'm very nervous when somebody wants to add new features or
> > add
> > > external dependencies to HMaster, especially add more works for the
> start
> > > up processing...
> > >
> > > Thanks.
> > >
> > > 2016-09-23 20:02 GMT+08:00 Ted Yu <yu...@gmail.com>:
> > >
> > > > I read through HADOOP-13433
> > > > <https://issues.apache.org/jira/browse/HADOOP-13433> - the cited
> race
> > > > condition is in jdk.
> > > >
> > > > Suggest pinging the reviewer on JIRA to get it moving.
> > > >
> > > > bq. But the ugly code in HMaster is readlly a problem...
> > > >
> > > > Can you be specific as to which code is ugly ? Is it in the backup /
> > > > restore mega patch ?
> > > >
> > > > Cheers
> > > >
> > > > On Thu, Sep 22, 2016 at 10:44 PM, 张铎 <pa...@gmail.com> wrote:
> > > >
> > > > > If you guys have already implemented the feature in the MR way and
> > the
> > > > > patch is ready for landing on master, I'm a -0 on it as I do not
> want
> > > to
> > > > > block the development progress.
> > > > >
> > > > > But I strongly suggest later we need to revisit the design and see
> if
> > > we
> > > > > can seperated the logic from HMaster as much as possible. HA is
> not a
> > > big
> > > > > problem if you do not store any metada locally. But the ugly code
> in
> > > > > HMaster is readlly a problem...
> > > > >
> > > > > And for security, I have a issue pending for a long time. Can
> someone
> > > > help
> > > > > taking a simple look at it? This is what I mean, ugly code...
> logout
> > > and
> > > > > destroy the credentials in a subject when it is still being used,
> and
> > > > > declared as LimitPrivacy so I can not change the behivor and the
> only
> > > way
> > > > > to fix it is to write another piece of ugly code...
> > > > >
> > > > > https://issues.apache.org/jira/browse/HADOOP-13433
> > > > >
> > > > > 2016-09-23 12:53 GMT+08:00 Vladimir Rodionov <
> vladrodionov@gmail.com
> > >:
> > > > >
> > > > > > >> If in the future, we find better ways of doing this without
> > using
> > > > MR,
> > > > > we
> > > > > > can certainly consider that
> > > > > >
> > > > > > Our framework for distributed operations is abstract and allows
> > > > > > different implementations. MR is just one implementation we
> > provide.
> > > > > >
> > > > > > -Vlad
> > > > > >
> > > > > > On Thu, Sep 22, 2016 at 9:38 PM, Devaraj Das <
> ddas@hortonworks.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Guys, first off apologies for bringing in the topic of MR-based
> > > > > > > compactions.. But I was thinking more about the SpliceMachine
> > > > approach
> > > > > of
> > > > > > > managing compactions in Spark where apparently they saw a lot
> of
> > > > > > benefits.
> > > > > > > Apologies for giving you that sore throat Andrew; I really
> didn't
> > > > mean
> > > > > to
> > > > > > > :-)
> > > > > > >
> > > > > > > So on this issue, we have these on the plate:
> > > > > > > 0. Somehow not use MR but something like that
> > > > > > > 1. Run a standalone service other than master
> > > > > > > 2. Shell out from the master
> > > > > > >
> > > > > > > I don't think we have a good answer to (0), and I don't think
> > it's
> > > > even
> > > > > > > worth the effort of trying to build something when MR is
> already
> > > > there,
> > > > > > and
> > > > > > > being used by HBase already for some operations.
> > > > > > >
> > > > > > > On (1), we have to deal with a myriad of issues - HA of the
> > server
> > > > not
> > > > > > > being the least of them all. Security (kerberos authentication,
> > > > another
> > > > > > > keytab to manage, etc. etc. etc.). IMO, that approach is DOA.
> > > Instead
> > > > > > let's
> > > > > > > substitute that (1) with the HBase Master. I haven't seen any
> > good
> > > > > reason
> > > > > > > why the HBase master shouldn't launch MR jobs if needed. It's
> not
> > > > > ideal;
> > > > > > > agreed.
> > > > > > >
> > > > > > > Now before going to (2), let's see what are the benefits of
> > running
> > > > the
> > > > > > > backup/restore jobs from the master. I think Ted has summarized
> > > some
> > > > of
> > > > > > the
> > > > > > > issues that we need to take care of - basically, the master can
> > > keep
> > > > > > track
> > > > > > > of running jobs, and should it fail, the backup master can
> > continue
> > > > > > keeping
> > > > > > > track of it (since the jobId would have been recorded in the
> proc
> > > > WAL).
> > > > > > The
> > > > > > > master can also do cleanup, etc. of failed backup/restore
> > > processes.
> > > > > > > Security is another issue - the job needs to run as 'hbase'
> since
> > > it
> > > > > owns
> > > > > > > the data. Having the master launch the job makes it get that
> > > > privilege.
> > > > > > In
> > > > > > > the (2) approach, it's hard to do some of the above management.
> > > > > > >
> > > > > > > Guys, just to reiterate, the patch as such is ready from the
> > > overall
> > > > > > > design/arch point of view (maybe code review is still pending
> > from
> > > > > > Matteo).
> > > > > > > If in the future, we find better ways of doing this without
> using
> > > MR,
> > > > > we
> > > > > > > can certainly consider that. But IMO don't think we should
> block
> > > this
> > > > > > patch
> > > > > > > from getting merged.
> > > > > > >
> > > > > > > ________________________________________
> > > > > > > From: 张铎 <pa...@gmail.com>
> > > > > > > Sent: Thursday, September 22, 2016 8:32 PM
> > > > > > > To: dev@hbase.apache.org
> > > > > > > Subject: Re: [DISCUSSION] MR jobs started by Master or RS
> > > > > > >
> > > > > > > So what about a standalone service other than master? You can
> use
> > > > your
> > > > > > own
> > > > > > > procedure store in that service?
> > > > > > >
> > > > > > > 2016-09-23 11:28 GMT+08:00 Ted Yu <yu...@gmail.com>:
> > > > > > >
> > > > > > > > An earlier implementation was client driven.
> > > > > > > >
> > > > > > > > But with that approach, it is hard to resume if there is
> error
> > > > > midway.
> > > > > > > > Using Procedure V2 makes the backup / restore more robust.
> > > > > > > >
> > > > > > > > Another consideration is for security. It is hard to enforce
> > > > security
> > > > > > (to
> > > > > > > > be implemented) for client driven actions.
> > > > > > > >
> > > > > > > > Cheers
> > > > > > > >
> > > > > > > > > On Sep 22, 2016, at 8:15 PM, Andrew Purtell <
> > > > > > andrew.purtell@gmail.com>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > No, this misses Matteo's finer point, which is "shelling
> out"
> > > > from
> > > > > > the
> > > > > > > > master directly to run MR is a first. Why not drive this
> with a
> > > > > utility
> > > > > > > > derived from Tool?
> > > > > > > > >
> > > > > > > > > On Sep 22, 2016, at 7:57 PM, Vladimir Rodionov <
> > > > > > vladrodionov@gmail.com
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >>>> In our production cluster, it is a common case we just
> > have
> > > > > HDFS
> > > > > > > and
> > > > > > > > >>>> HBase deployed.
> > > > > > > > >>>> If our Master/RS depend on MR framework (especially some
> > > > > features
> > > > > > we
> > > > > > > > >>>> have not used at all), it introduced another cost for
> > > > maintain.
> > > > > > I
> > > > > > > > >>>> don't think it is a good idea.
> > > > > > > > >>
> > > > > > > > >> So , you are not backup users in this case. Many our
> > customers
> > > > > have
> > > > > > > full
> > > > > > > > >> stack deployed and
> > > > > > > > >> want see backup to be a standard feature. Besides this,
> > > nothing
> > > > > will
> > > > > > > > happen
> > > > > > > > >> in your cluster
> > > > > > > > >> if you won't be doing backups.
> > > > > > > > >>
> > > > > > > > >> This discussion (we do not want see M/R dependency) goes
> to
> > > > > nowhere.
> > > > > > > We
> > > > > > > > >> asked already, at least twice, to suggest another
> framework
> > > > (other
> > > > > > > than
> > > > > > > > M/R)
> > > > > > > > >> for bulk data copy with *conversion*. Still waiting for
> > > > > suggestions.
> > > > > > > > >>
> > > > > > > > >> -Vlad
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>> On Thu, Sep 22, 2016 at 7:49 PM, Ted Yu <
> > yuzhihong@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > > >>>
> > > > > > > > >>> If MR framework is not deployed in the cluster, hbase
> still
> > > > > > functions
> > > > > > > > >>> normally (post merge).
> > > > > > > > >>>
> > > > > > > > >>> In terms of build time dependency, we have long been
> > > depending
> > > > on
> > > > > > > > >>> mapreduce. Take a look at ExportSnapshot.
> > > > > > > > >>>
> > > > > > > > >>> Cheers
> > > > > > > > >>>
> > > > > > > > >>> On Thu, Sep 22, 2016 at 7:42 PM, Heng Chen <
> > > > > > heng.chen.1986@gmail.com
> > > > > > > >
> > > > > > > > >>> wrote:
> > > > > > > > >>>
> > > > > > > > >>>> In our production cluster, it is a common case we just
> > have
> > > > > HDFS
> > > > > > > and
> > > > > > > > >>>> HBase deployed.
> > > > > > > > >>>> If our Master/RS depend on MR framework (especially some
> > > > > features
> > > > > > we
> > > > > > > > >>>> have not used at all), it introduced another cost for
> > > > maintain.
> > > > > > I
> > > > > > > > >>>> don't think it is a good idea.
> > > > > > > > >>>>
> > > > > > > > >>>> 2016-09-23 10:28 GMT+08:00 张铎 <pa...@gmail.com>:
> > > > > > > > >>>>> To be specific, for example, our nice Backup/Restore
> > > feature,
> > > > > if
> > > > > > we
> > > > > > > > >>> think
> > > > > > > > >>>>> this is not a core feature of HBase, then we could make
> > it
> > > > > depend
> > > > > > > on
> > > > > > > > >>> MR,
> > > > > > > > >>>>> and start a standalone BackupManager instance that
> > submits
> > > MR
> > > > > > jobs
> > > > > > > to
> > > > > > > > >>> do
> > > > > > > > >>>>> periodical maintenance job. And if we think this is a
> > core
> > > > > > feature
> > > > > > > > that
> > > > > > > > >>>>> everyone should use it, then we'd better implement it
> > > without
> > > > > MR
> > > > > > > > >>>>> dependency, like DLS.
> > > > > > > > >>>>>
> > > > > > > > >>>>> Thanks.
> > > > > > > > >>>>>
> > > > > > > > >>>>> 2016-09-23 10:11 GMT+08:00 张铎 <pa...@gmail.com>:
> > > > > > > > >>>>>
> > > > > > > > >>>>>> I‘m -1 on let master or rs launch MR jobs. It is OK
> that
> > > > some
> > > > > of
> > > > > > > our
> > > > > > > > >>>>>> features depend on MR but I think the bottom line is
> > that
> > > we
> > > > > > > should
> > > > > > > > >>>> launch
> > > > > > > > >>>>>> the jobs from outside manually or by other services.
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> 2016-09-23 9:47 GMT+08:00 Andrew Purtell <
> > > > > > > andrew.purtell@gmail.com
> > > > > > > > >:
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>> Ok, got it. Well "shelling out" is on the line I
> think,
> > > so
> > > > a
> > > > > > fair
> > > > > > > > >>>>>>> question.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> Can this be driven by a utility derived from Tool
> like
> > > our
> > > > > > other
> > > > > > > MR
> > > > > > > > >>>> apps?
> > > > > > > > >>>>>>> The issue is needing the AccessController to decide
> if
> > > > > allowed?
> > > > > > > But
> > > > > > > > >>>> nothing
> > > > > > > > >>>>>>> prevents the user from running the job
> > > > > manually/independently,
> > > > > > > > right?
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>>> On Sep 22, 2016, at 3:44 PM, Matteo Bertozzi <
> > > > > > > > >>>> theo.bertozzi@gmail.com>
> > > > > > > > >>>>>>> wrote:
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> just a remark. my query was not about tools using MR
> > > > > > (everyone i
> > > > > > > > >>>> think
> > > > > > > > >>>>>>> is
> > > > > > > > >>>>>>>> ok with those).
> > > > > > > > >>>>>>>> the topic was about: "are we ok with running MR jobs
> > > from
> > > > > > Master
> > > > > > > > >>> and
> > > > > > > > >>>> RSs
> > > > > > > > >>>>>>>> code?" since this will be the first time we do this
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> Matteo
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>>> On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das <
> > > > > > > > >>> ddas@hortonworks.com>
> > > > > > > > >>>>>>> wrote:
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> Very much agree; for tools like ExportSnapshot /
> > > Backup /
> > > > > > > > Restore,
> > > > > > > > >>>> it's
> > > > > > > > >>>>>>>>> fine to be dependent on MR. MR is the right
> framework
> > > for
> > > > > > such.
> > > > > > > > We
> > > > > > > > >>>>>>> should
> > > > > > > > >>>>>>>>> also do compactions using MR (just saying :) )
> > > > > > > > >>>>>>>>> ________________________________________
> > > > > > > > >>>>>>>>> From: Ted Yu <yu...@gmail.com>
> > > > > > > > >>>>>>>>> Sent: Thursday, September 22, 2016 2:00 PM
> > > > > > > > >>>>>>>>> To: dev@hbase.apache.org
> > > > > > > > >>>>>>>>> Subject: Re: [DISCUSSION] MR jobs started by Master
> > or
> > > RS
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> I agree - backup / restore is in the same category
> as
> > > > > import
> > > > > > /
> > > > > > > > >>>> export.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell <
> > > > > > > > >>>>>>> andrew.purtell@gmail.com>
> > > > > > > > >>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>>> Backup is extra tooling around core in my opinion.
> > > Like
> > > > > > import
> > > > > > > > or
> > > > > > > > >>>>>>> export.
> > > > > > > > >>>>>>>>>> Or the optional MOB tool. It's fine.
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>>> On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi <
> > > > > > > > >>>> mbertozzi@apache.org>
> > > > > > > > >>>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> What's the latest opinion around running MR jobs
> > from
> > > > > hbase
> > > > > > > > >>>> (Master
> > > > > > > > >>>>>>> or
> > > > > > > > >>>>>>>>>> RS)?
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> I remember in the past that there was discussion
> > > about
> > > > > not
> > > > > > > > >>> having
> > > > > > > > >>>> MR
> > > > > > > > >>>>>>>>> has
> > > > > > > > >>>>>>>>>>> direct dependency of hbase.
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> I think some of discussion where around MOB that
> > had
> > > a
> > > > MR
> > > > > > job
> > > > > > > > to
> > > > > > > > >>>>>>>>> compact,
> > > > > > > > >>>>>>>>>>> that later was transformed in a non-MR job to be
> > > > merged,
> > > > > I
> > > > > > > > think
> > > > > > > > >>>> we
> > > > > > > > >>>>>>>>> had a
> > > > > > > > >>>>>>>>>>> similar discussion for log split/replay.
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> the latest is the new Backup feature
> (HBASE-7912),
> > > that
> > > > > > runs
> > > > > > > a
> > > > > > > > >>> MR
> > > > > > > > >>>> job
> > > > > > > > >>>>>>>>>> from
> > > > > > > > >>>>>>>>>>> the master to copy data or restore data.
> > > > > > > > >>>>>>>>>>> (backup is also "not really core" as in.. if you
> > > don't
> > > > > use
> > > > > > > > >>> backup
> > > > > > > > >>>>>>>>> you'll
> > > > > > > > >>>>>>>>>>> not end up running MR jobs, but this was probably
> > > true
> > > > > for
> > > > > > > MOB
> > > > > > > > >>> as
> > > > > > > > >>>> in
> > > > > > > > >>>>>>>>> "if
> > > > > > > > >>>>>>>>>>> you don't enable MOB you don't need MR")
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> any thoughts? do we a rule that says "we don't
> want
> > > to
> > > > > have
> > > > > > > > >>> hbase
> > > > > > > > >>>> run
> > > > > > > > >>>>>>>>> MR
> > > > > > > > >>>>>>>>>>> jobs, only tool started manually by the user can
> do
> > > > > that".
> > > > > > or
> > > > > > > > >>> can
> > > > > > > > >>>> we
> > > > > > > > >>>>>>>>>> start
> > > > > > > > >>>>>>>>>>> adding MR calls around without problems?
> > > > > > > > >>>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSSION] MR jobs started by Master or RS
Posted by Matteo Bertozzi <th...@gmail.com>.
let me try to go back to my original topic.
this question was meant to be generic, and provide some rule for future
code.
from what I can gather, a rule that may satisfy everyone can be:
- we don't want any core feature (e.g. compaction/log-split/log-reply)
over MR, because some cluster may not want or may have an
external/uncontrolled MR setup.
- we allow non-core features (e.g. features enabled by a flag) to run MR
jobs from hbase, because unless you use the feature, MR is not required.
Matteo
On Fri, Sep 23, 2016 at 5:39 AM, Ted Yu <yu...@gmail.com> wrote:
> I suggest you look at Matteo's work for AssignmentManager which is to make
> Master more stable.
>
> Cheers
>
> On Fri, Sep 23, 2016 at 5:32 AM, 张铎 <pa...@gmail.com> wrote:
>
> > No, not your fault, at lease, not this time:)
> >
> > Why I call the code ugly? Can you simply tell me the sequence of calls
> when
> > starting up the HMaster? HMaster is also a regionserver so it extends
> > HRegionServer, and the initialization of HRegionServer sometimes needs to
> > make rpc calls to HMaster. A simple change would cause probabilistic dead
> > lock or some strange NPEs...
> >
> > That's why I'm very nervous when somebody wants to add new features or
> add
> > external dependencies to HMaster, especially add more works for the start
> > up processing...
> >
> > Thanks.
> >
> > 2016-09-23 20:02 GMT+08:00 Ted Yu <yu...@gmail.com>:
> >
> > > I read through HADOOP-13433
> > > <https://issues.apache.org/jira/browse/HADOOP-13433> - the cited race
> > > condition is in jdk.
> > >
> > > Suggest pinging the reviewer on JIRA to get it moving.
> > >
> > > bq. But the ugly code in HMaster is readlly a problem...
> > >
> > > Can you be specific as to which code is ugly ? Is it in the backup /
> > > restore mega patch ?
> > >
> > > Cheers
> > >
> > > On Thu, Sep 22, 2016 at 10:44 PM, 张铎 <pa...@gmail.com> wrote:
> > >
> > > > If you guys have already implemented the feature in the MR way and
> the
> > > > patch is ready for landing on master, I'm a -0 on it as I do not want
> > to
> > > > block the development progress.
> > > >
> > > > But I strongly suggest later we need to revisit the design and see if
> > we
> > > > can seperated the logic from HMaster as much as possible. HA is not a
> > big
> > > > problem if you do not store any metada locally. But the ugly code in
> > > > HMaster is readlly a problem...
> > > >
> > > > And for security, I have a issue pending for a long time. Can someone
> > > help
> > > > taking a simple look at it? This is what I mean, ugly code... logout
> > and
> > > > destroy the credentials in a subject when it is still being used, and
> > > > declared as LimitPrivacy so I can not change the behivor and the only
> > way
> > > > to fix it is to write another piece of ugly code...
> > > >
> > > > https://issues.apache.org/jira/browse/HADOOP-13433
> > > >
> > > > 2016-09-23 12:53 GMT+08:00 Vladimir Rodionov <vladrodionov@gmail.com
> >:
> > > >
> > > > > >> If in the future, we find better ways of doing this without
> using
> > > MR,
> > > > we
> > > > > can certainly consider that
> > > > >
> > > > > Our framework for distributed operations is abstract and allows
> > > > > different implementations. MR is just one implementation we
> provide.
> > > > >
> > > > > -Vlad
> > > > >
> > > > > On Thu, Sep 22, 2016 at 9:38 PM, Devaraj Das <ddas@hortonworks.com
> >
> > > > wrote:
> > > > >
> > > > > > Guys, first off apologies for bringing in the topic of MR-based
> > > > > > compactions.. But I was thinking more about the SpliceMachine
> > > approach
> > > > of
> > > > > > managing compactions in Spark where apparently they saw a lot of
> > > > > benefits.
> > > > > > Apologies for giving you that sore throat Andrew; I really didn't
> > > mean
> > > > to
> > > > > > :-)
> > > > > >
> > > > > > So on this issue, we have these on the plate:
> > > > > > 0. Somehow not use MR but something like that
> > > > > > 1. Run a standalone service other than master
> > > > > > 2. Shell out from the master
> > > > > >
> > > > > > I don't think we have a good answer to (0), and I don't think
> it's
> > > even
> > > > > > worth the effort of trying to build something when MR is already
> > > there,
> > > > > and
> > > > > > being used by HBase already for some operations.
> > > > > >
> > > > > > On (1), we have to deal with a myriad of issues - HA of the
> server
> > > not
> > > > > > being the least of them all. Security (kerberos authentication,
> > > another
> > > > > > keytab to manage, etc. etc. etc.). IMO, that approach is DOA.
> > Instead
> > > > > let's
> > > > > > substitute that (1) with the HBase Master. I haven't seen any
> good
> > > > reason
> > > > > > why the HBase master shouldn't launch MR jobs if needed. It's not
> > > > ideal;
> > > > > > agreed.
> > > > > >
> > > > > > Now before going to (2), let's see what are the benefits of
> running
> > > the
> > > > > > backup/restore jobs from the master. I think Ted has summarized
> > some
> > > of
> > > > > the
> > > > > > issues that we need to take care of - basically, the master can
> > keep
> > > > > track
> > > > > > of running jobs, and should it fail, the backup master can
> continue
> > > > > keeping
> > > > > > track of it (since the jobId would have been recorded in the proc
> > > WAL).
> > > > > The
> > > > > > master can also do cleanup, etc. of failed backup/restore
> > processes.
> > > > > > Security is another issue - the job needs to run as 'hbase' since
> > it
> > > > owns
> > > > > > the data. Having the master launch the job makes it get that
> > > privilege.
> > > > > In
> > > > > > the (2) approach, it's hard to do some of the above management.
> > > > > >
> > > > > > Guys, just to reiterate, the patch as such is ready from the
> > overall
> > > > > > design/arch point of view (maybe code review is still pending
> from
> > > > > Matteo).
> > > > > > If in the future, we find better ways of doing this without using
> > MR,
> > > > we
> > > > > > can certainly consider that. But IMO don't think we should block
> > this
> > > > > patch
> > > > > > from getting merged.
> > > > > >
> > > > > > ________________________________________
> > > > > > From: 张铎 <pa...@gmail.com>
> > > > > > Sent: Thursday, September 22, 2016 8:32 PM
> > > > > > To: dev@hbase.apache.org
> > > > > > Subject: Re: [DISCUSSION] MR jobs started by Master or RS
> > > > > >
> > > > > > So what about a standalone service other than master? You can use
> > > your
> > > > > own
> > > > > > procedure store in that service?
> > > > > >
> > > > > > 2016-09-23 11:28 GMT+08:00 Ted Yu <yu...@gmail.com>:
> > > > > >
> > > > > > > An earlier implementation was client driven.
> > > > > > >
> > > > > > > But with that approach, it is hard to resume if there is error
> > > > midway.
> > > > > > > Using Procedure V2 makes the backup / restore more robust.
> > > > > > >
> > > > > > > Another consideration is for security. It is hard to enforce
> > > security
> > > > > (to
> > > > > > > be implemented) for client driven actions.
> > > > > > >
> > > > > > > Cheers
> > > > > > >
> > > > > > > > On Sep 22, 2016, at 8:15 PM, Andrew Purtell <
> > > > > andrew.purtell@gmail.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > No, this misses Matteo's finer point, which is "shelling out"
> > > from
> > > > > the
> > > > > > > master directly to run MR is a first. Why not drive this with a
> > > > utility
> > > > > > > derived from Tool?
> > > > > > > >
> > > > > > > > On Sep 22, 2016, at 7:57 PM, Vladimir Rodionov <
> > > > > vladrodionov@gmail.com
> > > > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > >>>> In our production cluster, it is a common case we just
> have
> > > > HDFS
> > > > > > and
> > > > > > > >>>> HBase deployed.
> > > > > > > >>>> If our Master/RS depend on MR framework (especially some
> > > > features
> > > > > we
> > > > > > > >>>> have not used at all), it introduced another cost for
> > > maintain.
> > > > > I
> > > > > > > >>>> don't think it is a good idea.
> > > > > > > >>
> > > > > > > >> So , you are not backup users in this case. Many our
> customers
> > > > have
> > > > > > full
> > > > > > > >> stack deployed and
> > > > > > > >> want see backup to be a standard feature. Besides this,
> > nothing
> > > > will
> > > > > > > happen
> > > > > > > >> in your cluster
> > > > > > > >> if you won't be doing backups.
> > > > > > > >>
> > > > > > > >> This discussion (we do not want see M/R dependency) goes to
> > > > nowhere.
> > > > > > We
> > > > > > > >> asked already, at least twice, to suggest another framework
> > > (other
> > > > > > than
> > > > > > > M/R)
> > > > > > > >> for bulk data copy with *conversion*. Still waiting for
> > > > suggestions.
> > > > > > > >>
> > > > > > > >> -Vlad
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>> On Thu, Sep 22, 2016 at 7:49 PM, Ted Yu <
> yuzhihong@gmail.com
> > >
> > > > > wrote:
> > > > > > > >>>
> > > > > > > >>> If MR framework is not deployed in the cluster, hbase still
> > > > > functions
> > > > > > > >>> normally (post merge).
> > > > > > > >>>
> > > > > > > >>> In terms of build time dependency, we have long been
> > depending
> > > on
> > > > > > > >>> mapreduce. Take a look at ExportSnapshot.
> > > > > > > >>>
> > > > > > > >>> Cheers
> > > > > > > >>>
> > > > > > > >>> On Thu, Sep 22, 2016 at 7:42 PM, Heng Chen <
> > > > > heng.chen.1986@gmail.com
> > > > > > >
> > > > > > > >>> wrote:
> > > > > > > >>>
> > > > > > > >>>> In our production cluster, it is a common case we just
> have
> > > > HDFS
> > > > > > and
> > > > > > > >>>> HBase deployed.
> > > > > > > >>>> If our Master/RS depend on MR framework (especially some
> > > > features
> > > > > we
> > > > > > > >>>> have not used at all), it introduced another cost for
> > > maintain.
> > > > > I
> > > > > > > >>>> don't think it is a good idea.
> > > > > > > >>>>
> > > > > > > >>>> 2016-09-23 10:28 GMT+08:00 张铎 <pa...@gmail.com>:
> > > > > > > >>>>> To be specific, for example, our nice Backup/Restore
> > feature,
> > > > if
> > > > > we
> > > > > > > >>> think
> > > > > > > >>>>> this is not a core feature of HBase, then we could make
> it
> > > > depend
> > > > > > on
> > > > > > > >>> MR,
> > > > > > > >>>>> and start a standalone BackupManager instance that
> submits
> > MR
> > > > > jobs
> > > > > > to
> > > > > > > >>> do
> > > > > > > >>>>> periodical maintenance job. And if we think this is a
> core
> > > > > feature
> > > > > > > that
> > > > > > > >>>>> everyone should use it, then we'd better implement it
> > without
> > > > MR
> > > > > > > >>>>> dependency, like DLS.
> > > > > > > >>>>>
> > > > > > > >>>>> Thanks.
> > > > > > > >>>>>
> > > > > > > >>>>> 2016-09-23 10:11 GMT+08:00 张铎 <pa...@gmail.com>:
> > > > > > > >>>>>
> > > > > > > >>>>>> I‘m -1 on let master or rs launch MR jobs. It is OK that
> > > some
> > > > of
> > > > > > our
> > > > > > > >>>>>> features depend on MR but I think the bottom line is
> that
> > we
> > > > > > should
> > > > > > > >>>> launch
> > > > > > > >>>>>> the jobs from outside manually or by other services.
> > > > > > > >>>>>>
> > > > > > > >>>>>> 2016-09-23 9:47 GMT+08:00 Andrew Purtell <
> > > > > > andrew.purtell@gmail.com
> > > > > > > >:
> > > > > > > >>>>>>
> > > > > > > >>>>>>> Ok, got it. Well "shelling out" is on the line I think,
> > so
> > > a
> > > > > fair
> > > > > > > >>>>>>> question.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> Can this be driven by a utility derived from Tool like
> > our
> > > > > other
> > > > > > MR
> > > > > > > >>>> apps?
> > > > > > > >>>>>>> The issue is needing the AccessController to decide if
> > > > allowed?
> > > > > > But
> > > > > > > >>>> nothing
> > > > > > > >>>>>>> prevents the user from running the job
> > > > manually/independently,
> > > > > > > right?
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>> On Sep 22, 2016, at 3:44 PM, Matteo Bertozzi <
> > > > > > > >>>> theo.bertozzi@gmail.com>
> > > > > > > >>>>>>> wrote:
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> just a remark. my query was not about tools using MR
> > > > > (everyone i
> > > > > > > >>>> think
> > > > > > > >>>>>>> is
> > > > > > > >>>>>>>> ok with those).
> > > > > > > >>>>>>>> the topic was about: "are we ok with running MR jobs
> > from
> > > > > Master
> > > > > > > >>> and
> > > > > > > >>>> RSs
> > > > > > > >>>>>>>> code?" since this will be the first time we do this
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> Matteo
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>>> On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das <
> > > > > > > >>> ddas@hortonworks.com>
> > > > > > > >>>>>>> wrote:
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> Very much agree; for tools like ExportSnapshot /
> > Backup /
> > > > > > > Restore,
> > > > > > > >>>> it's
> > > > > > > >>>>>>>>> fine to be dependent on MR. MR is the right framework
> > for
> > > > > such.
> > > > > > > We
> > > > > > > >>>>>>> should
> > > > > > > >>>>>>>>> also do compactions using MR (just saying :) )
> > > > > > > >>>>>>>>> ________________________________________
> > > > > > > >>>>>>>>> From: Ted Yu <yu...@gmail.com>
> > > > > > > >>>>>>>>> Sent: Thursday, September 22, 2016 2:00 PM
> > > > > > > >>>>>>>>> To: dev@hbase.apache.org
> > > > > > > >>>>>>>>> Subject: Re: [DISCUSSION] MR jobs started by Master
> or
> > RS
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> I agree - backup / restore is in the same category as
> > > > import
> > > > > /
> > > > > > > >>>> export.
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>> On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell <
> > > > > > > >>>>>>> andrew.purtell@gmail.com>
> > > > > > > >>>>>>>>> wrote:
> > > > > > > >>>>>>>>>
> > > > > > > >>>>>>>>>> Backup is extra tooling around core in my opinion.
> > Like
> > > > > import
> > > > > > > or
> > > > > > > >>>>>>> export.
> > > > > > > >>>>>>>>>> Or the optional MOB tool. It's fine.
> > > > > > > >>>>>>>>>>
> > > > > > > >>>>>>>>>>> On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi <
> > > > > > > >>>> mbertozzi@apache.org>
> > > > > > > >>>>>>>>>> wrote:
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> What's the latest opinion around running MR jobs
> from
> > > > hbase
> > > > > > > >>>> (Master
> > > > > > > >>>>>>> or
> > > > > > > >>>>>>>>>> RS)?
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> I remember in the past that there was discussion
> > about
> > > > not
> > > > > > > >>> having
> > > > > > > >>>> MR
> > > > > > > >>>>>>>>> has
> > > > > > > >>>>>>>>>>> direct dependency of hbase.
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> I think some of discussion where around MOB that
> had
> > a
> > > MR
> > > > > job
> > > > > > > to
> > > > > > > >>>>>>>>> compact,
> > > > > > > >>>>>>>>>>> that later was transformed in a non-MR job to be
> > > merged,
> > > > I
> > > > > > > think
> > > > > > > >>>> we
> > > > > > > >>>>>>>>> had a
> > > > > > > >>>>>>>>>>> similar discussion for log split/replay.
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> the latest is the new Backup feature (HBASE-7912),
> > that
> > > > > runs
> > > > > > a
> > > > > > > >>> MR
> > > > > > > >>>> job
> > > > > > > >>>>>>>>>> from
> > > > > > > >>>>>>>>>>> the master to copy data or restore data.
> > > > > > > >>>>>>>>>>> (backup is also "not really core" as in.. if you
> > don't
> > > > use
> > > > > > > >>> backup
> > > > > > > >>>>>>>>> you'll
> > > > > > > >>>>>>>>>>> not end up running MR jobs, but this was probably
> > true
> > > > for
> > > > > > MOB
> > > > > > > >>> as
> > > > > > > >>>> in
> > > > > > > >>>>>>>>> "if
> > > > > > > >>>>>>>>>>> you don't enable MOB you don't need MR")
> > > > > > > >>>>>>>>>>>
> > > > > > > >>>>>>>>>>> any thoughts? do we a rule that says "we don't want
> > to
> > > > have
> > > > > > > >>> hbase
> > > > > > > >>>> run
> > > > > > > >>>>>>>>> MR
> > > > > > > >>>>>>>>>>> jobs, only tool started manually by the user can do
> > > > that".
> > > > > or
> > > > > > > >>> can
> > > > > > > >>>> we
> > > > > > > >>>>>>>>>> start
> > > > > > > >>>>>>>>>>> adding MR calls around without problems?
> > > > > > > >>>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSSION] MR jobs started by Master or RS
Posted by Ted Yu <yu...@gmail.com>.
I suggest you look at Matteo's work for AssignmentManager which is to make
Master more stable.
Cheers
On Fri, Sep 23, 2016 at 5:32 AM, 张铎 <pa...@gmail.com> wrote:
> No, not your fault, at lease, not this time:)
>
> Why I call the code ugly? Can you simply tell me the sequence of calls when
> starting up the HMaster? HMaster is also a regionserver so it extends
> HRegionServer, and the initialization of HRegionServer sometimes needs to
> make rpc calls to HMaster. A simple change would cause probabilistic dead
> lock or some strange NPEs...
>
> That's why I'm very nervous when somebody wants to add new features or add
> external dependencies to HMaster, especially add more works for the start
> up processing...
>
> Thanks.
>
> 2016-09-23 20:02 GMT+08:00 Ted Yu <yu...@gmail.com>:
>
> > I read through HADOOP-13433
> > <https://issues.apache.org/jira/browse/HADOOP-13433> - the cited race
> > condition is in jdk.
> >
> > Suggest pinging the reviewer on JIRA to get it moving.
> >
> > bq. But the ugly code in HMaster is readlly a problem...
> >
> > Can you be specific as to which code is ugly ? Is it in the backup /
> > restore mega patch ?
> >
> > Cheers
> >
> > On Thu, Sep 22, 2016 at 10:44 PM, 张铎 <pa...@gmail.com> wrote:
> >
> > > If you guys have already implemented the feature in the MR way and the
> > > patch is ready for landing on master, I'm a -0 on it as I do not want
> to
> > > block the development progress.
> > >
> > > But I strongly suggest later we need to revisit the design and see if
> we
> > > can seperated the logic from HMaster as much as possible. HA is not a
> big
> > > problem if you do not store any metada locally. But the ugly code in
> > > HMaster is readlly a problem...
> > >
> > > And for security, I have a issue pending for a long time. Can someone
> > help
> > > taking a simple look at it? This is what I mean, ugly code... logout
> and
> > > destroy the credentials in a subject when it is still being used, and
> > > declared as LimitPrivacy so I can not change the behivor and the only
> way
> > > to fix it is to write another piece of ugly code...
> > >
> > > https://issues.apache.org/jira/browse/HADOOP-13433
> > >
> > > 2016-09-23 12:53 GMT+08:00 Vladimir Rodionov <vl...@gmail.com>:
> > >
> > > > >> If in the future, we find better ways of doing this without using
> > MR,
> > > we
> > > > can certainly consider that
> > > >
> > > > Our framework for distributed operations is abstract and allows
> > > > different implementations. MR is just one implementation we provide.
> > > >
> > > > -Vlad
> > > >
> > > > On Thu, Sep 22, 2016 at 9:38 PM, Devaraj Das <dd...@hortonworks.com>
> > > wrote:
> > > >
> > > > > Guys, first off apologies for bringing in the topic of MR-based
> > > > > compactions.. But I was thinking more about the SpliceMachine
> > approach
> > > of
> > > > > managing compactions in Spark where apparently they saw a lot of
> > > > benefits.
> > > > > Apologies for giving you that sore throat Andrew; I really didn't
> > mean
> > > to
> > > > > :-)
> > > > >
> > > > > So on this issue, we have these on the plate:
> > > > > 0. Somehow not use MR but something like that
> > > > > 1. Run a standalone service other than master
> > > > > 2. Shell out from the master
> > > > >
> > > > > I don't think we have a good answer to (0), and I don't think it's
> > even
> > > > > worth the effort of trying to build something when MR is already
> > there,
> > > > and
> > > > > being used by HBase already for some operations.
> > > > >
> > > > > On (1), we have to deal with a myriad of issues - HA of the server
> > not
> > > > > being the least of them all. Security (kerberos authentication,
> > another
> > > > > keytab to manage, etc. etc. etc.). IMO, that approach is DOA.
> Instead
> > > > let's
> > > > > substitute that (1) with the HBase Master. I haven't seen any good
> > > reason
> > > > > why the HBase master shouldn't launch MR jobs if needed. It's not
> > > ideal;
> > > > > agreed.
> > > > >
> > > > > Now before going to (2), let's see what are the benefits of running
> > the
> > > > > backup/restore jobs from the master. I think Ted has summarized
> some
> > of
> > > > the
> > > > > issues that we need to take care of - basically, the master can
> keep
> > > > track
> > > > > of running jobs, and should it fail, the backup master can continue
> > > > keeping
> > > > > track of it (since the jobId would have been recorded in the proc
> > WAL).
> > > > The
> > > > > master can also do cleanup, etc. of failed backup/restore
> processes.
> > > > > Security is another issue - the job needs to run as 'hbase' since
> it
> > > owns
> > > > > the data. Having the master launch the job makes it get that
> > privilege.
> > > > In
> > > > > the (2) approach, it's hard to do some of the above management.
> > > > >
> > > > > Guys, just to reiterate, the patch as such is ready from the
> overall
> > > > > design/arch point of view (maybe code review is still pending from
> > > > Matteo).
> > > > > If in the future, we find better ways of doing this without using
> MR,
> > > we
> > > > > can certainly consider that. But IMO don't think we should block
> this
> > > > patch
> > > > > from getting merged.
> > > > >
> > > > > ________________________________________
> > > > > From: 张铎 <pa...@gmail.com>
> > > > > Sent: Thursday, September 22, 2016 8:32 PM
> > > > > To: dev@hbase.apache.org
> > > > > Subject: Re: [DISCUSSION] MR jobs started by Master or RS
> > > > >
> > > > > So what about a standalone service other than master? You can use
> > your
> > > > own
> > > > > procedure store in that service?
> > > > >
> > > > > 2016-09-23 11:28 GMT+08:00 Ted Yu <yu...@gmail.com>:
> > > > >
> > > > > > An earlier implementation was client driven.
> > > > > >
> > > > > > But with that approach, it is hard to resume if there is error
> > > midway.
> > > > > > Using Procedure V2 makes the backup / restore more robust.
> > > > > >
> > > > > > Another consideration is for security. It is hard to enforce
> > security
> > > > (to
> > > > > > be implemented) for client driven actions.
> > > > > >
> > > > > > Cheers
> > > > > >
> > > > > > > On Sep 22, 2016, at 8:15 PM, Andrew Purtell <
> > > > andrew.purtell@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > No, this misses Matteo's finer point, which is "shelling out"
> > from
> > > > the
> > > > > > master directly to run MR is a first. Why not drive this with a
> > > utility
> > > > > > derived from Tool?
> > > > > > >
> > > > > > > On Sep 22, 2016, at 7:57 PM, Vladimir Rodionov <
> > > > vladrodionov@gmail.com
> > > > > >
> > > > > > wrote:
> > > > > > >
> > > > > > >>>> In our production cluster, it is a common case we just have
> > > HDFS
> > > > > and
> > > > > > >>>> HBase deployed.
> > > > > > >>>> If our Master/RS depend on MR framework (especially some
> > > features
> > > > we
> > > > > > >>>> have not used at all), it introduced another cost for
> > maintain.
> > > > I
> > > > > > >>>> don't think it is a good idea.
> > > > > > >>
> > > > > > >> So , you are not backup users in this case. Many our customers
> > > have
> > > > > full
> > > > > > >> stack deployed and
> > > > > > >> want see backup to be a standard feature. Besides this,
> nothing
> > > will
> > > > > > happen
> > > > > > >> in your cluster
> > > > > > >> if you won't be doing backups.
> > > > > > >>
> > > > > > >> This discussion (we do not want see M/R dependency) goes to
> > > nowhere.
> > > > > We
> > > > > > >> asked already, at least twice, to suggest another framework
> > (other
> > > > > than
> > > > > > M/R)
> > > > > > >> for bulk data copy with *conversion*. Still waiting for
> > > suggestions.
> > > > > > >>
> > > > > > >> -Vlad
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>> On Thu, Sep 22, 2016 at 7:49 PM, Ted Yu <yuzhihong@gmail.com
> >
> > > > wrote:
> > > > > > >>>
> > > > > > >>> If MR framework is not deployed in the cluster, hbase still
> > > > functions
> > > > > > >>> normally (post merge).
> > > > > > >>>
> > > > > > >>> In terms of build time dependency, we have long been
> depending
> > on
> > > > > > >>> mapreduce. Take a look at ExportSnapshot.
> > > > > > >>>
> > > > > > >>> Cheers
> > > > > > >>>
> > > > > > >>> On Thu, Sep 22, 2016 at 7:42 PM, Heng Chen <
> > > > heng.chen.1986@gmail.com
> > > > > >
> > > > > > >>> wrote:
> > > > > > >>>
> > > > > > >>>> In our production cluster, it is a common case we just have
> > > HDFS
> > > > > and
> > > > > > >>>> HBase deployed.
> > > > > > >>>> If our Master/RS depend on MR framework (especially some
> > > features
> > > > we
> > > > > > >>>> have not used at all), it introduced another cost for
> > maintain.
> > > > I
> > > > > > >>>> don't think it is a good idea.
> > > > > > >>>>
> > > > > > >>>> 2016-09-23 10:28 GMT+08:00 张铎 <pa...@gmail.com>:
> > > > > > >>>>> To be specific, for example, our nice Backup/Restore
> feature,
> > > if
> > > > we
> > > > > > >>> think
> > > > > > >>>>> this is not a core feature of HBase, then we could make it
> > > depend
> > > > > on
> > > > > > >>> MR,
> > > > > > >>>>> and start a standalone BackupManager instance that submits
> MR
> > > > jobs
> > > > > to
> > > > > > >>> do
> > > > > > >>>>> periodical maintenance job. And if we think this is a core
> > > > feature
> > > > > > that
> > > > > > >>>>> everyone should use it, then we'd better implement it
> without
> > > MR
> > > > > > >>>>> dependency, like DLS.
> > > > > > >>>>>
> > > > > > >>>>> Thanks.
> > > > > > >>>>>
> > > > > > >>>>> 2016-09-23 10:11 GMT+08:00 张铎 <pa...@gmail.com>:
> > > > > > >>>>>
> > > > > > >>>>>> I‘m -1 on let master or rs launch MR jobs. It is OK that
> > some
> > > of
> > > > > our
> > > > > > >>>>>> features depend on MR but I think the bottom line is that
> we
> > > > > should
> > > > > > >>>> launch
> > > > > > >>>>>> the jobs from outside manually or by other services.
> > > > > > >>>>>>
> > > > > > >>>>>> 2016-09-23 9:47 GMT+08:00 Andrew Purtell <
> > > > > andrew.purtell@gmail.com
> > > > > > >:
> > > > > > >>>>>>
> > > > > > >>>>>>> Ok, got it. Well "shelling out" is on the line I think,
> so
> > a
> > > > fair
> > > > > > >>>>>>> question.
> > > > > > >>>>>>>
> > > > > > >>>>>>> Can this be driven by a utility derived from Tool like
> our
> > > > other
> > > > > MR
> > > > > > >>>> apps?
> > > > > > >>>>>>> The issue is needing the AccessController to decide if
> > > allowed?
> > > > > But
> > > > > > >>>> nothing
> > > > > > >>>>>>> prevents the user from running the job
> > > manually/independently,
> > > > > > right?
> > > > > > >>>>>>>
> > > > > > >>>>>>>> On Sep 22, 2016, at 3:44 PM, Matteo Bertozzi <
> > > > > > >>>> theo.bertozzi@gmail.com>
> > > > > > >>>>>>> wrote:
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> just a remark. my query was not about tools using MR
> > > > (everyone i
> > > > > > >>>> think
> > > > > > >>>>>>> is
> > > > > > >>>>>>>> ok with those).
> > > > > > >>>>>>>> the topic was about: "are we ok with running MR jobs
> from
> > > > Master
> > > > > > >>> and
> > > > > > >>>> RSs
> > > > > > >>>>>>>> code?" since this will be the first time we do this
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Matteo
> > > > > > >>>>>>>>
> > > > > > >>>>>>>>
> > > > > > >>>>>>>>> On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das <
> > > > > > >>> ddas@hortonworks.com>
> > > > > > >>>>>>> wrote:
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> Very much agree; for tools like ExportSnapshot /
> Backup /
> > > > > > Restore,
> > > > > > >>>> it's
> > > > > > >>>>>>>>> fine to be dependent on MR. MR is the right framework
> for
> > > > such.
> > > > > > We
> > > > > > >>>>>>> should
> > > > > > >>>>>>>>> also do compactions using MR (just saying :) )
> > > > > > >>>>>>>>> ________________________________________
> > > > > > >>>>>>>>> From: Ted Yu <yu...@gmail.com>
> > > > > > >>>>>>>>> Sent: Thursday, September 22, 2016 2:00 PM
> > > > > > >>>>>>>>> To: dev@hbase.apache.org
> > > > > > >>>>>>>>> Subject: Re: [DISCUSSION] MR jobs started by Master or
> RS
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> I agree - backup / restore is in the same category as
> > > import
> > > > /
> > > > > > >>>> export.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell <
> > > > > > >>>>>>> andrew.purtell@gmail.com>
> > > > > > >>>>>>>>> wrote:
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>>> Backup is extra tooling around core in my opinion.
> Like
> > > > import
> > > > > > or
> > > > > > >>>>>>> export.
> > > > > > >>>>>>>>>> Or the optional MOB tool. It's fine.
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>> On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi <
> > > > > > >>>> mbertozzi@apache.org>
> > > > > > >>>>>>>>>> wrote:
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> What's the latest opinion around running MR jobs from
> > > hbase
> > > > > > >>>> (Master
> > > > > > >>>>>>> or
> > > > > > >>>>>>>>>> RS)?
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> I remember in the past that there was discussion
> about
> > > not
> > > > > > >>> having
> > > > > > >>>> MR
> > > > > > >>>>>>>>> has
> > > > > > >>>>>>>>>>> direct dependency of hbase.
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> I think some of discussion where around MOB that had
> a
> > MR
> > > > job
> > > > > > to
> > > > > > >>>>>>>>> compact,
> > > > > > >>>>>>>>>>> that later was transformed in a non-MR job to be
> > merged,
> > > I
> > > > > > think
> > > > > > >>>> we
> > > > > > >>>>>>>>> had a
> > > > > > >>>>>>>>>>> similar discussion for log split/replay.
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> the latest is the new Backup feature (HBASE-7912),
> that
> > > > runs
> > > > > a
> > > > > > >>> MR
> > > > > > >>>> job
> > > > > > >>>>>>>>>> from
> > > > > > >>>>>>>>>>> the master to copy data or restore data.
> > > > > > >>>>>>>>>>> (backup is also "not really core" as in.. if you
> don't
> > > use
> > > > > > >>> backup
> > > > > > >>>>>>>>> you'll
> > > > > > >>>>>>>>>>> not end up running MR jobs, but this was probably
> true
> > > for
> > > > > MOB
> > > > > > >>> as
> > > > > > >>>> in
> > > > > > >>>>>>>>> "if
> > > > > > >>>>>>>>>>> you don't enable MOB you don't need MR")
> > > > > > >>>>>>>>>>>
> > > > > > >>>>>>>>>>> any thoughts? do we a rule that says "we don't want
> to
> > > have
> > > > > > >>> hbase
> > > > > > >>>> run
> > > > > > >>>>>>>>> MR
> > > > > > >>>>>>>>>>> jobs, only tool started manually by the user can do
> > > that".
> > > > or
> > > > > > >>> can
> > > > > > >>>> we
> > > > > > >>>>>>>>>> start
> > > > > > >>>>>>>>>>> adding MR calls around without problems?
> > > > > > >>>
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSSION] MR jobs started by Master or RS
Posted by 张铎 <pa...@gmail.com>.
No, not your fault, at lease, not this time:)
Why I call the code ugly? Can you simply tell me the sequence of calls when
starting up the HMaster? HMaster is also a regionserver so it extends
HRegionServer, and the initialization of HRegionServer sometimes needs to
make rpc calls to HMaster. A simple change would cause probabilistic dead
lock or some strange NPEs...
That's why I'm very nervous when somebody wants to add new features or add
external dependencies to HMaster, especially add more works for the start
up processing...
Thanks.
2016-09-23 20:02 GMT+08:00 Ted Yu <yu...@gmail.com>:
> I read through HADOOP-13433
> <https://issues.apache.org/jira/browse/HADOOP-13433> - the cited race
> condition is in jdk.
>
> Suggest pinging the reviewer on JIRA to get it moving.
>
> bq. But the ugly code in HMaster is readlly a problem...
>
> Can you be specific as to which code is ugly ? Is it in the backup /
> restore mega patch ?
>
> Cheers
>
> On Thu, Sep 22, 2016 at 10:44 PM, 张铎 <pa...@gmail.com> wrote:
>
> > If you guys have already implemented the feature in the MR way and the
> > patch is ready for landing on master, I'm a -0 on it as I do not want to
> > block the development progress.
> >
> > But I strongly suggest later we need to revisit the design and see if we
> > can seperated the logic from HMaster as much as possible. HA is not a big
> > problem if you do not store any metada locally. But the ugly code in
> > HMaster is readlly a problem...
> >
> > And for security, I have a issue pending for a long time. Can someone
> help
> > taking a simple look at it? This is what I mean, ugly code... logout and
> > destroy the credentials in a subject when it is still being used, and
> > declared as LimitPrivacy so I can not change the behivor and the only way
> > to fix it is to write another piece of ugly code...
> >
> > https://issues.apache.org/jira/browse/HADOOP-13433
> >
> > 2016-09-23 12:53 GMT+08:00 Vladimir Rodionov <vl...@gmail.com>:
> >
> > > >> If in the future, we find better ways of doing this without using
> MR,
> > we
> > > can certainly consider that
> > >
> > > Our framework for distributed operations is abstract and allows
> > > different implementations. MR is just one implementation we provide.
> > >
> > > -Vlad
> > >
> > > On Thu, Sep 22, 2016 at 9:38 PM, Devaraj Das <dd...@hortonworks.com>
> > wrote:
> > >
> > > > Guys, first off apologies for bringing in the topic of MR-based
> > > > compactions.. But I was thinking more about the SpliceMachine
> approach
> > of
> > > > managing compactions in Spark where apparently they saw a lot of
> > > benefits.
> > > > Apologies for giving you that sore throat Andrew; I really didn't
> mean
> > to
> > > > :-)
> > > >
> > > > So on this issue, we have these on the plate:
> > > > 0. Somehow not use MR but something like that
> > > > 1. Run a standalone service other than master
> > > > 2. Shell out from the master
> > > >
> > > > I don't think we have a good answer to (0), and I don't think it's
> even
> > > > worth the effort of trying to build something when MR is already
> there,
> > > and
> > > > being used by HBase already for some operations.
> > > >
> > > > On (1), we have to deal with a myriad of issues - HA of the server
> not
> > > > being the least of them all. Security (kerberos authentication,
> another
> > > > keytab to manage, etc. etc. etc.). IMO, that approach is DOA. Instead
> > > let's
> > > > substitute that (1) with the HBase Master. I haven't seen any good
> > reason
> > > > why the HBase master shouldn't launch MR jobs if needed. It's not
> > ideal;
> > > > agreed.
> > > >
> > > > Now before going to (2), let's see what are the benefits of running
> the
> > > > backup/restore jobs from the master. I think Ted has summarized some
> of
> > > the
> > > > issues that we need to take care of - basically, the master can keep
> > > track
> > > > of running jobs, and should it fail, the backup master can continue
> > > keeping
> > > > track of it (since the jobId would have been recorded in the proc
> WAL).
> > > The
> > > > master can also do cleanup, etc. of failed backup/restore processes.
> > > > Security is another issue - the job needs to run as 'hbase' since it
> > owns
> > > > the data. Having the master launch the job makes it get that
> privilege.
> > > In
> > > > the (2) approach, it's hard to do some of the above management.
> > > >
> > > > Guys, just to reiterate, the patch as such is ready from the overall
> > > > design/arch point of view (maybe code review is still pending from
> > > Matteo).
> > > > If in the future, we find better ways of doing this without using MR,
> > we
> > > > can certainly consider that. But IMO don't think we should block this
> > > patch
> > > > from getting merged.
> > > >
> > > > ________________________________________
> > > > From: 张铎 <pa...@gmail.com>
> > > > Sent: Thursday, September 22, 2016 8:32 PM
> > > > To: dev@hbase.apache.org
> > > > Subject: Re: [DISCUSSION] MR jobs started by Master or RS
> > > >
> > > > So what about a standalone service other than master? You can use
> your
> > > own
> > > > procedure store in that service?
> > > >
> > > > 2016-09-23 11:28 GMT+08:00 Ted Yu <yu...@gmail.com>:
> > > >
> > > > > An earlier implementation was client driven.
> > > > >
> > > > > But with that approach, it is hard to resume if there is error
> > midway.
> > > > > Using Procedure V2 makes the backup / restore more robust.
> > > > >
> > > > > Another consideration is for security. It is hard to enforce
> security
> > > (to
> > > > > be implemented) for client driven actions.
> > > > >
> > > > > Cheers
> > > > >
> > > > > > On Sep 22, 2016, at 8:15 PM, Andrew Purtell <
> > > andrew.purtell@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > No, this misses Matteo's finer point, which is "shelling out"
> from
> > > the
> > > > > master directly to run MR is a first. Why not drive this with a
> > utility
> > > > > derived from Tool?
> > > > > >
> > > > > > On Sep 22, 2016, at 7:57 PM, Vladimir Rodionov <
> > > vladrodionov@gmail.com
> > > > >
> > > > > wrote:
> > > > > >
> > > > > >>>> In our production cluster, it is a common case we just have
> > HDFS
> > > > and
> > > > > >>>> HBase deployed.
> > > > > >>>> If our Master/RS depend on MR framework (especially some
> > features
> > > we
> > > > > >>>> have not used at all), it introduced another cost for
> maintain.
> > > I
> > > > > >>>> don't think it is a good idea.
> > > > > >>
> > > > > >> So , you are not backup users in this case. Many our customers
> > have
> > > > full
> > > > > >> stack deployed and
> > > > > >> want see backup to be a standard feature. Besides this, nothing
> > will
> > > > > happen
> > > > > >> in your cluster
> > > > > >> if you won't be doing backups.
> > > > > >>
> > > > > >> This discussion (we do not want see M/R dependency) goes to
> > nowhere.
> > > > We
> > > > > >> asked already, at least twice, to suggest another framework
> (other
> > > > than
> > > > > M/R)
> > > > > >> for bulk data copy with *conversion*. Still waiting for
> > suggestions.
> > > > > >>
> > > > > >> -Vlad
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>> On Thu, Sep 22, 2016 at 7:49 PM, Ted Yu <yu...@gmail.com>
> > > wrote:
> > > > > >>>
> > > > > >>> If MR framework is not deployed in the cluster, hbase still
> > > functions
> > > > > >>> normally (post merge).
> > > > > >>>
> > > > > >>> In terms of build time dependency, we have long been depending
> on
> > > > > >>> mapreduce. Take a look at ExportSnapshot.
> > > > > >>>
> > > > > >>> Cheers
> > > > > >>>
> > > > > >>> On Thu, Sep 22, 2016 at 7:42 PM, Heng Chen <
> > > heng.chen.1986@gmail.com
> > > > >
> > > > > >>> wrote:
> > > > > >>>
> > > > > >>>> In our production cluster, it is a common case we just have
> > HDFS
> > > > and
> > > > > >>>> HBase deployed.
> > > > > >>>> If our Master/RS depend on MR framework (especially some
> > features
> > > we
> > > > > >>>> have not used at all), it introduced another cost for
> maintain.
> > > I
> > > > > >>>> don't think it is a good idea.
> > > > > >>>>
> > > > > >>>> 2016-09-23 10:28 GMT+08:00 张铎 <pa...@gmail.com>:
> > > > > >>>>> To be specific, for example, our nice Backup/Restore feature,
> > if
> > > we
> > > > > >>> think
> > > > > >>>>> this is not a core feature of HBase, then we could make it
> > depend
> > > > on
> > > > > >>> MR,
> > > > > >>>>> and start a standalone BackupManager instance that submits MR
> > > jobs
> > > > to
> > > > > >>> do
> > > > > >>>>> periodical maintenance job. And if we think this is a core
> > > feature
> > > > > that
> > > > > >>>>> everyone should use it, then we'd better implement it without
> > MR
> > > > > >>>>> dependency, like DLS.
> > > > > >>>>>
> > > > > >>>>> Thanks.
> > > > > >>>>>
> > > > > >>>>> 2016-09-23 10:11 GMT+08:00 张铎 <pa...@gmail.com>:
> > > > > >>>>>
> > > > > >>>>>> I‘m -1 on let master or rs launch MR jobs. It is OK that
> some
> > of
> > > > our
> > > > > >>>>>> features depend on MR but I think the bottom line is that we
> > > > should
> > > > > >>>> launch
> > > > > >>>>>> the jobs from outside manually or by other services.
> > > > > >>>>>>
> > > > > >>>>>> 2016-09-23 9:47 GMT+08:00 Andrew Purtell <
> > > > andrew.purtell@gmail.com
> > > > > >:
> > > > > >>>>>>
> > > > > >>>>>>> Ok, got it. Well "shelling out" is on the line I think, so
> a
> > > fair
> > > > > >>>>>>> question.
> > > > > >>>>>>>
> > > > > >>>>>>> Can this be driven by a utility derived from Tool like our
> > > other
> > > > MR
> > > > > >>>> apps?
> > > > > >>>>>>> The issue is needing the AccessController to decide if
> > allowed?
> > > > But
> > > > > >>>> nothing
> > > > > >>>>>>> prevents the user from running the job
> > manually/independently,
> > > > > right?
> > > > > >>>>>>>
> > > > > >>>>>>>> On Sep 22, 2016, at 3:44 PM, Matteo Bertozzi <
> > > > > >>>> theo.bertozzi@gmail.com>
> > > > > >>>>>>> wrote:
> > > > > >>>>>>>>
> > > > > >>>>>>>> just a remark. my query was not about tools using MR
> > > (everyone i
> > > > > >>>> think
> > > > > >>>>>>> is
> > > > > >>>>>>>> ok with those).
> > > > > >>>>>>>> the topic was about: "are we ok with running MR jobs from
> > > Master
> > > > > >>> and
> > > > > >>>> RSs
> > > > > >>>>>>>> code?" since this will be the first time we do this
> > > > > >>>>>>>>
> > > > > >>>>>>>> Matteo
> > > > > >>>>>>>>
> > > > > >>>>>>>>
> > > > > >>>>>>>>> On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das <
> > > > > >>> ddas@hortonworks.com>
> > > > > >>>>>>> wrote:
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Very much agree; for tools like ExportSnapshot / Backup /
> > > > > Restore,
> > > > > >>>> it's
> > > > > >>>>>>>>> fine to be dependent on MR. MR is the right framework for
> > > such.
> > > > > We
> > > > > >>>>>>> should
> > > > > >>>>>>>>> also do compactions using MR (just saying :) )
> > > > > >>>>>>>>> ________________________________________
> > > > > >>>>>>>>> From: Ted Yu <yu...@gmail.com>
> > > > > >>>>>>>>> Sent: Thursday, September 22, 2016 2:00 PM
> > > > > >>>>>>>>> To: dev@hbase.apache.org
> > > > > >>>>>>>>> Subject: Re: [DISCUSSION] MR jobs started by Master or RS
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> I agree - backup / restore is in the same category as
> > import
> > > /
> > > > > >>>> export.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell <
> > > > > >>>>>>> andrew.purtell@gmail.com>
> > > > > >>>>>>>>> wrote:
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>> Backup is extra tooling around core in my opinion. Like
> > > import
> > > > > or
> > > > > >>>>>>> export.
> > > > > >>>>>>>>>> Or the optional MOB tool. It's fine.
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>> On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi <
> > > > > >>>> mbertozzi@apache.org>
> > > > > >>>>>>>>>> wrote:
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> What's the latest opinion around running MR jobs from
> > hbase
> > > > > >>>> (Master
> > > > > >>>>>>> or
> > > > > >>>>>>>>>> RS)?
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> I remember in the past that there was discussion about
> > not
> > > > > >>> having
> > > > > >>>> MR
> > > > > >>>>>>>>> has
> > > > > >>>>>>>>>>> direct dependency of hbase.
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> I think some of discussion where around MOB that had a
> MR
> > > job
> > > > > to
> > > > > >>>>>>>>> compact,
> > > > > >>>>>>>>>>> that later was transformed in a non-MR job to be
> merged,
> > I
> > > > > think
> > > > > >>>> we
> > > > > >>>>>>>>> had a
> > > > > >>>>>>>>>>> similar discussion for log split/replay.
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> the latest is the new Backup feature (HBASE-7912), that
> > > runs
> > > > a
> > > > > >>> MR
> > > > > >>>> job
> > > > > >>>>>>>>>> from
> > > > > >>>>>>>>>>> the master to copy data or restore data.
> > > > > >>>>>>>>>>> (backup is also "not really core" as in.. if you don't
> > use
> > > > > >>> backup
> > > > > >>>>>>>>> you'll
> > > > > >>>>>>>>>>> not end up running MR jobs, but this was probably true
> > for
> > > > MOB
> > > > > >>> as
> > > > > >>>> in
> > > > > >>>>>>>>> "if
> > > > > >>>>>>>>>>> you don't enable MOB you don't need MR")
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> any thoughts? do we a rule that says "we don't want to
> > have
> > > > > >>> hbase
> > > > > >>>> run
> > > > > >>>>>>>>> MR
> > > > > >>>>>>>>>>> jobs, only tool started manually by the user can do
> > that".
> > > or
> > > > > >>> can
> > > > > >>>> we
> > > > > >>>>>>>>>> start
> > > > > >>>>>>>>>>> adding MR calls around without problems?
> > > > > >>>
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSSION] MR jobs started by Master or RS
Posted by Ted Yu <yu...@gmail.com>.
I read through HADOOP-13433
<https://issues.apache.org/jira/browse/HADOOP-13433> - the cited race
condition is in jdk.
Suggest pinging the reviewer on JIRA to get it moving.
bq. But the ugly code in HMaster is readlly a problem...
Can you be specific as to which code is ugly ? Is it in the backup /
restore mega patch ?
Cheers
On Thu, Sep 22, 2016 at 10:44 PM, 张铎 <pa...@gmail.com> wrote:
> If you guys have already implemented the feature in the MR way and the
> patch is ready for landing on master, I'm a -0 on it as I do not want to
> block the development progress.
>
> But I strongly suggest later we need to revisit the design and see if we
> can seperated the logic from HMaster as much as possible. HA is not a big
> problem if you do not store any metada locally. But the ugly code in
> HMaster is readlly a problem...
>
> And for security, I have a issue pending for a long time. Can someone help
> taking a simple look at it? This is what I mean, ugly code... logout and
> destroy the credentials in a subject when it is still being used, and
> declared as LimitPrivacy so I can not change the behivor and the only way
> to fix it is to write another piece of ugly code...
>
> https://issues.apache.org/jira/browse/HADOOP-13433
>
> 2016-09-23 12:53 GMT+08:00 Vladimir Rodionov <vl...@gmail.com>:
>
> > >> If in the future, we find better ways of doing this without using MR,
> we
> > can certainly consider that
> >
> > Our framework for distributed operations is abstract and allows
> > different implementations. MR is just one implementation we provide.
> >
> > -Vlad
> >
> > On Thu, Sep 22, 2016 at 9:38 PM, Devaraj Das <dd...@hortonworks.com>
> wrote:
> >
> > > Guys, first off apologies for bringing in the topic of MR-based
> > > compactions.. But I was thinking more about the SpliceMachine approach
> of
> > > managing compactions in Spark where apparently they saw a lot of
> > benefits.
> > > Apologies for giving you that sore throat Andrew; I really didn't mean
> to
> > > :-)
> > >
> > > So on this issue, we have these on the plate:
> > > 0. Somehow not use MR but something like that
> > > 1. Run a standalone service other than master
> > > 2. Shell out from the master
> > >
> > > I don't think we have a good answer to (0), and I don't think it's even
> > > worth the effort of trying to build something when MR is already there,
> > and
> > > being used by HBase already for some operations.
> > >
> > > On (1), we have to deal with a myriad of issues - HA of the server not
> > > being the least of them all. Security (kerberos authentication, another
> > > keytab to manage, etc. etc. etc.). IMO, that approach is DOA. Instead
> > let's
> > > substitute that (1) with the HBase Master. I haven't seen any good
> reason
> > > why the HBase master shouldn't launch MR jobs if needed. It's not
> ideal;
> > > agreed.
> > >
> > > Now before going to (2), let's see what are the benefits of running the
> > > backup/restore jobs from the master. I think Ted has summarized some of
> > the
> > > issues that we need to take care of - basically, the master can keep
> > track
> > > of running jobs, and should it fail, the backup master can continue
> > keeping
> > > track of it (since the jobId would have been recorded in the proc WAL).
> > The
> > > master can also do cleanup, etc. of failed backup/restore processes.
> > > Security is another issue - the job needs to run as 'hbase' since it
> owns
> > > the data. Having the master launch the job makes it get that privilege.
> > In
> > > the (2) approach, it's hard to do some of the above management.
> > >
> > > Guys, just to reiterate, the patch as such is ready from the overall
> > > design/arch point of view (maybe code review is still pending from
> > Matteo).
> > > If in the future, we find better ways of doing this without using MR,
> we
> > > can certainly consider that. But IMO don't think we should block this
> > patch
> > > from getting merged.
> > >
> > > ________________________________________
> > > From: 张铎 <pa...@gmail.com>
> > > Sent: Thursday, September 22, 2016 8:32 PM
> > > To: dev@hbase.apache.org
> > > Subject: Re: [DISCUSSION] MR jobs started by Master or RS
> > >
> > > So what about a standalone service other than master? You can use your
> > own
> > > procedure store in that service?
> > >
> > > 2016-09-23 11:28 GMT+08:00 Ted Yu <yu...@gmail.com>:
> > >
> > > > An earlier implementation was client driven.
> > > >
> > > > But with that approach, it is hard to resume if there is error
> midway.
> > > > Using Procedure V2 makes the backup / restore more robust.
> > > >
> > > > Another consideration is for security. It is hard to enforce security
> > (to
> > > > be implemented) for client driven actions.
> > > >
> > > > Cheers
> > > >
> > > > > On Sep 22, 2016, at 8:15 PM, Andrew Purtell <
> > andrew.purtell@gmail.com>
> > > > wrote:
> > > > >
> > > > > No, this misses Matteo's finer point, which is "shelling out" from
> > the
> > > > master directly to run MR is a first. Why not drive this with a
> utility
> > > > derived from Tool?
> > > > >
> > > > > On Sep 22, 2016, at 7:57 PM, Vladimir Rodionov <
> > vladrodionov@gmail.com
> > > >
> > > > wrote:
> > > > >
> > > > >>>> In our production cluster, it is a common case we just have
> HDFS
> > > and
> > > > >>>> HBase deployed.
> > > > >>>> If our Master/RS depend on MR framework (especially some
> features
> > we
> > > > >>>> have not used at all), it introduced another cost for maintain.
> > I
> > > > >>>> don't think it is a good idea.
> > > > >>
> > > > >> So , you are not backup users in this case. Many our customers
> have
> > > full
> > > > >> stack deployed and
> > > > >> want see backup to be a standard feature. Besides this, nothing
> will
> > > > happen
> > > > >> in your cluster
> > > > >> if you won't be doing backups.
> > > > >>
> > > > >> This discussion (we do not want see M/R dependency) goes to
> nowhere.
> > > We
> > > > >> asked already, at least twice, to suggest another framework (other
> > > than
> > > > M/R)
> > > > >> for bulk data copy with *conversion*. Still waiting for
> suggestions.
> > > > >>
> > > > >> -Vlad
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>> On Thu, Sep 22, 2016 at 7:49 PM, Ted Yu <yu...@gmail.com>
> > wrote:
> > > > >>>
> > > > >>> If MR framework is not deployed in the cluster, hbase still
> > functions
> > > > >>> normally (post merge).
> > > > >>>
> > > > >>> In terms of build time dependency, we have long been depending on
> > > > >>> mapreduce. Take a look at ExportSnapshot.
> > > > >>>
> > > > >>> Cheers
> > > > >>>
> > > > >>> On Thu, Sep 22, 2016 at 7:42 PM, Heng Chen <
> > heng.chen.1986@gmail.com
> > > >
> > > > >>> wrote:
> > > > >>>
> > > > >>>> In our production cluster, it is a common case we just have
> HDFS
> > > and
> > > > >>>> HBase deployed.
> > > > >>>> If our Master/RS depend on MR framework (especially some
> features
> > we
> > > > >>>> have not used at all), it introduced another cost for maintain.
> > I
> > > > >>>> don't think it is a good idea.
> > > > >>>>
> > > > >>>> 2016-09-23 10:28 GMT+08:00 张铎 <pa...@gmail.com>:
> > > > >>>>> To be specific, for example, our nice Backup/Restore feature,
> if
> > we
> > > > >>> think
> > > > >>>>> this is not a core feature of HBase, then we could make it
> depend
> > > on
> > > > >>> MR,
> > > > >>>>> and start a standalone BackupManager instance that submits MR
> > jobs
> > > to
> > > > >>> do
> > > > >>>>> periodical maintenance job. And if we think this is a core
> > feature
> > > > that
> > > > >>>>> everyone should use it, then we'd better implement it without
> MR
> > > > >>>>> dependency, like DLS.
> > > > >>>>>
> > > > >>>>> Thanks.
> > > > >>>>>
> > > > >>>>> 2016-09-23 10:11 GMT+08:00 张铎 <pa...@gmail.com>:
> > > > >>>>>
> > > > >>>>>> I‘m -1 on let master or rs launch MR jobs. It is OK that some
> of
> > > our
> > > > >>>>>> features depend on MR but I think the bottom line is that we
> > > should
> > > > >>>> launch
> > > > >>>>>> the jobs from outside manually or by other services.
> > > > >>>>>>
> > > > >>>>>> 2016-09-23 9:47 GMT+08:00 Andrew Purtell <
> > > andrew.purtell@gmail.com
> > > > >:
> > > > >>>>>>
> > > > >>>>>>> Ok, got it. Well "shelling out" is on the line I think, so a
> > fair
> > > > >>>>>>> question.
> > > > >>>>>>>
> > > > >>>>>>> Can this be driven by a utility derived from Tool like our
> > other
> > > MR
> > > > >>>> apps?
> > > > >>>>>>> The issue is needing the AccessController to decide if
> allowed?
> > > But
> > > > >>>> nothing
> > > > >>>>>>> prevents the user from running the job
> manually/independently,
> > > > right?
> > > > >>>>>>>
> > > > >>>>>>>> On Sep 22, 2016, at 3:44 PM, Matteo Bertozzi <
> > > > >>>> theo.bertozzi@gmail.com>
> > > > >>>>>>> wrote:
> > > > >>>>>>>>
> > > > >>>>>>>> just a remark. my query was not about tools using MR
> > (everyone i
> > > > >>>> think
> > > > >>>>>>> is
> > > > >>>>>>>> ok with those).
> > > > >>>>>>>> the topic was about: "are we ok with running MR jobs from
> > Master
> > > > >>> and
> > > > >>>> RSs
> > > > >>>>>>>> code?" since this will be the first time we do this
> > > > >>>>>>>>
> > > > >>>>>>>> Matteo
> > > > >>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>>> On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das <
> > > > >>> ddas@hortonworks.com>
> > > > >>>>>>> wrote:
> > > > >>>>>>>>>
> > > > >>>>>>>>> Very much agree; for tools like ExportSnapshot / Backup /
> > > > Restore,
> > > > >>>> it's
> > > > >>>>>>>>> fine to be dependent on MR. MR is the right framework for
> > such.
> > > > We
> > > > >>>>>>> should
> > > > >>>>>>>>> also do compactions using MR (just saying :) )
> > > > >>>>>>>>> ________________________________________
> > > > >>>>>>>>> From: Ted Yu <yu...@gmail.com>
> > > > >>>>>>>>> Sent: Thursday, September 22, 2016 2:00 PM
> > > > >>>>>>>>> To: dev@hbase.apache.org
> > > > >>>>>>>>> Subject: Re: [DISCUSSION] MR jobs started by Master or RS
> > > > >>>>>>>>>
> > > > >>>>>>>>> I agree - backup / restore is in the same category as
> import
> > /
> > > > >>>> export.
> > > > >>>>>>>>>
> > > > >>>>>>>>> On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell <
> > > > >>>>>>> andrew.purtell@gmail.com>
> > > > >>>>>>>>> wrote:
> > > > >>>>>>>>>
> > > > >>>>>>>>>> Backup is extra tooling around core in my opinion. Like
> > import
> > > > or
> > > > >>>>>>> export.
> > > > >>>>>>>>>> Or the optional MOB tool. It's fine.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>> On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi <
> > > > >>>> mbertozzi@apache.org>
> > > > >>>>>>>>>> wrote:
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> What's the latest opinion around running MR jobs from
> hbase
> > > > >>>> (Master
> > > > >>>>>>> or
> > > > >>>>>>>>>> RS)?
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> I remember in the past that there was discussion about
> not
> > > > >>> having
> > > > >>>> MR
> > > > >>>>>>>>> has
> > > > >>>>>>>>>>> direct dependency of hbase.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> I think some of discussion where around MOB that had a MR
> > job
> > > > to
> > > > >>>>>>>>> compact,
> > > > >>>>>>>>>>> that later was transformed in a non-MR job to be merged,
> I
> > > > think
> > > > >>>> we
> > > > >>>>>>>>> had a
> > > > >>>>>>>>>>> similar discussion for log split/replay.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> the latest is the new Backup feature (HBASE-7912), that
> > runs
> > > a
> > > > >>> MR
> > > > >>>> job
> > > > >>>>>>>>>> from
> > > > >>>>>>>>>>> the master to copy data or restore data.
> > > > >>>>>>>>>>> (backup is also "not really core" as in.. if you don't
> use
> > > > >>> backup
> > > > >>>>>>>>> you'll
> > > > >>>>>>>>>>> not end up running MR jobs, but this was probably true
> for
> > > MOB
> > > > >>> as
> > > > >>>> in
> > > > >>>>>>>>> "if
> > > > >>>>>>>>>>> you don't enable MOB you don't need MR")
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> any thoughts? do we a rule that says "we don't want to
> have
> > > > >>> hbase
> > > > >>>> run
> > > > >>>>>>>>> MR
> > > > >>>>>>>>>>> jobs, only tool started manually by the user can do
> that".
> > or
> > > > >>> can
> > > > >>>> we
> > > > >>>>>>>>>> start
> > > > >>>>>>>>>>> adding MR calls around without problems?
> > > > >>>
> > > >
> > >
> >
>
Re: [DISCUSSION] MR jobs started by Master or RS
Posted by 张铎 <pa...@gmail.com>.
If you guys have already implemented the feature in the MR way and the
patch is ready for landing on master, I'm a -0 on it as I do not want to
block the development progress.
But I strongly suggest later we need to revisit the design and see if we
can seperated the logic from HMaster as much as possible. HA is not a big
problem if you do not store any metada locally. But the ugly code in
HMaster is readlly a problem...
And for security, I have a issue pending for a long time. Can someone help
taking a simple look at it? This is what I mean, ugly code... logout and
destroy the credentials in a subject when it is still being used, and
declared as LimitPrivacy so I can not change the behivor and the only way
to fix it is to write another piece of ugly code...
https://issues.apache.org/jira/browse/HADOOP-13433
2016-09-23 12:53 GMT+08:00 Vladimir Rodionov <vl...@gmail.com>:
> >> If in the future, we find better ways of doing this without using MR, we
> can certainly consider that
>
> Our framework for distributed operations is abstract and allows
> different implementations. MR is just one implementation we provide.
>
> -Vlad
>
> On Thu, Sep 22, 2016 at 9:38 PM, Devaraj Das <dd...@hortonworks.com> wrote:
>
> > Guys, first off apologies for bringing in the topic of MR-based
> > compactions.. But I was thinking more about the SpliceMachine approach of
> > managing compactions in Spark where apparently they saw a lot of
> benefits.
> > Apologies for giving you that sore throat Andrew; I really didn't mean to
> > :-)
> >
> > So on this issue, we have these on the plate:
> > 0. Somehow not use MR but something like that
> > 1. Run a standalone service other than master
> > 2. Shell out from the master
> >
> > I don't think we have a good answer to (0), and I don't think it's even
> > worth the effort of trying to build something when MR is already there,
> and
> > being used by HBase already for some operations.
> >
> > On (1), we have to deal with a myriad of issues - HA of the server not
> > being the least of them all. Security (kerberos authentication, another
> > keytab to manage, etc. etc. etc.). IMO, that approach is DOA. Instead
> let's
> > substitute that (1) with the HBase Master. I haven't seen any good reason
> > why the HBase master shouldn't launch MR jobs if needed. It's not ideal;
> > agreed.
> >
> > Now before going to (2), let's see what are the benefits of running the
> > backup/restore jobs from the master. I think Ted has summarized some of
> the
> > issues that we need to take care of - basically, the master can keep
> track
> > of running jobs, and should it fail, the backup master can continue
> keeping
> > track of it (since the jobId would have been recorded in the proc WAL).
> The
> > master can also do cleanup, etc. of failed backup/restore processes.
> > Security is another issue - the job needs to run as 'hbase' since it owns
> > the data. Having the master launch the job makes it get that privilege.
> In
> > the (2) approach, it's hard to do some of the above management.
> >
> > Guys, just to reiterate, the patch as such is ready from the overall
> > design/arch point of view (maybe code review is still pending from
> Matteo).
> > If in the future, we find better ways of doing this without using MR, we
> > can certainly consider that. But IMO don't think we should block this
> patch
> > from getting merged.
> >
> > ________________________________________
> > From: 张铎 <pa...@gmail.com>
> > Sent: Thursday, September 22, 2016 8:32 PM
> > To: dev@hbase.apache.org
> > Subject: Re: [DISCUSSION] MR jobs started by Master or RS
> >
> > So what about a standalone service other than master? You can use your
> own
> > procedure store in that service?
> >
> > 2016-09-23 11:28 GMT+08:00 Ted Yu <yu...@gmail.com>:
> >
> > > An earlier implementation was client driven.
> > >
> > > But with that approach, it is hard to resume if there is error midway.
> > > Using Procedure V2 makes the backup / restore more robust.
> > >
> > > Another consideration is for security. It is hard to enforce security
> (to
> > > be implemented) for client driven actions.
> > >
> > > Cheers
> > >
> > > > On Sep 22, 2016, at 8:15 PM, Andrew Purtell <
> andrew.purtell@gmail.com>
> > > wrote:
> > > >
> > > > No, this misses Matteo's finer point, which is "shelling out" from
> the
> > > master directly to run MR is a first. Why not drive this with a utility
> > > derived from Tool?
> > > >
> > > > On Sep 22, 2016, at 7:57 PM, Vladimir Rodionov <
> vladrodionov@gmail.com
> > >
> > > wrote:
> > > >
> > > >>>> In our production cluster, it is a common case we just have HDFS
> > and
> > > >>>> HBase deployed.
> > > >>>> If our Master/RS depend on MR framework (especially some features
> we
> > > >>>> have not used at all), it introduced another cost for maintain.
> I
> > > >>>> don't think it is a good idea.
> > > >>
> > > >> So , you are not backup users in this case. Many our customers have
> > full
> > > >> stack deployed and
> > > >> want see backup to be a standard feature. Besides this, nothing will
> > > happen
> > > >> in your cluster
> > > >> if you won't be doing backups.
> > > >>
> > > >> This discussion (we do not want see M/R dependency) goes to nowhere.
> > We
> > > >> asked already, at least twice, to suggest another framework (other
> > than
> > > M/R)
> > > >> for bulk data copy with *conversion*. Still waiting for suggestions.
> > > >>
> > > >> -Vlad
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>> On Thu, Sep 22, 2016 at 7:49 PM, Ted Yu <yu...@gmail.com>
> wrote:
> > > >>>
> > > >>> If MR framework is not deployed in the cluster, hbase still
> functions
> > > >>> normally (post merge).
> > > >>>
> > > >>> In terms of build time dependency, we have long been depending on
> > > >>> mapreduce. Take a look at ExportSnapshot.
> > > >>>
> > > >>> Cheers
> > > >>>
> > > >>> On Thu, Sep 22, 2016 at 7:42 PM, Heng Chen <
> heng.chen.1986@gmail.com
> > >
> > > >>> wrote:
> > > >>>
> > > >>>> In our production cluster, it is a common case we just have HDFS
> > and
> > > >>>> HBase deployed.
> > > >>>> If our Master/RS depend on MR framework (especially some features
> we
> > > >>>> have not used at all), it introduced another cost for maintain.
> I
> > > >>>> don't think it is a good idea.
> > > >>>>
> > > >>>> 2016-09-23 10:28 GMT+08:00 张铎 <pa...@gmail.com>:
> > > >>>>> To be specific, for example, our nice Backup/Restore feature, if
> we
> > > >>> think
> > > >>>>> this is not a core feature of HBase, then we could make it depend
> > on
> > > >>> MR,
> > > >>>>> and start a standalone BackupManager instance that submits MR
> jobs
> > to
> > > >>> do
> > > >>>>> periodical maintenance job. And if we think this is a core
> feature
> > > that
> > > >>>>> everyone should use it, then we'd better implement it without MR
> > > >>>>> dependency, like DLS.
> > > >>>>>
> > > >>>>> Thanks.
> > > >>>>>
> > > >>>>> 2016-09-23 10:11 GMT+08:00 张铎 <pa...@gmail.com>:
> > > >>>>>
> > > >>>>>> I‘m -1 on let master or rs launch MR jobs. It is OK that some of
> > our
> > > >>>>>> features depend on MR but I think the bottom line is that we
> > should
> > > >>>> launch
> > > >>>>>> the jobs from outside manually or by other services.
> > > >>>>>>
> > > >>>>>> 2016-09-23 9:47 GMT+08:00 Andrew Purtell <
> > andrew.purtell@gmail.com
> > > >:
> > > >>>>>>
> > > >>>>>>> Ok, got it. Well "shelling out" is on the line I think, so a
> fair
> > > >>>>>>> question.
> > > >>>>>>>
> > > >>>>>>> Can this be driven by a utility derived from Tool like our
> other
> > MR
> > > >>>> apps?
> > > >>>>>>> The issue is needing the AccessController to decide if allowed?
> > But
> > > >>>> nothing
> > > >>>>>>> prevents the user from running the job manually/independently,
> > > right?
> > > >>>>>>>
> > > >>>>>>>> On Sep 22, 2016, at 3:44 PM, Matteo Bertozzi <
> > > >>>> theo.bertozzi@gmail.com>
> > > >>>>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>> just a remark. my query was not about tools using MR
> (everyone i
> > > >>>> think
> > > >>>>>>> is
> > > >>>>>>>> ok with those).
> > > >>>>>>>> the topic was about: "are we ok with running MR jobs from
> Master
> > > >>> and
> > > >>>> RSs
> > > >>>>>>>> code?" since this will be the first time we do this
> > > >>>>>>>>
> > > >>>>>>>> Matteo
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>> On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das <
> > > >>> ddas@hortonworks.com>
> > > >>>>>>> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>> Very much agree; for tools like ExportSnapshot / Backup /
> > > Restore,
> > > >>>> it's
> > > >>>>>>>>> fine to be dependent on MR. MR is the right framework for
> such.
> > > We
> > > >>>>>>> should
> > > >>>>>>>>> also do compactions using MR (just saying :) )
> > > >>>>>>>>> ________________________________________
> > > >>>>>>>>> From: Ted Yu <yu...@gmail.com>
> > > >>>>>>>>> Sent: Thursday, September 22, 2016 2:00 PM
> > > >>>>>>>>> To: dev@hbase.apache.org
> > > >>>>>>>>> Subject: Re: [DISCUSSION] MR jobs started by Master or RS
> > > >>>>>>>>>
> > > >>>>>>>>> I agree - backup / restore is in the same category as import
> /
> > > >>>> export.
> > > >>>>>>>>>
> > > >>>>>>>>> On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell <
> > > >>>>>>> andrew.purtell@gmail.com>
> > > >>>>>>>>> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>>> Backup is extra tooling around core in my opinion. Like
> import
> > > or
> > > >>>>>>> export.
> > > >>>>>>>>>> Or the optional MOB tool. It's fine.
> > > >>>>>>>>>>
> > > >>>>>>>>>>> On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi <
> > > >>>> mbertozzi@apache.org>
> > > >>>>>>>>>> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> What's the latest opinion around running MR jobs from hbase
> > > >>>> (Master
> > > >>>>>>> or
> > > >>>>>>>>>> RS)?
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I remember in the past that there was discussion about not
> > > >>> having
> > > >>>> MR
> > > >>>>>>>>> has
> > > >>>>>>>>>>> direct dependency of hbase.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I think some of discussion where around MOB that had a MR
> job
> > > to
> > > >>>>>>>>> compact,
> > > >>>>>>>>>>> that later was transformed in a non-MR job to be merged, I
> > > think
> > > >>>> we
> > > >>>>>>>>> had a
> > > >>>>>>>>>>> similar discussion for log split/replay.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> the latest is the new Backup feature (HBASE-7912), that
> runs
> > a
> > > >>> MR
> > > >>>> job
> > > >>>>>>>>>> from
> > > >>>>>>>>>>> the master to copy data or restore data.
> > > >>>>>>>>>>> (backup is also "not really core" as in.. if you don't use
> > > >>> backup
> > > >>>>>>>>> you'll
> > > >>>>>>>>>>> not end up running MR jobs, but this was probably true for
> > MOB
> > > >>> as
> > > >>>> in
> > > >>>>>>>>> "if
> > > >>>>>>>>>>> you don't enable MOB you don't need MR")
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> any thoughts? do we a rule that says "we don't want to have
> > > >>> hbase
> > > >>>> run
> > > >>>>>>>>> MR
> > > >>>>>>>>>>> jobs, only tool started manually by the user can do that".
> or
> > > >>> can
> > > >>>> we
> > > >>>>>>>>>> start
> > > >>>>>>>>>>> adding MR calls around without problems?
> > > >>>
> > >
> >
>
Re: [DISCUSSION] MR jobs started by Master or RS
Posted by Vladimir Rodionov <vl...@gmail.com>.
Just wanted to add one argument of doing this in a Master way :
Client - based backups/restore are very hard (if possible) to make fully
fault tolerant. If client fails abruptly half way, some system data will be
broken, cluster will never return into original state. We disable, for
example splits/merges, balancer during full backup and restore. Failed
client will leave cluster in that state (disabled splits/merges)
-Vlad
On Thu, Sep 22, 2016 at 9:53 PM, Vladimir Rodionov <vl...@gmail.com>
wrote:
> >> If in the future, we find better ways of doing this without using MR,
> we can certainly consider that
>
> Our framework for distributed operations is abstract and allows
> different implementations. MR is just one implementation we provide.
>
> -Vlad
>
> On Thu, Sep 22, 2016 at 9:38 PM, Devaraj Das <dd...@hortonworks.com> wrote:
>
>> Guys, first off apologies for bringing in the topic of MR-based
>> compactions.. But I was thinking more about the SpliceMachine approach of
>> managing compactions in Spark where apparently they saw a lot of benefits.
>> Apologies for giving you that sore throat Andrew; I really didn't mean to
>> :-)
>>
>> So on this issue, we have these on the plate:
>> 0. Somehow not use MR but something like that
>> 1. Run a standalone service other than master
>> 2. Shell out from the master
>>
>> I don't think we have a good answer to (0), and I don't think it's even
>> worth the effort of trying to build something when MR is already there, and
>> being used by HBase already for some operations.
>>
>> On (1), we have to deal with a myriad of issues - HA of the server not
>> being the least of them all. Security (kerberos authentication, another
>> keytab to manage, etc. etc. etc.). IMO, that approach is DOA. Instead let's
>> substitute that (1) with the HBase Master. I haven't seen any good reason
>> why the HBase master shouldn't launch MR jobs if needed. It's not ideal;
>> agreed.
>>
>> Now before going to (2), let's see what are the benefits of running the
>> backup/restore jobs from the master. I think Ted has summarized some of the
>> issues that we need to take care of - basically, the master can keep track
>> of running jobs, and should it fail, the backup master can continue keeping
>> track of it (since the jobId would have been recorded in the proc WAL). The
>> master can also do cleanup, etc. of failed backup/restore processes.
>> Security is another issue - the job needs to run as 'hbase' since it owns
>> the data. Having the master launch the job makes it get that privilege. In
>> the (2) approach, it's hard to do some of the above management.
>>
>> Guys, just to reiterate, the patch as such is ready from the overall
>> design/arch point of view (maybe code review is still pending from Matteo).
>> If in the future, we find better ways of doing this without using MR, we
>> can certainly consider that. But IMO don't think we should block this patch
>> from getting merged.
>>
>> ________________________________________
>> From: 张铎 <pa...@gmail.com>
>> Sent: Thursday, September 22, 2016 8:32 PM
>> To: dev@hbase.apache.org
>> Subject: Re: [DISCUSSION] MR jobs started by Master or RS
>>
>> So what about a standalone service other than master? You can use your own
>> procedure store in that service?
>>
>> 2016-09-23 11:28 GMT+08:00 Ted Yu <yu...@gmail.com>:
>>
>> > An earlier implementation was client driven.
>> >
>> > But with that approach, it is hard to resume if there is error midway.
>> > Using Procedure V2 makes the backup / restore more robust.
>> >
>> > Another consideration is for security. It is hard to enforce security
>> (to
>> > be implemented) for client driven actions.
>> >
>> > Cheers
>> >
>> > > On Sep 22, 2016, at 8:15 PM, Andrew Purtell <andrew.purtell@gmail.com
>> >
>> > wrote:
>> > >
>> > > No, this misses Matteo's finer point, which is "shelling out" from the
>> > master directly to run MR is a first. Why not drive this with a utility
>> > derived from Tool?
>> > >
>> > > On Sep 22, 2016, at 7:57 PM, Vladimir Rodionov <
>> vladrodionov@gmail.com>
>> > wrote:
>> > >
>> > >>>> In our production cluster, it is a common case we just have HDFS
>> and
>> > >>>> HBase deployed.
>> > >>>> If our Master/RS depend on MR framework (especially some features
>> we
>> > >>>> have not used at all), it introduced another cost for maintain. I
>> > >>>> don't think it is a good idea.
>> > >>
>> > >> So , you are not backup users in this case. Many our customers have
>> full
>> > >> stack deployed and
>> > >> want see backup to be a standard feature. Besides this, nothing will
>> > happen
>> > >> in your cluster
>> > >> if you won't be doing backups.
>> > >>
>> > >> This discussion (we do not want see M/R dependency) goes to nowhere.
>> We
>> > >> asked already, at least twice, to suggest another framework (other
>> than
>> > M/R)
>> > >> for bulk data copy with *conversion*. Still waiting for suggestions.
>> > >>
>> > >> -Vlad
>> > >>
>> > >>
>> > >>
>> > >>
>> > >>> On Thu, Sep 22, 2016 at 7:49 PM, Ted Yu <yu...@gmail.com>
>> wrote:
>> > >>>
>> > >>> If MR framework is not deployed in the cluster, hbase still
>> functions
>> > >>> normally (post merge).
>> > >>>
>> > >>> In terms of build time dependency, we have long been depending on
>> > >>> mapreduce. Take a look at ExportSnapshot.
>> > >>>
>> > >>> Cheers
>> > >>>
>> > >>> On Thu, Sep 22, 2016 at 7:42 PM, Heng Chen <
>> heng.chen.1986@gmail.com>
>> > >>> wrote:
>> > >>>
>> > >>>> In our production cluster, it is a common case we just have HDFS
>> and
>> > >>>> HBase deployed.
>> > >>>> If our Master/RS depend on MR framework (especially some features
>> we
>> > >>>> have not used at all), it introduced another cost for maintain. I
>> > >>>> don't think it is a good idea.
>> > >>>>
>> > >>>> 2016-09-23 10:28 GMT+08:00 张铎 <pa...@gmail.com>:
>> > >>>>> To be specific, for example, our nice Backup/Restore feature, if
>> we
>> > >>> think
>> > >>>>> this is not a core feature of HBase, then we could make it depend
>> on
>> > >>> MR,
>> > >>>>> and start a standalone BackupManager instance that submits MR
>> jobs to
>> > >>> do
>> > >>>>> periodical maintenance job. And if we think this is a core feature
>> > that
>> > >>>>> everyone should use it, then we'd better implement it without MR
>> > >>>>> dependency, like DLS.
>> > >>>>>
>> > >>>>> Thanks.
>> > >>>>>
>> > >>>>> 2016-09-23 10:11 GMT+08:00 张铎 <pa...@gmail.com>:
>> > >>>>>
>> > >>>>>> I‘m -1 on let master or rs launch MR jobs. It is OK that some of
>> our
>> > >>>>>> features depend on MR but I think the bottom line is that we
>> should
>> > >>>> launch
>> > >>>>>> the jobs from outside manually or by other services.
>> > >>>>>>
>> > >>>>>> 2016-09-23 9:47 GMT+08:00 Andrew Purtell <
>> andrew.purtell@gmail.com
>> > >:
>> > >>>>>>
>> > >>>>>>> Ok, got it. Well "shelling out" is on the line I think, so a
>> fair
>> > >>>>>>> question.
>> > >>>>>>>
>> > >>>>>>> Can this be driven by a utility derived from Tool like our
>> other MR
>> > >>>> apps?
>> > >>>>>>> The issue is needing the AccessController to decide if allowed?
>> But
>> > >>>> nothing
>> > >>>>>>> prevents the user from running the job manually/independently,
>> > right?
>> > >>>>>>>
>> > >>>>>>>> On Sep 22, 2016, at 3:44 PM, Matteo Bertozzi <
>> > >>>> theo.bertozzi@gmail.com>
>> > >>>>>>> wrote:
>> > >>>>>>>>
>> > >>>>>>>> just a remark. my query was not about tools using MR (everyone
>> i
>> > >>>> think
>> > >>>>>>> is
>> > >>>>>>>> ok with those).
>> > >>>>>>>> the topic was about: "are we ok with running MR jobs from
>> Master
>> > >>> and
>> > >>>> RSs
>> > >>>>>>>> code?" since this will be the first time we do this
>> > >>>>>>>>
>> > >>>>>>>> Matteo
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>>> On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das <
>> > >>> ddas@hortonworks.com>
>> > >>>>>>> wrote:
>> > >>>>>>>>>
>> > >>>>>>>>> Very much agree; for tools like ExportSnapshot / Backup /
>> > Restore,
>> > >>>> it's
>> > >>>>>>>>> fine to be dependent on MR. MR is the right framework for
>> such.
>> > We
>> > >>>>>>> should
>> > >>>>>>>>> also do compactions using MR (just saying :) )
>> > >>>>>>>>> ________________________________________
>> > >>>>>>>>> From: Ted Yu <yu...@gmail.com>
>> > >>>>>>>>> Sent: Thursday, September 22, 2016 2:00 PM
>> > >>>>>>>>> To: dev@hbase.apache.org
>> > >>>>>>>>> Subject: Re: [DISCUSSION] MR jobs started by Master or RS
>> > >>>>>>>>>
>> > >>>>>>>>> I agree - backup / restore is in the same category as import /
>> > >>>> export.
>> > >>>>>>>>>
>> > >>>>>>>>> On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell <
>> > >>>>>>> andrew.purtell@gmail.com>
>> > >>>>>>>>> wrote:
>> > >>>>>>>>>
>> > >>>>>>>>>> Backup is extra tooling around core in my opinion. Like
>> import
>> > or
>> > >>>>>>> export.
>> > >>>>>>>>>> Or the optional MOB tool. It's fine.
>> > >>>>>>>>>>
>> > >>>>>>>>>>> On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi <
>> > >>>> mbertozzi@apache.org>
>> > >>>>>>>>>> wrote:
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> What's the latest opinion around running MR jobs from hbase
>> > >>>> (Master
>> > >>>>>>> or
>> > >>>>>>>>>> RS)?
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> I remember in the past that there was discussion about not
>> > >>> having
>> > >>>> MR
>> > >>>>>>>>> has
>> > >>>>>>>>>>> direct dependency of hbase.
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> I think some of discussion where around MOB that had a MR
>> job
>> > to
>> > >>>>>>>>> compact,
>> > >>>>>>>>>>> that later was transformed in a non-MR job to be merged, I
>> > think
>> > >>>> we
>> > >>>>>>>>> had a
>> > >>>>>>>>>>> similar discussion for log split/replay.
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> the latest is the new Backup feature (HBASE-7912), that
>> runs a
>> > >>> MR
>> > >>>> job
>> > >>>>>>>>>> from
>> > >>>>>>>>>>> the master to copy data or restore data.
>> > >>>>>>>>>>> (backup is also "not really core" as in.. if you don't use
>> > >>> backup
>> > >>>>>>>>> you'll
>> > >>>>>>>>>>> not end up running MR jobs, but this was probably true for
>> MOB
>> > >>> as
>> > >>>> in
>> > >>>>>>>>> "if
>> > >>>>>>>>>>> you don't enable MOB you don't need MR")
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> any thoughts? do we a rule that says "we don't want to have
>> > >>> hbase
>> > >>>> run
>> > >>>>>>>>> MR
>> > >>>>>>>>>>> jobs, only tool started manually by the user can do that".
>> or
>> > >>> can
>> > >>>> we
>> > >>>>>>>>>> start
>> > >>>>>>>>>>> adding MR calls around without problems?
>> > >>>
>> >
>>
>
>
Re: [DISCUSSION] MR jobs started by Master or RS
Posted by Devaraj Das <dd...@hortonworks.com>.
All the better, Vlad!
On Thu, Sep 22, 2016 at 9:53 PM -0700, "Vladimir Rodionov" <vl...@gmail.com>> wrote:
>> If in the future, we find better ways of doing this without using MR, we
can certainly consider that
Our framework for distributed operations is abstract and allows
different implementations. MR is just one implementation we provide.
-Vlad
On Thu, Sep 22, 2016 at 9:38 PM, Devaraj Das <dd...@hortonworks.com> wrote:
> Guys, first off apologies for bringing in the topic of MR-based
> compactions.. But I was thinking more about the SpliceMachine approach of
> managing compactions in Spark where apparently they saw a lot of benefits.
> Apologies for giving you that sore throat Andrew; I really didn't mean to
> :-)
>
> So on this issue, we have these on the plate:
> 0. Somehow not use MR but something like that
> 1. Run a standalone service other than master
> 2. Shell out from the master
>
> I don't think we have a good answer to (0), and I don't think it's even
> worth the effort of trying to build something when MR is already there, and
> being used by HBase already for some operations.
>
> On (1), we have to deal with a myriad of issues - HA of the server not
> being the least of them all. Security (kerberos authentication, another
> keytab to manage, etc. etc. etc.). IMO, that approach is DOA. Instead let's
> substitute that (1) with the HBase Master. I haven't seen any good reason
> why the HBase master shouldn't launch MR jobs if needed. It's not ideal;
> agreed.
>
> Now before going to (2), let's see what are the benefits of running the
> backup/restore jobs from the master. I think Ted has summarized some of the
> issues that we need to take care of - basically, the master can keep track
> of running jobs, and should it fail, the backup master can continue keeping
> track of it (since the jobId would have been recorded in the proc WAL). The
> master can also do cleanup, etc. of failed backup/restore processes.
> Security is another issue - the job needs to run as 'hbase' since it owns
> the data. Having the master launch the job makes it get that privilege. In
> the (2) approach, it's hard to do some of the above management.
>
> Guys, just to reiterate, the patch as such is ready from the overall
> design/arch point of view (maybe code review is still pending from Matteo).
> If in the future, we find better ways of doing this without using MR, we
> can certainly consider that. But IMO don't think we should block this patch
> from getting merged.
>
> ________________________________________
> From: 张铎 <pa...@gmail.com>
> Sent: Thursday, September 22, 2016 8:32 PM
> To: dev@hbase.apache.org
> Subject: Re: [DISCUSSION] MR jobs started by Master or RS
>
> So what about a standalone service other than master? You can use your own
> procedure store in that service?
>
> 2016-09-23 11:28 GMT+08:00 Ted Yu <yu...@gmail.com>:
>
> > An earlier implementation was client driven.
> >
> > But with that approach, it is hard to resume if there is error midway.
> > Using Procedure V2 makes the backup / restore more robust.
> >
> > Another consideration is for security. It is hard to enforce security (to
> > be implemented) for client driven actions.
> >
> > Cheers
> >
> > > On Sep 22, 2016, at 8:15 PM, Andrew Purtell <an...@gmail.com>
> > wrote:
> > >
> > > No, this misses Matteo's finer point, which is "shelling out" from the
> > master directly to run MR is a first. Why not drive this with a utility
> > derived from Tool?
> > >
> > > On Sep 22, 2016, at 7:57 PM, Vladimir Rodionov <vladrodionov@gmail.com
> >
> > wrote:
> > >
> > >>>> In our production cluster, it is a common case we just have HDFS
> and
> > >>>> HBase deployed.
> > >>>> If our Master/RS depend on MR framework (especially some features we
> > >>>> have not used at all), it introduced another cost for maintain. I
> > >>>> don't think it is a good idea.
> > >>
> > >> So , you are not backup users in this case. Many our customers have
> full
> > >> stack deployed and
> > >> want see backup to be a standard feature. Besides this, nothing will
> > happen
> > >> in your cluster
> > >> if you won't be doing backups.
> > >>
> > >> This discussion (we do not want see M/R dependency) goes to nowhere.
> We
> > >> asked already, at least twice, to suggest another framework (other
> than
> > M/R)
> > >> for bulk data copy with *conversion*. Still waiting for suggestions.
> > >>
> > >> -Vlad
> > >>
> > >>
> > >>
> > >>
> > >>> On Thu, Sep 22, 2016 at 7:49 PM, Ted Yu <yu...@gmail.com> wrote:
> > >>>
> > >>> If MR framework is not deployed in the cluster, hbase still functions
> > >>> normally (post merge).
> > >>>
> > >>> In terms of build time dependency, we have long been depending on
> > >>> mapreduce. Take a look at ExportSnapshot.
> > >>>
> > >>> Cheers
> > >>>
> > >>> On Thu, Sep 22, 2016 at 7:42 PM, Heng Chen <heng.chen.1986@gmail.com
> >
> > >>> wrote:
> > >>>
> > >>>> In our production cluster, it is a common case we just have HDFS
> and
> > >>>> HBase deployed.
> > >>>> If our Master/RS depend on MR framework (especially some features we
> > >>>> have not used at all), it introduced another cost for maintain. I
> > >>>> don't think it is a good idea.
> > >>>>
> > >>>> 2016-09-23 10:28 GMT+08:00 张铎 <pa...@gmail.com>:
> > >>>>> To be specific, for example, our nice Backup/Restore feature, if we
> > >>> think
> > >>>>> this is not a core feature of HBase, then we could make it depend
> on
> > >>> MR,
> > >>>>> and start a standalone BackupManager instance that submits MR jobs
> to
> > >>> do
> > >>>>> periodical maintenance job. And if we think this is a core feature
> > that
> > >>>>> everyone should use it, then we'd better implement it without MR
> > >>>>> dependency, like DLS.
> > >>>>>
> > >>>>> Thanks.
> > >>>>>
> > >>>>> 2016-09-23 10:11 GMT+08:00 张铎 <pa...@gmail.com>:
> > >>>>>
> > >>>>>> I‘m -1 on let master or rs launch MR jobs. It is OK that some of
> our
> > >>>>>> features depend on MR but I think the bottom line is that we
> should
> > >>>> launch
> > >>>>>> the jobs from outside manually or by other services.
> > >>>>>>
> > >>>>>> 2016-09-23 9:47 GMT+08:00 Andrew Purtell <
> andrew.purtell@gmail.com
> > >:
> > >>>>>>
> > >>>>>>> Ok, got it. Well "shelling out" is on the line I think, so a fair
> > >>>>>>> question.
> > >>>>>>>
> > >>>>>>> Can this be driven by a utility derived from Tool like our other
> MR
> > >>>> apps?
> > >>>>>>> The issue is needing the AccessController to decide if allowed?
> But
> > >>>> nothing
> > >>>>>>> prevents the user from running the job manually/independently,
> > right?
> > >>>>>>>
> > >>>>>>>> On Sep 22, 2016, at 3:44 PM, Matteo Bertozzi <
> > >>>> theo.bertozzi@gmail.com>
> > >>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>> just a remark. my query was not about tools using MR (everyone i
> > >>>> think
> > >>>>>>> is
> > >>>>>>>> ok with those).
> > >>>>>>>> the topic was about: "are we ok with running MR jobs from Master
> > >>> and
> > >>>> RSs
> > >>>>>>>> code?" since this will be the first time we do this
> > >>>>>>>>
> > >>>>>>>> Matteo
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>> On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das <
> > >>> ddas@hortonworks.com>
> > >>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>> Very much agree; for tools like ExportSnapshot / Backup /
> > Restore,
> > >>>> it's
> > >>>>>>>>> fine to be dependent on MR. MR is the right framework for such.
> > We
> > >>>>>>> should
> > >>>>>>>>> also do compactions using MR (just saying :) )
> > >>>>>>>>> ________________________________________
> > >>>>>>>>> From: Ted Yu <yu...@gmail.com>
> > >>>>>>>>> Sent: Thursday, September 22, 2016 2:00 PM
> > >>>>>>>>> To: dev@hbase.apache.org
> > >>>>>>>>> Subject: Re: [DISCUSSION] MR jobs started by Master or RS
> > >>>>>>>>>
> > >>>>>>>>> I agree - backup / restore is in the same category as import /
> > >>>> export.
> > >>>>>>>>>
> > >>>>>>>>> On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell <
> > >>>>>>> andrew.purtell@gmail.com>
> > >>>>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> Backup is extra tooling around core in my opinion. Like import
> > or
> > >>>>>>> export.
> > >>>>>>>>>> Or the optional MOB tool. It's fine.
> > >>>>>>>>>>
> > >>>>>>>>>>> On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi <
> > >>>> mbertozzi@apache.org>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>> What's the latest opinion around running MR jobs from hbase
> > >>>> (Master
> > >>>>>>> or
> > >>>>>>>>>> RS)?
> > >>>>>>>>>>>
> > >>>>>>>>>>> I remember in the past that there was discussion about not
> > >>> having
> > >>>> MR
> > >>>>>>>>> has
> > >>>>>>>>>>> direct dependency of hbase.
> > >>>>>>>>>>>
> > >>>>>>>>>>> I think some of discussion where around MOB that had a MR job
> > to
> > >>>>>>>>> compact,
> > >>>>>>>>>>> that later was transformed in a non-MR job to be merged, I
> > think
> > >>>> we
> > >>>>>>>>> had a
> > >>>>>>>>>>> similar discussion for log split/replay.
> > >>>>>>>>>>>
> > >>>>>>>>>>> the latest is the new Backup feature (HBASE-7912), that runs
> a
> > >>> MR
> > >>>> job
> > >>>>>>>>>> from
> > >>>>>>>>>>> the master to copy data or restore data.
> > >>>>>>>>>>> (backup is also "not really core" as in.. if you don't use
> > >>> backup
> > >>>>>>>>> you'll
> > >>>>>>>>>>> not end up running MR jobs, but this was probably true for
> MOB
> > >>> as
> > >>>> in
> > >>>>>>>>> "if
> > >>>>>>>>>>> you don't enable MOB you don't need MR")
> > >>>>>>>>>>>
> > >>>>>>>>>>> any thoughts? do we a rule that says "we don't want to have
> > >>> hbase
> > >>>> run
> > >>>>>>>>> MR
> > >>>>>>>>>>> jobs, only tool started manually by the user can do that". or
> > >>> can
> > >>>> we
> > >>>>>>>>>> start
> > >>>>>>>>>>> adding MR calls around without problems?
> > >>>
> >
>
Re: [DISCUSSION] MR jobs started by Master or RS
Posted by Vladimir Rodionov <vl...@gmail.com>.
>> If in the future, we find better ways of doing this without using MR, we
can certainly consider that
Our framework for distributed operations is abstract and allows
different implementations. MR is just one implementation we provide.
-Vlad
On Thu, Sep 22, 2016 at 9:38 PM, Devaraj Das <dd...@hortonworks.com> wrote:
> Guys, first off apologies for bringing in the topic of MR-based
> compactions.. But I was thinking more about the SpliceMachine approach of
> managing compactions in Spark where apparently they saw a lot of benefits.
> Apologies for giving you that sore throat Andrew; I really didn't mean to
> :-)
>
> So on this issue, we have these on the plate:
> 0. Somehow not use MR but something like that
> 1. Run a standalone service other than master
> 2. Shell out from the master
>
> I don't think we have a good answer to (0), and I don't think it's even
> worth the effort of trying to build something when MR is already there, and
> being used by HBase already for some operations.
>
> On (1), we have to deal with a myriad of issues - HA of the server not
> being the least of them all. Security (kerberos authentication, another
> keytab to manage, etc. etc. etc.). IMO, that approach is DOA. Instead let's
> substitute that (1) with the HBase Master. I haven't seen any good reason
> why the HBase master shouldn't launch MR jobs if needed. It's not ideal;
> agreed.
>
> Now before going to (2), let's see what are the benefits of running the
> backup/restore jobs from the master. I think Ted has summarized some of the
> issues that we need to take care of - basically, the master can keep track
> of running jobs, and should it fail, the backup master can continue keeping
> track of it (since the jobId would have been recorded in the proc WAL). The
> master can also do cleanup, etc. of failed backup/restore processes.
> Security is another issue - the job needs to run as 'hbase' since it owns
> the data. Having the master launch the job makes it get that privilege. In
> the (2) approach, it's hard to do some of the above management.
>
> Guys, just to reiterate, the patch as such is ready from the overall
> design/arch point of view (maybe code review is still pending from Matteo).
> If in the future, we find better ways of doing this without using MR, we
> can certainly consider that. But IMO don't think we should block this patch
> from getting merged.
>
> ________________________________________
> From: 张铎 <pa...@gmail.com>
> Sent: Thursday, September 22, 2016 8:32 PM
> To: dev@hbase.apache.org
> Subject: Re: [DISCUSSION] MR jobs started by Master or RS
>
> So what about a standalone service other than master? You can use your own
> procedure store in that service?
>
> 2016-09-23 11:28 GMT+08:00 Ted Yu <yu...@gmail.com>:
>
> > An earlier implementation was client driven.
> >
> > But with that approach, it is hard to resume if there is error midway.
> > Using Procedure V2 makes the backup / restore more robust.
> >
> > Another consideration is for security. It is hard to enforce security (to
> > be implemented) for client driven actions.
> >
> > Cheers
> >
> > > On Sep 22, 2016, at 8:15 PM, Andrew Purtell <an...@gmail.com>
> > wrote:
> > >
> > > No, this misses Matteo's finer point, which is "shelling out" from the
> > master directly to run MR is a first. Why not drive this with a utility
> > derived from Tool?
> > >
> > > On Sep 22, 2016, at 7:57 PM, Vladimir Rodionov <vladrodionov@gmail.com
> >
> > wrote:
> > >
> > >>>> In our production cluster, it is a common case we just have HDFS
> and
> > >>>> HBase deployed.
> > >>>> If our Master/RS depend on MR framework (especially some features we
> > >>>> have not used at all), it introduced another cost for maintain. I
> > >>>> don't think it is a good idea.
> > >>
> > >> So , you are not backup users in this case. Many our customers have
> full
> > >> stack deployed and
> > >> want see backup to be a standard feature. Besides this, nothing will
> > happen
> > >> in your cluster
> > >> if you won't be doing backups.
> > >>
> > >> This discussion (we do not want see M/R dependency) goes to nowhere.
> We
> > >> asked already, at least twice, to suggest another framework (other
> than
> > M/R)
> > >> for bulk data copy with *conversion*. Still waiting for suggestions.
> > >>
> > >> -Vlad
> > >>
> > >>
> > >>
> > >>
> > >>> On Thu, Sep 22, 2016 at 7:49 PM, Ted Yu <yu...@gmail.com> wrote:
> > >>>
> > >>> If MR framework is not deployed in the cluster, hbase still functions
> > >>> normally (post merge).
> > >>>
> > >>> In terms of build time dependency, we have long been depending on
> > >>> mapreduce. Take a look at ExportSnapshot.
> > >>>
> > >>> Cheers
> > >>>
> > >>> On Thu, Sep 22, 2016 at 7:42 PM, Heng Chen <heng.chen.1986@gmail.com
> >
> > >>> wrote:
> > >>>
> > >>>> In our production cluster, it is a common case we just have HDFS
> and
> > >>>> HBase deployed.
> > >>>> If our Master/RS depend on MR framework (especially some features we
> > >>>> have not used at all), it introduced another cost for maintain. I
> > >>>> don't think it is a good idea.
> > >>>>
> > >>>> 2016-09-23 10:28 GMT+08:00 张铎 <pa...@gmail.com>:
> > >>>>> To be specific, for example, our nice Backup/Restore feature, if we
> > >>> think
> > >>>>> this is not a core feature of HBase, then we could make it depend
> on
> > >>> MR,
> > >>>>> and start a standalone BackupManager instance that submits MR jobs
> to
> > >>> do
> > >>>>> periodical maintenance job. And if we think this is a core feature
> > that
> > >>>>> everyone should use it, then we'd better implement it without MR
> > >>>>> dependency, like DLS.
> > >>>>>
> > >>>>> Thanks.
> > >>>>>
> > >>>>> 2016-09-23 10:11 GMT+08:00 张铎 <pa...@gmail.com>:
> > >>>>>
> > >>>>>> I‘m -1 on let master or rs launch MR jobs. It is OK that some of
> our
> > >>>>>> features depend on MR but I think the bottom line is that we
> should
> > >>>> launch
> > >>>>>> the jobs from outside manually or by other services.
> > >>>>>>
> > >>>>>> 2016-09-23 9:47 GMT+08:00 Andrew Purtell <
> andrew.purtell@gmail.com
> > >:
> > >>>>>>
> > >>>>>>> Ok, got it. Well "shelling out" is on the line I think, so a fair
> > >>>>>>> question.
> > >>>>>>>
> > >>>>>>> Can this be driven by a utility derived from Tool like our other
> MR
> > >>>> apps?
> > >>>>>>> The issue is needing the AccessController to decide if allowed?
> But
> > >>>> nothing
> > >>>>>>> prevents the user from running the job manually/independently,
> > right?
> > >>>>>>>
> > >>>>>>>> On Sep 22, 2016, at 3:44 PM, Matteo Bertozzi <
> > >>>> theo.bertozzi@gmail.com>
> > >>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>> just a remark. my query was not about tools using MR (everyone i
> > >>>> think
> > >>>>>>> is
> > >>>>>>>> ok with those).
> > >>>>>>>> the topic was about: "are we ok with running MR jobs from Master
> > >>> and
> > >>>> RSs
> > >>>>>>>> code?" since this will be the first time we do this
> > >>>>>>>>
> > >>>>>>>> Matteo
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>> On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das <
> > >>> ddas@hortonworks.com>
> > >>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>> Very much agree; for tools like ExportSnapshot / Backup /
> > Restore,
> > >>>> it's
> > >>>>>>>>> fine to be dependent on MR. MR is the right framework for such.
> > We
> > >>>>>>> should
> > >>>>>>>>> also do compactions using MR (just saying :) )
> > >>>>>>>>> ________________________________________
> > >>>>>>>>> From: Ted Yu <yu...@gmail.com>
> > >>>>>>>>> Sent: Thursday, September 22, 2016 2:00 PM
> > >>>>>>>>> To: dev@hbase.apache.org
> > >>>>>>>>> Subject: Re: [DISCUSSION] MR jobs started by Master or RS
> > >>>>>>>>>
> > >>>>>>>>> I agree - backup / restore is in the same category as import /
> > >>>> export.
> > >>>>>>>>>
> > >>>>>>>>> On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell <
> > >>>>>>> andrew.purtell@gmail.com>
> > >>>>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> Backup is extra tooling around core in my opinion. Like import
> > or
> > >>>>>>> export.
> > >>>>>>>>>> Or the optional MOB tool. It's fine.
> > >>>>>>>>>>
> > >>>>>>>>>>> On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi <
> > >>>> mbertozzi@apache.org>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>> What's the latest opinion around running MR jobs from hbase
> > >>>> (Master
> > >>>>>>> or
> > >>>>>>>>>> RS)?
> > >>>>>>>>>>>
> > >>>>>>>>>>> I remember in the past that there was discussion about not
> > >>> having
> > >>>> MR
> > >>>>>>>>> has
> > >>>>>>>>>>> direct dependency of hbase.
> > >>>>>>>>>>>
> > >>>>>>>>>>> I think some of discussion where around MOB that had a MR job
> > to
> > >>>>>>>>> compact,
> > >>>>>>>>>>> that later was transformed in a non-MR job to be merged, I
> > think
> > >>>> we
> > >>>>>>>>> had a
> > >>>>>>>>>>> similar discussion for log split/replay.
> > >>>>>>>>>>>
> > >>>>>>>>>>> the latest is the new Backup feature (HBASE-7912), that runs
> a
> > >>> MR
> > >>>> job
> > >>>>>>>>>> from
> > >>>>>>>>>>> the master to copy data or restore data.
> > >>>>>>>>>>> (backup is also "not really core" as in.. if you don't use
> > >>> backup
> > >>>>>>>>> you'll
> > >>>>>>>>>>> not end up running MR jobs, but this was probably true for
> MOB
> > >>> as
> > >>>> in
> > >>>>>>>>> "if
> > >>>>>>>>>>> you don't enable MOB you don't need MR")
> > >>>>>>>>>>>
> > >>>>>>>>>>> any thoughts? do we a rule that says "we don't want to have
> > >>> hbase
> > >>>> run
> > >>>>>>>>> MR
> > >>>>>>>>>>> jobs, only tool started manually by the user can do that". or
> > >>> can
> > >>>> we
> > >>>>>>>>>> start
> > >>>>>>>>>>> adding MR calls around without problems?
> > >>>
> >
>
Re: [DISCUSSION] MR jobs started by Master or RS
Posted by 张铎 <pa...@gmail.com>.
2016-09-23 12:38 GMT+08:00 Devaraj Das <dd...@hortonworks.com>:
> Guys, first off apologies for bringing in the topic of MR-based
> compactions.. But I was thinking more about the SpliceMachine approach of
> managing compactions in Spark where apparently they saw a lot of benefits.
> Apologies for giving you that sore throat Andrew; I really didn't mean to
> :-)
>
> So on this issue, we have these on the plate:
> 0. Somehow not use MR but something like that
> 1. Run a standalone service other than master
> 2. Shell out from the master
>
> I don't think we have a good answer to (0), and I don't think it's even
> worth the effort of trying to build something when MR is already there, and
> being used by HBase already for some operations.
>
> On (1), we have to deal with a myriad of issues - HA of the server not
> being the least of them all. Security (kerberos authentication, another
> keytab to manage, etc. etc. etc.). IMO, that approach is DOA. Instead let's
> substitute that (1) with the HBase Master. I haven't seen any good reason
> why the HBase master shouldn't launch MR jobs if needed. It's not ideal;
> agreed.
>
> We have already put lots of stuffs in HMaster, especially on startup, we
even need to run the initializations in a background thread to let the
master get up quickly and causes lots of races. I do not think it is a good
idea to keep messing up the code there.
For MR, as I said above, configuration. I do not want to restart HMaster if
I just want to tune the config of a backup MR job. Yes, you could introduce
new shell commands to do that, change job config, change YARN cluster, and
persist the change to some places, maybe zookeeper? Oh no...
> Now before going to (2), let's see what are the benefits of running the
> backup/restore jobs from the master. I think Ted has summarized some of the
> issues that we need to take care of - basically, the master can keep track
> of running jobs, and should it fail, the backup master can continue keeping
> track of it (since the jobId would have been recorded in the proc WAL). The
> master can also do cleanup, etc. of failed backup/restore processes.
> Security is another issue - the job needs to run as 'hbase' since it owns
> the data. Having the master launch the job makes it get that privilege. In
> the (2) approach, it's hard to do some of the above management.
>
> Guys, just to reiterate, the patch as such is ready from the overall
> design/arch point of view (maybe code review is still pending from Matteo).
> If in the future, we find better ways of doing this without using MR, we
> can certainly consider that. But IMO don't think we should block this patch
> from getting merged.
>
> ________________________________________
> From: 张铎 <pa...@gmail.com>
> Sent: Thursday, September 22, 2016 8:32 PM
> To: dev@hbase.apache.org
> Subject: Re: [DISCUSSION] MR jobs started by Master or RS
>
> So what about a standalone service other than master? You can use your own
> procedure store in that service?
>
> 2016-09-23 11:28 GMT+08:00 Ted Yu <yu...@gmail.com>:
>
> > An earlier implementation was client driven.
> >
> > But with that approach, it is hard to resume if there is error midway.
> > Using Procedure V2 makes the backup / restore more robust.
> >
> > Another consideration is for security. It is hard to enforce security (to
> > be implemented) for client driven actions.
> >
> > Cheers
> >
> > > On Sep 22, 2016, at 8:15 PM, Andrew Purtell <an...@gmail.com>
> > wrote:
> > >
> > > No, this misses Matteo's finer point, which is "shelling out" from the
> > master directly to run MR is a first. Why not drive this with a utility
> > derived from Tool?
> > >
> > > On Sep 22, 2016, at 7:57 PM, Vladimir Rodionov <vladrodionov@gmail.com
> >
> > wrote:
> > >
> > >>>> In our production cluster, it is a common case we just have HDFS
> and
> > >>>> HBase deployed.
> > >>>> If our Master/RS depend on MR framework (especially some features we
> > >>>> have not used at all), it introduced another cost for maintain. I
> > >>>> don't think it is a good idea.
> > >>
> > >> So , you are not backup users in this case. Many our customers have
> full
> > >> stack deployed and
> > >> want see backup to be a standard feature. Besides this, nothing will
> > happen
> > >> in your cluster
> > >> if you won't be doing backups.
> > >>
> > >> This discussion (we do not want see M/R dependency) goes to nowhere.
> We
> > >> asked already, at least twice, to suggest another framework (other
> than
> > M/R)
> > >> for bulk data copy with *conversion*. Still waiting for suggestions.
> > >>
> > >> -Vlad
> > >>
> > >>
> > >>
> > >>
> > >>> On Thu, Sep 22, 2016 at 7:49 PM, Ted Yu <yu...@gmail.com> wrote:
> > >>>
> > >>> If MR framework is not deployed in the cluster, hbase still functions
> > >>> normally (post merge).
> > >>>
> > >>> In terms of build time dependency, we have long been depending on
> > >>> mapreduce. Take a look at ExportSnapshot.
> > >>>
> > >>> Cheers
> > >>>
> > >>> On Thu, Sep 22, 2016 at 7:42 PM, Heng Chen <heng.chen.1986@gmail.com
> >
> > >>> wrote:
> > >>>
> > >>>> In our production cluster, it is a common case we just have HDFS
> and
> > >>>> HBase deployed.
> > >>>> If our Master/RS depend on MR framework (especially some features we
> > >>>> have not used at all), it introduced another cost for maintain. I
> > >>>> don't think it is a good idea.
> > >>>>
> > >>>> 2016-09-23 10:28 GMT+08:00 张铎 <pa...@gmail.com>:
> > >>>>> To be specific, for example, our nice Backup/Restore feature, if we
> > >>> think
> > >>>>> this is not a core feature of HBase, then we could make it depend
> on
> > >>> MR,
> > >>>>> and start a standalone BackupManager instance that submits MR jobs
> to
> > >>> do
> > >>>>> periodical maintenance job. And if we think this is a core feature
> > that
> > >>>>> everyone should use it, then we'd better implement it without MR
> > >>>>> dependency, like DLS.
> > >>>>>
> > >>>>> Thanks.
> > >>>>>
> > >>>>> 2016-09-23 10:11 GMT+08:00 张铎 <pa...@gmail.com>:
> > >>>>>
> > >>>>>> I‘m -1 on let master or rs launch MR jobs. It is OK that some of
> our
> > >>>>>> features depend on MR but I think the bottom line is that we
> should
> > >>>> launch
> > >>>>>> the jobs from outside manually or by other services.
> > >>>>>>
> > >>>>>> 2016-09-23 9:47 GMT+08:00 Andrew Purtell <
> andrew.purtell@gmail.com
> > >:
> > >>>>>>
> > >>>>>>> Ok, got it. Well "shelling out" is on the line I think, so a fair
> > >>>>>>> question.
> > >>>>>>>
> > >>>>>>> Can this be driven by a utility derived from Tool like our other
> MR
> > >>>> apps?
> > >>>>>>> The issue is needing the AccessController to decide if allowed?
> But
> > >>>> nothing
> > >>>>>>> prevents the user from running the job manually/independently,
> > right?
> > >>>>>>>
> > >>>>>>>> On Sep 22, 2016, at 3:44 PM, Matteo Bertozzi <
> > >>>> theo.bertozzi@gmail.com>
> > >>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>> just a remark. my query was not about tools using MR (everyone i
> > >>>> think
> > >>>>>>> is
> > >>>>>>>> ok with those).
> > >>>>>>>> the topic was about: "are we ok with running MR jobs from Master
> > >>> and
> > >>>> RSs
> > >>>>>>>> code?" since this will be the first time we do this
> > >>>>>>>>
> > >>>>>>>> Matteo
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>> On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das <
> > >>> ddas@hortonworks.com>
> > >>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>> Very much agree; for tools like ExportSnapshot / Backup /
> > Restore,
> > >>>> it's
> > >>>>>>>>> fine to be dependent on MR. MR is the right framework for such.
> > We
> > >>>>>>> should
> > >>>>>>>>> also do compactions using MR (just saying :) )
> > >>>>>>>>> ________________________________________
> > >>>>>>>>> From: Ted Yu <yu...@gmail.com>
> > >>>>>>>>> Sent: Thursday, September 22, 2016 2:00 PM
> > >>>>>>>>> To: dev@hbase.apache.org
> > >>>>>>>>> Subject: Re: [DISCUSSION] MR jobs started by Master or RS
> > >>>>>>>>>
> > >>>>>>>>> I agree - backup / restore is in the same category as import /
> > >>>> export.
> > >>>>>>>>>
> > >>>>>>>>> On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell <
> > >>>>>>> andrew.purtell@gmail.com>
> > >>>>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> Backup is extra tooling around core in my opinion. Like import
> > or
> > >>>>>>> export.
> > >>>>>>>>>> Or the optional MOB tool. It's fine.
> > >>>>>>>>>>
> > >>>>>>>>>>> On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi <
> > >>>> mbertozzi@apache.org>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>> What's the latest opinion around running MR jobs from hbase
> > >>>> (Master
> > >>>>>>> or
> > >>>>>>>>>> RS)?
> > >>>>>>>>>>>
> > >>>>>>>>>>> I remember in the past that there was discussion about not
> > >>> having
> > >>>> MR
> > >>>>>>>>> has
> > >>>>>>>>>>> direct dependency of hbase.
> > >>>>>>>>>>>
> > >>>>>>>>>>> I think some of discussion where around MOB that had a MR job
> > to
> > >>>>>>>>> compact,
> > >>>>>>>>>>> that later was transformed in a non-MR job to be merged, I
> > think
> > >>>> we
> > >>>>>>>>> had a
> > >>>>>>>>>>> similar discussion for log split/replay.
> > >>>>>>>>>>>
> > >>>>>>>>>>> the latest is the new Backup feature (HBASE-7912), that runs
> a
> > >>> MR
> > >>>> job
> > >>>>>>>>>> from
> > >>>>>>>>>>> the master to copy data or restore data.
> > >>>>>>>>>>> (backup is also "not really core" as in.. if you don't use
> > >>> backup
> > >>>>>>>>> you'll
> > >>>>>>>>>>> not end up running MR jobs, but this was probably true for
> MOB
> > >>> as
> > >>>> in
> > >>>>>>>>> "if
> > >>>>>>>>>>> you don't enable MOB you don't need MR")
> > >>>>>>>>>>>
> > >>>>>>>>>>> any thoughts? do we a rule that says "we don't want to have
> > >>> hbase
> > >>>> run
> > >>>>>>>>> MR
> > >>>>>>>>>>> jobs, only tool started manually by the user can do that". or
> > >>> can
> > >>>> we
> > >>>>>>>>>> start
> > >>>>>>>>>>> adding MR calls around without problems?
> > >>>
> >
>
Re: [DISCUSSION] MR jobs started by Master or RS
Posted by Devaraj Das <dd...@hortonworks.com>.
Guys, first off apologies for bringing in the topic of MR-based compactions.. But I was thinking more about the SpliceMachine approach of managing compactions in Spark where apparently they saw a lot of benefits. Apologies for giving you that sore throat Andrew; I really didn't mean to :-)
So on this issue, we have these on the plate:
0. Somehow not use MR but something like that
1. Run a standalone service other than master
2. Shell out from the master
I don't think we have a good answer to (0), and I don't think it's even worth the effort of trying to build something when MR is already there, and being used by HBase already for some operations.
On (1), we have to deal with a myriad of issues - HA of the server not being the least of them all. Security (kerberos authentication, another keytab to manage, etc. etc. etc.). IMO, that approach is DOA. Instead let's substitute that (1) with the HBase Master. I haven't seen any good reason why the HBase master shouldn't launch MR jobs if needed. It's not ideal; agreed.
Now before going to (2), let's see what are the benefits of running the backup/restore jobs from the master. I think Ted has summarized some of the issues that we need to take care of - basically, the master can keep track of running jobs, and should it fail, the backup master can continue keeping track of it (since the jobId would have been recorded in the proc WAL). The master can also do cleanup, etc. of failed backup/restore processes. Security is another issue - the job needs to run as 'hbase' since it owns the data. Having the master launch the job makes it get that privilege. In the (2) approach, it's hard to do some of the above management.
Guys, just to reiterate, the patch as such is ready from the overall design/arch point of view (maybe code review is still pending from Matteo). If in the future, we find better ways of doing this without using MR, we can certainly consider that. But IMO don't think we should block this patch from getting merged.
________________________________________
From: 张铎 <pa...@gmail.com>
Sent: Thursday, September 22, 2016 8:32 PM
To: dev@hbase.apache.org
Subject: Re: [DISCUSSION] MR jobs started by Master or RS
So what about a standalone service other than master? You can use your own
procedure store in that service?
2016-09-23 11:28 GMT+08:00 Ted Yu <yu...@gmail.com>:
> An earlier implementation was client driven.
>
> But with that approach, it is hard to resume if there is error midway.
> Using Procedure V2 makes the backup / restore more robust.
>
> Another consideration is for security. It is hard to enforce security (to
> be implemented) for client driven actions.
>
> Cheers
>
> > On Sep 22, 2016, at 8:15 PM, Andrew Purtell <an...@gmail.com>
> wrote:
> >
> > No, this misses Matteo's finer point, which is "shelling out" from the
> master directly to run MR is a first. Why not drive this with a utility
> derived from Tool?
> >
> > On Sep 22, 2016, at 7:57 PM, Vladimir Rodionov <vl...@gmail.com>
> wrote:
> >
> >>>> In our production cluster, it is a common case we just have HDFS and
> >>>> HBase deployed.
> >>>> If our Master/RS depend on MR framework (especially some features we
> >>>> have not used at all), it introduced another cost for maintain. I
> >>>> don't think it is a good idea.
> >>
> >> So , you are not backup users in this case. Many our customers have full
> >> stack deployed and
> >> want see backup to be a standard feature. Besides this, nothing will
> happen
> >> in your cluster
> >> if you won't be doing backups.
> >>
> >> This discussion (we do not want see M/R dependency) goes to nowhere. We
> >> asked already, at least twice, to suggest another framework (other than
> M/R)
> >> for bulk data copy with *conversion*. Still waiting for suggestions.
> >>
> >> -Vlad
> >>
> >>
> >>
> >>
> >>> On Thu, Sep 22, 2016 at 7:49 PM, Ted Yu <yu...@gmail.com> wrote:
> >>>
> >>> If MR framework is not deployed in the cluster, hbase still functions
> >>> normally (post merge).
> >>>
> >>> In terms of build time dependency, we have long been depending on
> >>> mapreduce. Take a look at ExportSnapshot.
> >>>
> >>> Cheers
> >>>
> >>> On Thu, Sep 22, 2016 at 7:42 PM, Heng Chen <he...@gmail.com>
> >>> wrote:
> >>>
> >>>> In our production cluster, it is a common case we just have HDFS and
> >>>> HBase deployed.
> >>>> If our Master/RS depend on MR framework (especially some features we
> >>>> have not used at all), it introduced another cost for maintain. I
> >>>> don't think it is a good idea.
> >>>>
> >>>> 2016-09-23 10:28 GMT+08:00 张铎 <pa...@gmail.com>:
> >>>>> To be specific, for example, our nice Backup/Restore feature, if we
> >>> think
> >>>>> this is not a core feature of HBase, then we could make it depend on
> >>> MR,
> >>>>> and start a standalone BackupManager instance that submits MR jobs to
> >>> do
> >>>>> periodical maintenance job. And if we think this is a core feature
> that
> >>>>> everyone should use it, then we'd better implement it without MR
> >>>>> dependency, like DLS.
> >>>>>
> >>>>> Thanks.
> >>>>>
> >>>>> 2016-09-23 10:11 GMT+08:00 张铎 <pa...@gmail.com>:
> >>>>>
> >>>>>> I‘m -1 on let master or rs launch MR jobs. It is OK that some of our
> >>>>>> features depend on MR but I think the bottom line is that we should
> >>>> launch
> >>>>>> the jobs from outside manually or by other services.
> >>>>>>
> >>>>>> 2016-09-23 9:47 GMT+08:00 Andrew Purtell <andrew.purtell@gmail.com
> >:
> >>>>>>
> >>>>>>> Ok, got it. Well "shelling out" is on the line I think, so a fair
> >>>>>>> question.
> >>>>>>>
> >>>>>>> Can this be driven by a utility derived from Tool like our other MR
> >>>> apps?
> >>>>>>> The issue is needing the AccessController to decide if allowed? But
> >>>> nothing
> >>>>>>> prevents the user from running the job manually/independently,
> right?
> >>>>>>>
> >>>>>>>> On Sep 22, 2016, at 3:44 PM, Matteo Bertozzi <
> >>>> theo.bertozzi@gmail.com>
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> just a remark. my query was not about tools using MR (everyone i
> >>>> think
> >>>>>>> is
> >>>>>>>> ok with those).
> >>>>>>>> the topic was about: "are we ok with running MR jobs from Master
> >>> and
> >>>> RSs
> >>>>>>>> code?" since this will be the first time we do this
> >>>>>>>>
> >>>>>>>> Matteo
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das <
> >>> ddas@hortonworks.com>
> >>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>> Very much agree; for tools like ExportSnapshot / Backup /
> Restore,
> >>>> it's
> >>>>>>>>> fine to be dependent on MR. MR is the right framework for such.
> We
> >>>>>>> should
> >>>>>>>>> also do compactions using MR (just saying :) )
> >>>>>>>>> ________________________________________
> >>>>>>>>> From: Ted Yu <yu...@gmail.com>
> >>>>>>>>> Sent: Thursday, September 22, 2016 2:00 PM
> >>>>>>>>> To: dev@hbase.apache.org
> >>>>>>>>> Subject: Re: [DISCUSSION] MR jobs started by Master or RS
> >>>>>>>>>
> >>>>>>>>> I agree - backup / restore is in the same category as import /
> >>>> export.
> >>>>>>>>>
> >>>>>>>>> On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell <
> >>>>>>> andrew.purtell@gmail.com>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Backup is extra tooling around core in my opinion. Like import
> or
> >>>>>>> export.
> >>>>>>>>>> Or the optional MOB tool. It's fine.
> >>>>>>>>>>
> >>>>>>>>>>> On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi <
> >>>> mbertozzi@apache.org>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> What's the latest opinion around running MR jobs from hbase
> >>>> (Master
> >>>>>>> or
> >>>>>>>>>> RS)?
> >>>>>>>>>>>
> >>>>>>>>>>> I remember in the past that there was discussion about not
> >>> having
> >>>> MR
> >>>>>>>>> has
> >>>>>>>>>>> direct dependency of hbase.
> >>>>>>>>>>>
> >>>>>>>>>>> I think some of discussion where around MOB that had a MR job
> to
> >>>>>>>>> compact,
> >>>>>>>>>>> that later was transformed in a non-MR job to be merged, I
> think
> >>>> we
> >>>>>>>>> had a
> >>>>>>>>>>> similar discussion for log split/replay.
> >>>>>>>>>>>
> >>>>>>>>>>> the latest is the new Backup feature (HBASE-7912), that runs a
> >>> MR
> >>>> job
> >>>>>>>>>> from
> >>>>>>>>>>> the master to copy data or restore data.
> >>>>>>>>>>> (backup is also "not really core" as in.. if you don't use
> >>> backup
> >>>>>>>>> you'll
> >>>>>>>>>>> not end up running MR jobs, but this was probably true for MOB
> >>> as
> >>>> in
> >>>>>>>>> "if
> >>>>>>>>>>> you don't enable MOB you don't need MR")
> >>>>>>>>>>>
> >>>>>>>>>>> any thoughts? do we a rule that says "we don't want to have
> >>> hbase
> >>>> run
> >>>>>>>>> MR
> >>>>>>>>>>> jobs, only tool started manually by the user can do that". or
> >>> can
> >>>> we
> >>>>>>>>>> start
> >>>>>>>>>>> adding MR calls around without problems?
> >>>
>
Re: [DISCUSSION] MR jobs started by Master or RS
Posted by 张铎 <pa...@gmail.com>.
So what about a standalone service other than master? You can use your own
procedure store in that service?
2016-09-23 11:28 GMT+08:00 Ted Yu <yu...@gmail.com>:
> An earlier implementation was client driven.
>
> But with that approach, it is hard to resume if there is error midway.
> Using Procedure V2 makes the backup / restore more robust.
>
> Another consideration is for security. It is hard to enforce security (to
> be implemented) for client driven actions.
>
> Cheers
>
> > On Sep 22, 2016, at 8:15 PM, Andrew Purtell <an...@gmail.com>
> wrote:
> >
> > No, this misses Matteo's finer point, which is "shelling out" from the
> master directly to run MR is a first. Why not drive this with a utility
> derived from Tool?
> >
> > On Sep 22, 2016, at 7:57 PM, Vladimir Rodionov <vl...@gmail.com>
> wrote:
> >
> >>>> In our production cluster, it is a common case we just have HDFS and
> >>>> HBase deployed.
> >>>> If our Master/RS depend on MR framework (especially some features we
> >>>> have not used at all), it introduced another cost for maintain. I
> >>>> don't think it is a good idea.
> >>
> >> So , you are not backup users in this case. Many our customers have full
> >> stack deployed and
> >> want see backup to be a standard feature. Besides this, nothing will
> happen
> >> in your cluster
> >> if you won't be doing backups.
> >>
> >> This discussion (we do not want see M/R dependency) goes to nowhere. We
> >> asked already, at least twice, to suggest another framework (other than
> M/R)
> >> for bulk data copy with *conversion*. Still waiting for suggestions.
> >>
> >> -Vlad
> >>
> >>
> >>
> >>
> >>> On Thu, Sep 22, 2016 at 7:49 PM, Ted Yu <yu...@gmail.com> wrote:
> >>>
> >>> If MR framework is not deployed in the cluster, hbase still functions
> >>> normally (post merge).
> >>>
> >>> In terms of build time dependency, we have long been depending on
> >>> mapreduce. Take a look at ExportSnapshot.
> >>>
> >>> Cheers
> >>>
> >>> On Thu, Sep 22, 2016 at 7:42 PM, Heng Chen <he...@gmail.com>
> >>> wrote:
> >>>
> >>>> In our production cluster, it is a common case we just have HDFS and
> >>>> HBase deployed.
> >>>> If our Master/RS depend on MR framework (especially some features we
> >>>> have not used at all), it introduced another cost for maintain. I
> >>>> don't think it is a good idea.
> >>>>
> >>>> 2016-09-23 10:28 GMT+08:00 张铎 <pa...@gmail.com>:
> >>>>> To be specific, for example, our nice Backup/Restore feature, if we
> >>> think
> >>>>> this is not a core feature of HBase, then we could make it depend on
> >>> MR,
> >>>>> and start a standalone BackupManager instance that submits MR jobs to
> >>> do
> >>>>> periodical maintenance job. And if we think this is a core feature
> that
> >>>>> everyone should use it, then we'd better implement it without MR
> >>>>> dependency, like DLS.
> >>>>>
> >>>>> Thanks.
> >>>>>
> >>>>> 2016-09-23 10:11 GMT+08:00 张铎 <pa...@gmail.com>:
> >>>>>
> >>>>>> I‘m -1 on let master or rs launch MR jobs. It is OK that some of our
> >>>>>> features depend on MR but I think the bottom line is that we should
> >>>> launch
> >>>>>> the jobs from outside manually or by other services.
> >>>>>>
> >>>>>> 2016-09-23 9:47 GMT+08:00 Andrew Purtell <andrew.purtell@gmail.com
> >:
> >>>>>>
> >>>>>>> Ok, got it. Well "shelling out" is on the line I think, so a fair
> >>>>>>> question.
> >>>>>>>
> >>>>>>> Can this be driven by a utility derived from Tool like our other MR
> >>>> apps?
> >>>>>>> The issue is needing the AccessController to decide if allowed? But
> >>>> nothing
> >>>>>>> prevents the user from running the job manually/independently,
> right?
> >>>>>>>
> >>>>>>>> On Sep 22, 2016, at 3:44 PM, Matteo Bertozzi <
> >>>> theo.bertozzi@gmail.com>
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> just a remark. my query was not about tools using MR (everyone i
> >>>> think
> >>>>>>> is
> >>>>>>>> ok with those).
> >>>>>>>> the topic was about: "are we ok with running MR jobs from Master
> >>> and
> >>>> RSs
> >>>>>>>> code?" since this will be the first time we do this
> >>>>>>>>
> >>>>>>>> Matteo
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das <
> >>> ddas@hortonworks.com>
> >>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>> Very much agree; for tools like ExportSnapshot / Backup /
> Restore,
> >>>> it's
> >>>>>>>>> fine to be dependent on MR. MR is the right framework for such.
> We
> >>>>>>> should
> >>>>>>>>> also do compactions using MR (just saying :) )
> >>>>>>>>> ________________________________________
> >>>>>>>>> From: Ted Yu <yu...@gmail.com>
> >>>>>>>>> Sent: Thursday, September 22, 2016 2:00 PM
> >>>>>>>>> To: dev@hbase.apache.org
> >>>>>>>>> Subject: Re: [DISCUSSION] MR jobs started by Master or RS
> >>>>>>>>>
> >>>>>>>>> I agree - backup / restore is in the same category as import /
> >>>> export.
> >>>>>>>>>
> >>>>>>>>> On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell <
> >>>>>>> andrew.purtell@gmail.com>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Backup is extra tooling around core in my opinion. Like import
> or
> >>>>>>> export.
> >>>>>>>>>> Or the optional MOB tool. It's fine.
> >>>>>>>>>>
> >>>>>>>>>>> On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi <
> >>>> mbertozzi@apache.org>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> What's the latest opinion around running MR jobs from hbase
> >>>> (Master
> >>>>>>> or
> >>>>>>>>>> RS)?
> >>>>>>>>>>>
> >>>>>>>>>>> I remember in the past that there was discussion about not
> >>> having
> >>>> MR
> >>>>>>>>> has
> >>>>>>>>>>> direct dependency of hbase.
> >>>>>>>>>>>
> >>>>>>>>>>> I think some of discussion where around MOB that had a MR job
> to
> >>>>>>>>> compact,
> >>>>>>>>>>> that later was transformed in a non-MR job to be merged, I
> think
> >>>> we
> >>>>>>>>> had a
> >>>>>>>>>>> similar discussion for log split/replay.
> >>>>>>>>>>>
> >>>>>>>>>>> the latest is the new Backup feature (HBASE-7912), that runs a
> >>> MR
> >>>> job
> >>>>>>>>>> from
> >>>>>>>>>>> the master to copy data or restore data.
> >>>>>>>>>>> (backup is also "not really core" as in.. if you don't use
> >>> backup
> >>>>>>>>> you'll
> >>>>>>>>>>> not end up running MR jobs, but this was probably true for MOB
> >>> as
> >>>> in
> >>>>>>>>> "if
> >>>>>>>>>>> you don't enable MOB you don't need MR")
> >>>>>>>>>>>
> >>>>>>>>>>> any thoughts? do we a rule that says "we don't want to have
> >>> hbase
> >>>> run
> >>>>>>>>> MR
> >>>>>>>>>>> jobs, only tool started manually by the user can do that". or
> >>> can
> >>>> we
> >>>>>>>>>> start
> >>>>>>>>>>> adding MR calls around without problems?
> >>>
>
Re: [DISCUSSION] MR jobs started by Master or RS
Posted by Ted Yu <yu...@gmail.com>.
An earlier implementation was client driven.
But with that approach, it is hard to resume if there is error midway.
Using Procedure V2 makes the backup / restore more robust.
Another consideration is for security. It is hard to enforce security (to be implemented) for client driven actions.
Cheers
> On Sep 22, 2016, at 8:15 PM, Andrew Purtell <an...@gmail.com> wrote:
>
> No, this misses Matteo's finer point, which is "shelling out" from the master directly to run MR is a first. Why not drive this with a utility derived from Tool?
>
> On Sep 22, 2016, at 7:57 PM, Vladimir Rodionov <vl...@gmail.com> wrote:
>
>>>> In our production cluster, it is a common case we just have HDFS and
>>>> HBase deployed.
>>>> If our Master/RS depend on MR framework (especially some features we
>>>> have not used at all), it introduced another cost for maintain. I
>>>> don't think it is a good idea.
>>
>> So , you are not backup users in this case. Many our customers have full
>> stack deployed and
>> want see backup to be a standard feature. Besides this, nothing will happen
>> in your cluster
>> if you won't be doing backups.
>>
>> This discussion (we do not want see M/R dependency) goes to nowhere. We
>> asked already, at least twice, to suggest another framework (other than M/R)
>> for bulk data copy with *conversion*. Still waiting for suggestions.
>>
>> -Vlad
>>
>>
>>
>>
>>> On Thu, Sep 22, 2016 at 7:49 PM, Ted Yu <yu...@gmail.com> wrote:
>>>
>>> If MR framework is not deployed in the cluster, hbase still functions
>>> normally (post merge).
>>>
>>> In terms of build time dependency, we have long been depending on
>>> mapreduce. Take a look at ExportSnapshot.
>>>
>>> Cheers
>>>
>>> On Thu, Sep 22, 2016 at 7:42 PM, Heng Chen <he...@gmail.com>
>>> wrote:
>>>
>>>> In our production cluster, it is a common case we just have HDFS and
>>>> HBase deployed.
>>>> If our Master/RS depend on MR framework (especially some features we
>>>> have not used at all), it introduced another cost for maintain. I
>>>> don't think it is a good idea.
>>>>
>>>> 2016-09-23 10:28 GMT+08:00 张铎 <pa...@gmail.com>:
>>>>> To be specific, for example, our nice Backup/Restore feature, if we
>>> think
>>>>> this is not a core feature of HBase, then we could make it depend on
>>> MR,
>>>>> and start a standalone BackupManager instance that submits MR jobs to
>>> do
>>>>> periodical maintenance job. And if we think this is a core feature that
>>>>> everyone should use it, then we'd better implement it without MR
>>>>> dependency, like DLS.
>>>>>
>>>>> Thanks.
>>>>>
>>>>> 2016-09-23 10:11 GMT+08:00 张铎 <pa...@gmail.com>:
>>>>>
>>>>>> I‘m -1 on let master or rs launch MR jobs. It is OK that some of our
>>>>>> features depend on MR but I think the bottom line is that we should
>>>> launch
>>>>>> the jobs from outside manually or by other services.
>>>>>>
>>>>>> 2016-09-23 9:47 GMT+08:00 Andrew Purtell <an...@gmail.com>:
>>>>>>
>>>>>>> Ok, got it. Well "shelling out" is on the line I think, so a fair
>>>>>>> question.
>>>>>>>
>>>>>>> Can this be driven by a utility derived from Tool like our other MR
>>>> apps?
>>>>>>> The issue is needing the AccessController to decide if allowed? But
>>>> nothing
>>>>>>> prevents the user from running the job manually/independently, right?
>>>>>>>
>>>>>>>> On Sep 22, 2016, at 3:44 PM, Matteo Bertozzi <
>>>> theo.bertozzi@gmail.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> just a remark. my query was not about tools using MR (everyone i
>>>> think
>>>>>>> is
>>>>>>>> ok with those).
>>>>>>>> the topic was about: "are we ok with running MR jobs from Master
>>> and
>>>> RSs
>>>>>>>> code?" since this will be the first time we do this
>>>>>>>>
>>>>>>>> Matteo
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das <
>>> ddas@hortonworks.com>
>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Very much agree; for tools like ExportSnapshot / Backup / Restore,
>>>> it's
>>>>>>>>> fine to be dependent on MR. MR is the right framework for such. We
>>>>>>> should
>>>>>>>>> also do compactions using MR (just saying :) )
>>>>>>>>> ________________________________________
>>>>>>>>> From: Ted Yu <yu...@gmail.com>
>>>>>>>>> Sent: Thursday, September 22, 2016 2:00 PM
>>>>>>>>> To: dev@hbase.apache.org
>>>>>>>>> Subject: Re: [DISCUSSION] MR jobs started by Master or RS
>>>>>>>>>
>>>>>>>>> I agree - backup / restore is in the same category as import /
>>>> export.
>>>>>>>>>
>>>>>>>>> On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell <
>>>>>>> andrew.purtell@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Backup is extra tooling around core in my opinion. Like import or
>>>>>>> export.
>>>>>>>>>> Or the optional MOB tool. It's fine.
>>>>>>>>>>
>>>>>>>>>>> On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi <
>>>> mbertozzi@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> What's the latest opinion around running MR jobs from hbase
>>>> (Master
>>>>>>> or
>>>>>>>>>> RS)?
>>>>>>>>>>>
>>>>>>>>>>> I remember in the past that there was discussion about not
>>> having
>>>> MR
>>>>>>>>> has
>>>>>>>>>>> direct dependency of hbase.
>>>>>>>>>>>
>>>>>>>>>>> I think some of discussion where around MOB that had a MR job to
>>>>>>>>> compact,
>>>>>>>>>>> that later was transformed in a non-MR job to be merged, I think
>>>> we
>>>>>>>>> had a
>>>>>>>>>>> similar discussion for log split/replay.
>>>>>>>>>>>
>>>>>>>>>>> the latest is the new Backup feature (HBASE-7912), that runs a
>>> MR
>>>> job
>>>>>>>>>> from
>>>>>>>>>>> the master to copy data or restore data.
>>>>>>>>>>> (backup is also "not really core" as in.. if you don't use
>>> backup
>>>>>>>>> you'll
>>>>>>>>>>> not end up running MR jobs, but this was probably true for MOB
>>> as
>>>> in
>>>>>>>>> "if
>>>>>>>>>>> you don't enable MOB you don't need MR")
>>>>>>>>>>>
>>>>>>>>>>> any thoughts? do we a rule that says "we don't want to have
>>> hbase
>>>> run
>>>>>>>>> MR
>>>>>>>>>>> jobs, only tool started manually by the user can do that". or
>>> can
>>>> we
>>>>>>>>>> start
>>>>>>>>>>> adding MR calls around without problems?
>>>
Re: [DISCUSSION] MR jobs started by Master or RS
Posted by Andrew Purtell <an...@gmail.com>.
No, this misses Matteo's finer point, which is "shelling out" from the master directly to run MR is a first. Why not drive this with a utility derived from Tool?
On Sep 22, 2016, at 7:57 PM, Vladimir Rodionov <vl...@gmail.com> wrote:
>>> In our production cluster, it is a common case we just have HDFS and
>>> HBase deployed.
>>> If our Master/RS depend on MR framework (especially some features we
>>> have not used at all), it introduced another cost for maintain. I
>>> don't think it is a good idea.
>
> So , you are not backup users in this case. Many our customers have full
> stack deployed and
> want see backup to be a standard feature. Besides this, nothing will happen
> in your cluster
> if you won't be doing backups.
>
> This discussion (we do not want see M/R dependency) goes to nowhere. We
> asked already, at least twice, to suggest another framework (other than M/R)
> for bulk data copy with *conversion*. Still waiting for suggestions.
>
> -Vlad
>
>
>
>
>> On Thu, Sep 22, 2016 at 7:49 PM, Ted Yu <yu...@gmail.com> wrote:
>>
>> If MR framework is not deployed in the cluster, hbase still functions
>> normally (post merge).
>>
>> In terms of build time dependency, we have long been depending on
>> mapreduce. Take a look at ExportSnapshot.
>>
>> Cheers
>>
>> On Thu, Sep 22, 2016 at 7:42 PM, Heng Chen <he...@gmail.com>
>> wrote:
>>
>>> In our production cluster, it is a common case we just have HDFS and
>>> HBase deployed.
>>> If our Master/RS depend on MR framework (especially some features we
>>> have not used at all), it introduced another cost for maintain. I
>>> don't think it is a good idea.
>>>
>>> 2016-09-23 10:28 GMT+08:00 张铎 <pa...@gmail.com>:
>>>> To be specific, for example, our nice Backup/Restore feature, if we
>> think
>>>> this is not a core feature of HBase, then we could make it depend on
>> MR,
>>>> and start a standalone BackupManager instance that submits MR jobs to
>> do
>>>> periodical maintenance job. And if we think this is a core feature that
>>>> everyone should use it, then we'd better implement it without MR
>>>> dependency, like DLS.
>>>>
>>>> Thanks.
>>>>
>>>> 2016-09-23 10:11 GMT+08:00 张铎 <pa...@gmail.com>:
>>>>
>>>>> I‘m -1 on let master or rs launch MR jobs. It is OK that some of our
>>>>> features depend on MR but I think the bottom line is that we should
>>> launch
>>>>> the jobs from outside manually or by other services.
>>>>>
>>>>> 2016-09-23 9:47 GMT+08:00 Andrew Purtell <an...@gmail.com>:
>>>>>
>>>>>> Ok, got it. Well "shelling out" is on the line I think, so a fair
>>>>>> question.
>>>>>>
>>>>>> Can this be driven by a utility derived from Tool like our other MR
>>> apps?
>>>>>> The issue is needing the AccessController to decide if allowed? But
>>> nothing
>>>>>> prevents the user from running the job manually/independently, right?
>>>>>>
>>>>>>> On Sep 22, 2016, at 3:44 PM, Matteo Bertozzi <
>>> theo.bertozzi@gmail.com>
>>>>>> wrote:
>>>>>>>
>>>>>>> just a remark. my query was not about tools using MR (everyone i
>>> think
>>>>>> is
>>>>>>> ok with those).
>>>>>>> the topic was about: "are we ok with running MR jobs from Master
>> and
>>> RSs
>>>>>>> code?" since this will be the first time we do this
>>>>>>>
>>>>>>> Matteo
>>>>>>>
>>>>>>>
>>>>>>>> On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das <
>> ddas@hortonworks.com>
>>>>>> wrote:
>>>>>>>>
>>>>>>>> Very much agree; for tools like ExportSnapshot / Backup / Restore,
>>> it's
>>>>>>>> fine to be dependent on MR. MR is the right framework for such. We
>>>>>> should
>>>>>>>> also do compactions using MR (just saying :) )
>>>>>>>> ________________________________________
>>>>>>>> From: Ted Yu <yu...@gmail.com>
>>>>>>>> Sent: Thursday, September 22, 2016 2:00 PM
>>>>>>>> To: dev@hbase.apache.org
>>>>>>>> Subject: Re: [DISCUSSION] MR jobs started by Master or RS
>>>>>>>>
>>>>>>>> I agree - backup / restore is in the same category as import /
>>> export.
>>>>>>>>
>>>>>>>> On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell <
>>>>>> andrew.purtell@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Backup is extra tooling around core in my opinion. Like import or
>>>>>> export.
>>>>>>>>> Or the optional MOB tool. It's fine.
>>>>>>>>>
>>>>>>>>>> On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi <
>>> mbertozzi@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> What's the latest opinion around running MR jobs from hbase
>>> (Master
>>>>>> or
>>>>>>>>> RS)?
>>>>>>>>>>
>>>>>>>>>> I remember in the past that there was discussion about not
>> having
>>> MR
>>>>>>>> has
>>>>>>>>>> direct dependency of hbase.
>>>>>>>>>>
>>>>>>>>>> I think some of discussion where around MOB that had a MR job to
>>>>>>>> compact,
>>>>>>>>>> that later was transformed in a non-MR job to be merged, I think
>>> we
>>>>>>>> had a
>>>>>>>>>> similar discussion for log split/replay.
>>>>>>>>>>
>>>>>>>>>> the latest is the new Backup feature (HBASE-7912), that runs a
>> MR
>>> job
>>>>>>>>> from
>>>>>>>>>> the master to copy data or restore data.
>>>>>>>>>> (backup is also "not really core" as in.. if you don't use
>> backup
>>>>>>>> you'll
>>>>>>>>>> not end up running MR jobs, but this was probably true for MOB
>> as
>>> in
>>>>>>>> "if
>>>>>>>>>> you don't enable MOB you don't need MR")
>>>>>>>>>>
>>>>>>>>>> any thoughts? do we a rule that says "we don't want to have
>> hbase
>>> run
>>>>>>>> MR
>>>>>>>>>> jobs, only tool started manually by the user can do that". or
>> can
>>> we
>>>>>>>>> start
>>>>>>>>>> adding MR calls around without problems?
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>
>>
Re: [DISCUSSION] MR jobs started by Master or RS
Posted by Vladimir Rodionov <vl...@gmail.com>.
>> In our production cluster, it is a common case we just have HDFS and
>> HBase deployed.
>> If our Master/RS depend on MR framework (especially some features we
>> have not used at all), it introduced another cost for maintain. I
>> don't think it is a good idea.
So , you are not backup users in this case. Many our customers have full
stack deployed and
want see backup to be a standard feature. Besides this, nothing will happen
in your cluster
if you won't be doing backups.
This discussion (we do not want see M/R dependency) goes to nowhere. We
asked already, at least twice, to suggest another framework (other than M/R)
for bulk data copy with *conversion*. Still waiting for suggestions.
-Vlad
On Thu, Sep 22, 2016 at 7:49 PM, Ted Yu <yu...@gmail.com> wrote:
> If MR framework is not deployed in the cluster, hbase still functions
> normally (post merge).
>
> In terms of build time dependency, we have long been depending on
> mapreduce. Take a look at ExportSnapshot.
>
> Cheers
>
> On Thu, Sep 22, 2016 at 7:42 PM, Heng Chen <he...@gmail.com>
> wrote:
>
> > In our production cluster, it is a common case we just have HDFS and
> > HBase deployed.
> > If our Master/RS depend on MR framework (especially some features we
> > have not used at all), it introduced another cost for maintain. I
> > don't think it is a good idea.
> >
> > 2016-09-23 10:28 GMT+08:00 张铎 <pa...@gmail.com>:
> > > To be specific, for example, our nice Backup/Restore feature, if we
> think
> > > this is not a core feature of HBase, then we could make it depend on
> MR,
> > > and start a standalone BackupManager instance that submits MR jobs to
> do
> > > periodical maintenance job. And if we think this is a core feature that
> > > everyone should use it, then we'd better implement it without MR
> > > dependency, like DLS.
> > >
> > > Thanks.
> > >
> > > 2016-09-23 10:11 GMT+08:00 张铎 <pa...@gmail.com>:
> > >
> > >> I‘m -1 on let master or rs launch MR jobs. It is OK that some of our
> > >> features depend on MR but I think the bottom line is that we should
> > launch
> > >> the jobs from outside manually or by other services.
> > >>
> > >> 2016-09-23 9:47 GMT+08:00 Andrew Purtell <an...@gmail.com>:
> > >>
> > >>> Ok, got it. Well "shelling out" is on the line I think, so a fair
> > >>> question.
> > >>>
> > >>> Can this be driven by a utility derived from Tool like our other MR
> > apps?
> > >>> The issue is needing the AccessController to decide if allowed? But
> > nothing
> > >>> prevents the user from running the job manually/independently, right?
> > >>>
> > >>> > On Sep 22, 2016, at 3:44 PM, Matteo Bertozzi <
> > theo.bertozzi@gmail.com>
> > >>> wrote:
> > >>> >
> > >>> > just a remark. my query was not about tools using MR (everyone i
> > think
> > >>> is
> > >>> > ok with those).
> > >>> > the topic was about: "are we ok with running MR jobs from Master
> and
> > RSs
> > >>> > code?" since this will be the first time we do this
> > >>> >
> > >>> > Matteo
> > >>> >
> > >>> >
> > >>> >> On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das <
> ddas@hortonworks.com>
> > >>> wrote:
> > >>> >>
> > >>> >> Very much agree; for tools like ExportSnapshot / Backup / Restore,
> > it's
> > >>> >> fine to be dependent on MR. MR is the right framework for such. We
> > >>> should
> > >>> >> also do compactions using MR (just saying :) )
> > >>> >> ________________________________________
> > >>> >> From: Ted Yu <yu...@gmail.com>
> > >>> >> Sent: Thursday, September 22, 2016 2:00 PM
> > >>> >> To: dev@hbase.apache.org
> > >>> >> Subject: Re: [DISCUSSION] MR jobs started by Master or RS
> > >>> >>
> > >>> >> I agree - backup / restore is in the same category as import /
> > export.
> > >>> >>
> > >>> >> On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell <
> > >>> andrew.purtell@gmail.com>
> > >>> >> wrote:
> > >>> >>
> > >>> >>> Backup is extra tooling around core in my opinion. Like import or
> > >>> export.
> > >>> >>> Or the optional MOB tool. It's fine.
> > >>> >>>
> > >>> >>>> On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi <
> > mbertozzi@apache.org>
> > >>> >>> wrote:
> > >>> >>>>
> > >>> >>>> What's the latest opinion around running MR jobs from hbase
> > (Master
> > >>> or
> > >>> >>> RS)?
> > >>> >>>>
> > >>> >>>> I remember in the past that there was discussion about not
> having
> > MR
> > >>> >> has
> > >>> >>>> direct dependency of hbase.
> > >>> >>>>
> > >>> >>>> I think some of discussion where around MOB that had a MR job to
> > >>> >> compact,
> > >>> >>>> that later was transformed in a non-MR job to be merged, I think
> > we
> > >>> >> had a
> > >>> >>>> similar discussion for log split/replay.
> > >>> >>>>
> > >>> >>>> the latest is the new Backup feature (HBASE-7912), that runs a
> MR
> > job
> > >>> >>> from
> > >>> >>>> the master to copy data or restore data.
> > >>> >>>> (backup is also "not really core" as in.. if you don't use
> backup
> > >>> >> you'll
> > >>> >>>> not end up running MR jobs, but this was probably true for MOB
> as
> > in
> > >>> >> "if
> > >>> >>>> you don't enable MOB you don't need MR")
> > >>> >>>>
> > >>> >>>> any thoughts? do we a rule that says "we don't want to have
> hbase
> > run
> > >>> >> MR
> > >>> >>>> jobs, only tool started manually by the user can do that". or
> can
> > we
> > >>> >>> start
> > >>> >>>> adding MR calls around without problems?
> > >>> >>>
> > >>> >>
> > >>>
> > >>
> > >>
> >
>
Re: [DISCUSSION] MR jobs started by Master or RS
Posted by Ted Yu <yu...@gmail.com>.
If MR framework is not deployed in the cluster, hbase still functions
normally (post merge).
In terms of build time dependency, we have long been depending on
mapreduce. Take a look at ExportSnapshot.
Cheers
On Thu, Sep 22, 2016 at 7:42 PM, Heng Chen <he...@gmail.com> wrote:
> In our production cluster, it is a common case we just have HDFS and
> HBase deployed.
> If our Master/RS depend on MR framework (especially some features we
> have not used at all), it introduced another cost for maintain. I
> don't think it is a good idea.
>
> 2016-09-23 10:28 GMT+08:00 张铎 <pa...@gmail.com>:
> > To be specific, for example, our nice Backup/Restore feature, if we think
> > this is not a core feature of HBase, then we could make it depend on MR,
> > and start a standalone BackupManager instance that submits MR jobs to do
> > periodical maintenance job. And if we think this is a core feature that
> > everyone should use it, then we'd better implement it without MR
> > dependency, like DLS.
> >
> > Thanks.
> >
> > 2016-09-23 10:11 GMT+08:00 张铎 <pa...@gmail.com>:
> >
> >> I‘m -1 on let master or rs launch MR jobs. It is OK that some of our
> >> features depend on MR but I think the bottom line is that we should
> launch
> >> the jobs from outside manually or by other services.
> >>
> >> 2016-09-23 9:47 GMT+08:00 Andrew Purtell <an...@gmail.com>:
> >>
> >>> Ok, got it. Well "shelling out" is on the line I think, so a fair
> >>> question.
> >>>
> >>> Can this be driven by a utility derived from Tool like our other MR
> apps?
> >>> The issue is needing the AccessController to decide if allowed? But
> nothing
> >>> prevents the user from running the job manually/independently, right?
> >>>
> >>> > On Sep 22, 2016, at 3:44 PM, Matteo Bertozzi <
> theo.bertozzi@gmail.com>
> >>> wrote:
> >>> >
> >>> > just a remark. my query was not about tools using MR (everyone i
> think
> >>> is
> >>> > ok with those).
> >>> > the topic was about: "are we ok with running MR jobs from Master and
> RSs
> >>> > code?" since this will be the first time we do this
> >>> >
> >>> > Matteo
> >>> >
> >>> >
> >>> >> On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das <dd...@hortonworks.com>
> >>> wrote:
> >>> >>
> >>> >> Very much agree; for tools like ExportSnapshot / Backup / Restore,
> it's
> >>> >> fine to be dependent on MR. MR is the right framework for such. We
> >>> should
> >>> >> also do compactions using MR (just saying :) )
> >>> >> ________________________________________
> >>> >> From: Ted Yu <yu...@gmail.com>
> >>> >> Sent: Thursday, September 22, 2016 2:00 PM
> >>> >> To: dev@hbase.apache.org
> >>> >> Subject: Re: [DISCUSSION] MR jobs started by Master or RS
> >>> >>
> >>> >> I agree - backup / restore is in the same category as import /
> export.
> >>> >>
> >>> >> On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell <
> >>> andrew.purtell@gmail.com>
> >>> >> wrote:
> >>> >>
> >>> >>> Backup is extra tooling around core in my opinion. Like import or
> >>> export.
> >>> >>> Or the optional MOB tool. It's fine.
> >>> >>>
> >>> >>>> On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi <
> mbertozzi@apache.org>
> >>> >>> wrote:
> >>> >>>>
> >>> >>>> What's the latest opinion around running MR jobs from hbase
> (Master
> >>> or
> >>> >>> RS)?
> >>> >>>>
> >>> >>>> I remember in the past that there was discussion about not having
> MR
> >>> >> has
> >>> >>>> direct dependency of hbase.
> >>> >>>>
> >>> >>>> I think some of discussion where around MOB that had a MR job to
> >>> >> compact,
> >>> >>>> that later was transformed in a non-MR job to be merged, I think
> we
> >>> >> had a
> >>> >>>> similar discussion for log split/replay.
> >>> >>>>
> >>> >>>> the latest is the new Backup feature (HBASE-7912), that runs a MR
> job
> >>> >>> from
> >>> >>>> the master to copy data or restore data.
> >>> >>>> (backup is also "not really core" as in.. if you don't use backup
> >>> >> you'll
> >>> >>>> not end up running MR jobs, but this was probably true for MOB as
> in
> >>> >> "if
> >>> >>>> you don't enable MOB you don't need MR")
> >>> >>>>
> >>> >>>> any thoughts? do we a rule that says "we don't want to have hbase
> run
> >>> >> MR
> >>> >>>> jobs, only tool started manually by the user can do that". or can
> we
> >>> >>> start
> >>> >>>> adding MR calls around without problems?
> >>> >>>
> >>> >>
> >>>
> >>
> >>
>
Re: [DISCUSSION] MR jobs started by Master or RS
Posted by Heng Chen <he...@gmail.com>.
In our production cluster, it is a common case we just have HDFS and
HBase deployed.
If our Master/RS depend on MR framework (especially some features we
have not used at all), it introduced another cost for maintain. I
don't think it is a good idea.
2016-09-23 10:28 GMT+08:00 张铎 <pa...@gmail.com>:
> To be specific, for example, our nice Backup/Restore feature, if we think
> this is not a core feature of HBase, then we could make it depend on MR,
> and start a standalone BackupManager instance that submits MR jobs to do
> periodical maintenance job. And if we think this is a core feature that
> everyone should use it, then we'd better implement it without MR
> dependency, like DLS.
>
> Thanks.
>
> 2016-09-23 10:11 GMT+08:00 张铎 <pa...@gmail.com>:
>
>> I‘m -1 on let master or rs launch MR jobs. It is OK that some of our
>> features depend on MR but I think the bottom line is that we should launch
>> the jobs from outside manually or by other services.
>>
>> 2016-09-23 9:47 GMT+08:00 Andrew Purtell <an...@gmail.com>:
>>
>>> Ok, got it. Well "shelling out" is on the line I think, so a fair
>>> question.
>>>
>>> Can this be driven by a utility derived from Tool like our other MR apps?
>>> The issue is needing the AccessController to decide if allowed? But nothing
>>> prevents the user from running the job manually/independently, right?
>>>
>>> > On Sep 22, 2016, at 3:44 PM, Matteo Bertozzi <th...@gmail.com>
>>> wrote:
>>> >
>>> > just a remark. my query was not about tools using MR (everyone i think
>>> is
>>> > ok with those).
>>> > the topic was about: "are we ok with running MR jobs from Master and RSs
>>> > code?" since this will be the first time we do this
>>> >
>>> > Matteo
>>> >
>>> >
>>> >> On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das <dd...@hortonworks.com>
>>> wrote:
>>> >>
>>> >> Very much agree; for tools like ExportSnapshot / Backup / Restore, it's
>>> >> fine to be dependent on MR. MR is the right framework for such. We
>>> should
>>> >> also do compactions using MR (just saying :) )
>>> >> ________________________________________
>>> >> From: Ted Yu <yu...@gmail.com>
>>> >> Sent: Thursday, September 22, 2016 2:00 PM
>>> >> To: dev@hbase.apache.org
>>> >> Subject: Re: [DISCUSSION] MR jobs started by Master or RS
>>> >>
>>> >> I agree - backup / restore is in the same category as import / export.
>>> >>
>>> >> On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell <
>>> andrew.purtell@gmail.com>
>>> >> wrote:
>>> >>
>>> >>> Backup is extra tooling around core in my opinion. Like import or
>>> export.
>>> >>> Or the optional MOB tool. It's fine.
>>> >>>
>>> >>>> On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi <mb...@apache.org>
>>> >>> wrote:
>>> >>>>
>>> >>>> What's the latest opinion around running MR jobs from hbase (Master
>>> or
>>> >>> RS)?
>>> >>>>
>>> >>>> I remember in the past that there was discussion about not having MR
>>> >> has
>>> >>>> direct dependency of hbase.
>>> >>>>
>>> >>>> I think some of discussion where around MOB that had a MR job to
>>> >> compact,
>>> >>>> that later was transformed in a non-MR job to be merged, I think we
>>> >> had a
>>> >>>> similar discussion for log split/replay.
>>> >>>>
>>> >>>> the latest is the new Backup feature (HBASE-7912), that runs a MR job
>>> >>> from
>>> >>>> the master to copy data or restore data.
>>> >>>> (backup is also "not really core" as in.. if you don't use backup
>>> >> you'll
>>> >>>> not end up running MR jobs, but this was probably true for MOB as in
>>> >> "if
>>> >>>> you don't enable MOB you don't need MR")
>>> >>>>
>>> >>>> any thoughts? do we a rule that says "we don't want to have hbase run
>>> >> MR
>>> >>>> jobs, only tool started manually by the user can do that". or can we
>>> >>> start
>>> >>>> adding MR calls around without problems?
>>> >>>
>>> >>
>>>
>>
>>
Re: [DISCUSSION] MR jobs started by Master or RS
Posted by 张铎 <pa...@gmail.com>.
To be specific, for example, our nice Backup/Restore feature, if we think
this is not a core feature of HBase, then we could make it depend on MR,
and start a standalone BackupManager instance that submits MR jobs to do
periodical maintenance job. And if we think this is a core feature that
everyone should use it, then we'd better implement it without MR
dependency, like DLS.
Thanks.
2016-09-23 10:11 GMT+08:00 张铎 <pa...@gmail.com>:
> I‘m -1 on let master or rs launch MR jobs. It is OK that some of our
> features depend on MR but I think the bottom line is that we should launch
> the jobs from outside manually or by other services.
>
> 2016-09-23 9:47 GMT+08:00 Andrew Purtell <an...@gmail.com>:
>
>> Ok, got it. Well "shelling out" is on the line I think, so a fair
>> question.
>>
>> Can this be driven by a utility derived from Tool like our other MR apps?
>> The issue is needing the AccessController to decide if allowed? But nothing
>> prevents the user from running the job manually/independently, right?
>>
>> > On Sep 22, 2016, at 3:44 PM, Matteo Bertozzi <th...@gmail.com>
>> wrote:
>> >
>> > just a remark. my query was not about tools using MR (everyone i think
>> is
>> > ok with those).
>> > the topic was about: "are we ok with running MR jobs from Master and RSs
>> > code?" since this will be the first time we do this
>> >
>> > Matteo
>> >
>> >
>> >> On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das <dd...@hortonworks.com>
>> wrote:
>> >>
>> >> Very much agree; for tools like ExportSnapshot / Backup / Restore, it's
>> >> fine to be dependent on MR. MR is the right framework for such. We
>> should
>> >> also do compactions using MR (just saying :) )
>> >> ________________________________________
>> >> From: Ted Yu <yu...@gmail.com>
>> >> Sent: Thursday, September 22, 2016 2:00 PM
>> >> To: dev@hbase.apache.org
>> >> Subject: Re: [DISCUSSION] MR jobs started by Master or RS
>> >>
>> >> I agree - backup / restore is in the same category as import / export.
>> >>
>> >> On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell <
>> andrew.purtell@gmail.com>
>> >> wrote:
>> >>
>> >>> Backup is extra tooling around core in my opinion. Like import or
>> export.
>> >>> Or the optional MOB tool. It's fine.
>> >>>
>> >>>> On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi <mb...@apache.org>
>> >>> wrote:
>> >>>>
>> >>>> What's the latest opinion around running MR jobs from hbase (Master
>> or
>> >>> RS)?
>> >>>>
>> >>>> I remember in the past that there was discussion about not having MR
>> >> has
>> >>>> direct dependency of hbase.
>> >>>>
>> >>>> I think some of discussion where around MOB that had a MR job to
>> >> compact,
>> >>>> that later was transformed in a non-MR job to be merged, I think we
>> >> had a
>> >>>> similar discussion for log split/replay.
>> >>>>
>> >>>> the latest is the new Backup feature (HBASE-7912), that runs a MR job
>> >>> from
>> >>>> the master to copy data or restore data.
>> >>>> (backup is also "not really core" as in.. if you don't use backup
>> >> you'll
>> >>>> not end up running MR jobs, but this was probably true for MOB as in
>> >> "if
>> >>>> you don't enable MOB you don't need MR")
>> >>>>
>> >>>> any thoughts? do we a rule that says "we don't want to have hbase run
>> >> MR
>> >>>> jobs, only tool started manually by the user can do that". or can we
>> >>> start
>> >>>> adding MR calls around without problems?
>> >>>
>> >>
>>
>
>
Re: [DISCUSSION] MR jobs started by Master or RS
Posted by 张铎 <pa...@gmail.com>.
I‘m -1 on let master or rs launch MR jobs. It is OK that some of our
features depend on MR but I think the bottom line is that we should launch
the jobs from outside manually or by other services.
2016-09-23 9:47 GMT+08:00 Andrew Purtell <an...@gmail.com>:
> Ok, got it. Well "shelling out" is on the line I think, so a fair question.
>
> Can this be driven by a utility derived from Tool like our other MR apps?
> The issue is needing the AccessController to decide if allowed? But nothing
> prevents the user from running the job manually/independently, right?
>
> > On Sep 22, 2016, at 3:44 PM, Matteo Bertozzi <th...@gmail.com>
> wrote:
> >
> > just a remark. my query was not about tools using MR (everyone i think is
> > ok with those).
> > the topic was about: "are we ok with running MR jobs from Master and RSs
> > code?" since this will be the first time we do this
> >
> > Matteo
> >
> >
> >> On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das <dd...@hortonworks.com>
> wrote:
> >>
> >> Very much agree; for tools like ExportSnapshot / Backup / Restore, it's
> >> fine to be dependent on MR. MR is the right framework for such. We
> should
> >> also do compactions using MR (just saying :) )
> >> ________________________________________
> >> From: Ted Yu <yu...@gmail.com>
> >> Sent: Thursday, September 22, 2016 2:00 PM
> >> To: dev@hbase.apache.org
> >> Subject: Re: [DISCUSSION] MR jobs started by Master or RS
> >>
> >> I agree - backup / restore is in the same category as import / export.
> >>
> >> On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell <
> andrew.purtell@gmail.com>
> >> wrote:
> >>
> >>> Backup is extra tooling around core in my opinion. Like import or
> export.
> >>> Or the optional MOB tool. It's fine.
> >>>
> >>>> On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi <mb...@apache.org>
> >>> wrote:
> >>>>
> >>>> What's the latest opinion around running MR jobs from hbase (Master or
> >>> RS)?
> >>>>
> >>>> I remember in the past that there was discussion about not having MR
> >> has
> >>>> direct dependency of hbase.
> >>>>
> >>>> I think some of discussion where around MOB that had a MR job to
> >> compact,
> >>>> that later was transformed in a non-MR job to be merged, I think we
> >> had a
> >>>> similar discussion for log split/replay.
> >>>>
> >>>> the latest is the new Backup feature (HBASE-7912), that runs a MR job
> >>> from
> >>>> the master to copy data or restore data.
> >>>> (backup is also "not really core" as in.. if you don't use backup
> >> you'll
> >>>> not end up running MR jobs, but this was probably true for MOB as in
> >> "if
> >>>> you don't enable MOB you don't need MR")
> >>>>
> >>>> any thoughts? do we a rule that says "we don't want to have hbase run
> >> MR
> >>>> jobs, only tool started manually by the user can do that". or can we
> >>> start
> >>>> adding MR calls around without problems?
> >>>
> >>
>
Re: [DISCUSSION] MR jobs started by Master or RS
Posted by Andrew Purtell <an...@gmail.com>.
Ok, got it. Well "shelling out" is on the line I think, so a fair question.
Can this be driven by a utility derived from Tool like our other MR apps? The issue is needing the AccessController to decide if allowed? But nothing prevents the user from running the job manually/independently, right?
> On Sep 22, 2016, at 3:44 PM, Matteo Bertozzi <th...@gmail.com> wrote:
>
> just a remark. my query was not about tools using MR (everyone i think is
> ok with those).
> the topic was about: "are we ok with running MR jobs from Master and RSs
> code?" since this will be the first time we do this
>
> Matteo
>
>
>> On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das <dd...@hortonworks.com> wrote:
>>
>> Very much agree; for tools like ExportSnapshot / Backup / Restore, it's
>> fine to be dependent on MR. MR is the right framework for such. We should
>> also do compactions using MR (just saying :) )
>> ________________________________________
>> From: Ted Yu <yu...@gmail.com>
>> Sent: Thursday, September 22, 2016 2:00 PM
>> To: dev@hbase.apache.org
>> Subject: Re: [DISCUSSION] MR jobs started by Master or RS
>>
>> I agree - backup / restore is in the same category as import / export.
>>
>> On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell <an...@gmail.com>
>> wrote:
>>
>>> Backup is extra tooling around core in my opinion. Like import or export.
>>> Or the optional MOB tool. It's fine.
>>>
>>>> On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi <mb...@apache.org>
>>> wrote:
>>>>
>>>> What's the latest opinion around running MR jobs from hbase (Master or
>>> RS)?
>>>>
>>>> I remember in the past that there was discussion about not having MR
>> has
>>>> direct dependency of hbase.
>>>>
>>>> I think some of discussion where around MOB that had a MR job to
>> compact,
>>>> that later was transformed in a non-MR job to be merged, I think we
>> had a
>>>> similar discussion for log split/replay.
>>>>
>>>> the latest is the new Backup feature (HBASE-7912), that runs a MR job
>>> from
>>>> the master to copy data or restore data.
>>>> (backup is also "not really core" as in.. if you don't use backup
>> you'll
>>>> not end up running MR jobs, but this was probably true for MOB as in
>> "if
>>>> you don't enable MOB you don't need MR")
>>>>
>>>> any thoughts? do we a rule that says "we don't want to have hbase run
>> MR
>>>> jobs, only tool started manually by the user can do that". or can we
>>> start
>>>> adding MR calls around without problems?
>>>
>>
Re: [DISCUSSION] MR jobs started by Master or RS
Posted by Enis Söztutar <en...@apache.org>.
Once you are in the game of coordinating large scale tasks with
distribution, fault tolerance, etc other than implementing a similar
framework inside HBase, MR will be the way to go. Things like exporting
snapshots, dist cp, or backups (which uses these) must use such a
framework.
The issue about master launching MR jobs came in the review around that
time, and we concluded that it was fine since backups by definition require
such a framework.
Enis
On Thu, Sep 22, 2016 at 4:32 PM, Devaraj Das <dd...@hortonworks.com> wrote:
> Not practical to do those tools without MR, JM. We should be using the
> right framework for the use cases in hand. MR fits this really well.
> JM, when you say "if we can do without MR, then, why not?", do you have a
> framework in mind that performs/scale as well as MR? Curious.
> ________________________________________
> From: Jean-Marc Spaggiari <je...@spaggiari.org>
> Sent: Thursday, September 22, 2016 4:29 PM
> To: dev
> Subject: Re: [DISCUSSION] MR jobs started by Master or RS
>
> Well, I'm just not using those features ;) But was hopping for the MOBs ;)
> My point is, if we can do it without MR, then, why not? )
>
> 2016-09-22 19:25 GMT-04:00 Vladimir Rodionov <vl...@gmail.com>:
>
> > Forgot WALPlayer :)
> >
> > -Vlad
> >
> > On Thu, Sep 22, 2016 at 4:21 PM, Vladimir Rodionov <
> vladrodionov@gmail.com
> > >
> > wrote:
> >
> > > >> and
> > > >> backups too, but don't want to bother having to install and
> configure
> > > YARN
> > > >> just for that, as well as removing resources from HBase to give it
> to
> > >
> > > Any suggestions on how to do bulk data move with transformation from/to
> > > HBase cluster w/o MapReduce?
> > >
> > > Opposition to M/R does not make sense imo, as since we have a lot of
> > tools
> > > in HBase which depend on MapReduce:
> > >
> > > CountRows
> > > CountCells
> > > Import
> > > Export
> > > ImportTsv
> > > ExportTsv
> > > CopyTable
> > > VerifyReplication
> > > ExportSnapshot
> > >
> > > and new backup create/restore of course.
> > >
> > >
> > > -Vlad
> > >
> > >
> > >
> > >
> > > On Thu, Sep 22, 2016 at 4:15 PM, Jean-Marc Spaggiari <
> > > jean-marc@spaggiari.org> wrote:
> > >
> > >> My 2¢: I have a strong preference for NOT having a dependency on MR
> > >> anywhere :( I run my HBase cluste without YARN. Just HBase and HDFS. I
> > >> like
> > >> all the features that we built. Would love to be able to use MOBs and
> > >> backups too, but don't want to bother having to install and configure
> > YARN
> > >> just for that, as well as removing resources from HBase to give it to
> > >> yarn....
> > >>
> > >> JMS
> > >>
> > >> 2016-09-22 18:44 GMT-04:00 Matteo Bertozzi <th...@gmail.com>:
> > >>
> > >> > just a remark. my query was not about tools using MR (everyone i
> think
> > >> is
> > >> > ok with those).
> > >> > the topic was about: "are we ok with running MR jobs from Master and
> > RSs
> > >> > code?" since this will be the first time we do this
> > >> >
> > >> > Matteo
> > >> >
> > >> >
> > >> > On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das <dd...@hortonworks.com>
> > >> wrote:
> > >> >
> > >> > > Very much agree; for tools like ExportSnapshot / Backup / Restore,
> > >> it's
> > >> > > fine to be dependent on MR. MR is the right framework for such. We
> > >> should
> > >> > > also do compactions using MR (just saying :) )
> > >> > > ________________________________________
> > >> > > From: Ted Yu <yu...@gmail.com>
> > >> > > Sent: Thursday, September 22, 2016 2:00 PM
> > >> > > To: dev@hbase.apache.org
> > >> > > Subject: Re: [DISCUSSION] MR jobs started by Master or RS
> > >> > >
> > >> > > I agree - backup / restore is in the same category as import /
> > export.
> > >> > >
> > >> > > On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell <
> > >> > andrew.purtell@gmail.com>
> > >> > > wrote:
> > >> > >
> > >> > > > Backup is extra tooling around core in my opinion. Like import
> or
> > >> > export.
> > >> > > > Or the optional MOB tool. It's fine.
> > >> > > >
> > >> > > > > On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi <
> > >> mbertozzi@apache.org>
> > >> > > > wrote:
> > >> > > > >
> > >> > > > > What's the latest opinion around running MR jobs from hbase
> > >> (Master
> > >> > or
> > >> > > > RS)?
> > >> > > > >
> > >> > > > > I remember in the past that there was discussion about not
> > having
> > >> MR
> > >> > > has
> > >> > > > > direct dependency of hbase.
> > >> > > > >
> > >> > > > > I think some of discussion where around MOB that had a MR job
> to
> > >> > > compact,
> > >> > > > > that later was transformed in a non-MR job to be merged, I
> think
> > >> we
> > >> > > had a
> > >> > > > > similar discussion for log split/replay.
> > >> > > > >
> > >> > > > > the latest is the new Backup feature (HBASE-7912), that runs a
> > MR
> > >> job
> > >> > > > from
> > >> > > > > the master to copy data or restore data.
> > >> > > > > (backup is also "not really core" as in.. if you don't use
> > backup
> > >> > > you'll
> > >> > > > > not end up running MR jobs, but this was probably true for MOB
> > as
> > >> in
> > >> > > "if
> > >> > > > > you don't enable MOB you don't need MR")
> > >> > > > >
> > >> > > > > any thoughts? do we a rule that says "we don't want to have
> > hbase
> > >> run
> > >> > > MR
> > >> > > > > jobs, only tool started manually by the user can do that". or
> > can
> > >> we
> > >> > > > start
> > >> > > > > adding MR calls around without problems?
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>
Re: [DISCUSSION] MR jobs started by Master or RS
Posted by Devaraj Das <dd...@hortonworks.com>.
Not practical to do those tools without MR, JM. We should be using the right framework for the use cases in hand. MR fits this really well.
JM, when you say "if we can do without MR, then, why not?", do you have a framework in mind that performs/scale as well as MR? Curious.
________________________________________
From: Jean-Marc Spaggiari <je...@spaggiari.org>
Sent: Thursday, September 22, 2016 4:29 PM
To: dev
Subject: Re: [DISCUSSION] MR jobs started by Master or RS
Well, I'm just not using those features ;) But was hopping for the MOBs ;)
My point is, if we can do it without MR, then, why not? )
2016-09-22 19:25 GMT-04:00 Vladimir Rodionov <vl...@gmail.com>:
> Forgot WALPlayer :)
>
> -Vlad
>
> On Thu, Sep 22, 2016 at 4:21 PM, Vladimir Rodionov <vladrodionov@gmail.com
> >
> wrote:
>
> > >> and
> > >> backups too, but don't want to bother having to install and configure
> > YARN
> > >> just for that, as well as removing resources from HBase to give it to
> >
> > Any suggestions on how to do bulk data move with transformation from/to
> > HBase cluster w/o MapReduce?
> >
> > Opposition to M/R does not make sense imo, as since we have a lot of
> tools
> > in HBase which depend on MapReduce:
> >
> > CountRows
> > CountCells
> > Import
> > Export
> > ImportTsv
> > ExportTsv
> > CopyTable
> > VerifyReplication
> > ExportSnapshot
> >
> > and new backup create/restore of course.
> >
> >
> > -Vlad
> >
> >
> >
> >
> > On Thu, Sep 22, 2016 at 4:15 PM, Jean-Marc Spaggiari <
> > jean-marc@spaggiari.org> wrote:
> >
> >> My 2¢: I have a strong preference for NOT having a dependency on MR
> >> anywhere :( I run my HBase cluste without YARN. Just HBase and HDFS. I
> >> like
> >> all the features that we built. Would love to be able to use MOBs and
> >> backups too, but don't want to bother having to install and configure
> YARN
> >> just for that, as well as removing resources from HBase to give it to
> >> yarn....
> >>
> >> JMS
> >>
> >> 2016-09-22 18:44 GMT-04:00 Matteo Bertozzi <th...@gmail.com>:
> >>
> >> > just a remark. my query was not about tools using MR (everyone i think
> >> is
> >> > ok with those).
> >> > the topic was about: "are we ok with running MR jobs from Master and
> RSs
> >> > code?" since this will be the first time we do this
> >> >
> >> > Matteo
> >> >
> >> >
> >> > On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das <dd...@hortonworks.com>
> >> wrote:
> >> >
> >> > > Very much agree; for tools like ExportSnapshot / Backup / Restore,
> >> it's
> >> > > fine to be dependent on MR. MR is the right framework for such. We
> >> should
> >> > > also do compactions using MR (just saying :) )
> >> > > ________________________________________
> >> > > From: Ted Yu <yu...@gmail.com>
> >> > > Sent: Thursday, September 22, 2016 2:00 PM
> >> > > To: dev@hbase.apache.org
> >> > > Subject: Re: [DISCUSSION] MR jobs started by Master or RS
> >> > >
> >> > > I agree - backup / restore is in the same category as import /
> export.
> >> > >
> >> > > On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell <
> >> > andrew.purtell@gmail.com>
> >> > > wrote:
> >> > >
> >> > > > Backup is extra tooling around core in my opinion. Like import or
> >> > export.
> >> > > > Or the optional MOB tool. It's fine.
> >> > > >
> >> > > > > On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi <
> >> mbertozzi@apache.org>
> >> > > > wrote:
> >> > > > >
> >> > > > > What's the latest opinion around running MR jobs from hbase
> >> (Master
> >> > or
> >> > > > RS)?
> >> > > > >
> >> > > > > I remember in the past that there was discussion about not
> having
> >> MR
> >> > > has
> >> > > > > direct dependency of hbase.
> >> > > > >
> >> > > > > I think some of discussion where around MOB that had a MR job to
> >> > > compact,
> >> > > > > that later was transformed in a non-MR job to be merged, I think
> >> we
> >> > > had a
> >> > > > > similar discussion for log split/replay.
> >> > > > >
> >> > > > > the latest is the new Backup feature (HBASE-7912), that runs a
> MR
> >> job
> >> > > > from
> >> > > > > the master to copy data or restore data.
> >> > > > > (backup is also "not really core" as in.. if you don't use
> backup
> >> > > you'll
> >> > > > > not end up running MR jobs, but this was probably true for MOB
> as
> >> in
> >> > > "if
> >> > > > > you don't enable MOB you don't need MR")
> >> > > > >
> >> > > > > any thoughts? do we a rule that says "we don't want to have
> hbase
> >> run
> >> > > MR
> >> > > > > jobs, only tool started manually by the user can do that". or
> can
> >> we
> >> > > > start
> >> > > > > adding MR calls around without problems?
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>
Re: [DISCUSSION] MR jobs started by Master or RS
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Well, I'm just not using those features ;) But was hopping for the MOBs ;)
My point is, if we can do it without MR, then, why not? )
2016-09-22 19:25 GMT-04:00 Vladimir Rodionov <vl...@gmail.com>:
> Forgot WALPlayer :)
>
> -Vlad
>
> On Thu, Sep 22, 2016 at 4:21 PM, Vladimir Rodionov <vladrodionov@gmail.com
> >
> wrote:
>
> > >> and
> > >> backups too, but don't want to bother having to install and configure
> > YARN
> > >> just for that, as well as removing resources from HBase to give it to
> >
> > Any suggestions on how to do bulk data move with transformation from/to
> > HBase cluster w/o MapReduce?
> >
> > Opposition to M/R does not make sense imo, as since we have a lot of
> tools
> > in HBase which depend on MapReduce:
> >
> > CountRows
> > CountCells
> > Import
> > Export
> > ImportTsv
> > ExportTsv
> > CopyTable
> > VerifyReplication
> > ExportSnapshot
> >
> > and new backup create/restore of course.
> >
> >
> > -Vlad
> >
> >
> >
> >
> > On Thu, Sep 22, 2016 at 4:15 PM, Jean-Marc Spaggiari <
> > jean-marc@spaggiari.org> wrote:
> >
> >> My 2¢: I have a strong preference for NOT having a dependency on MR
> >> anywhere :( I run my HBase cluste without YARN. Just HBase and HDFS. I
> >> like
> >> all the features that we built. Would love to be able to use MOBs and
> >> backups too, but don't want to bother having to install and configure
> YARN
> >> just for that, as well as removing resources from HBase to give it to
> >> yarn....
> >>
> >> JMS
> >>
> >> 2016-09-22 18:44 GMT-04:00 Matteo Bertozzi <th...@gmail.com>:
> >>
> >> > just a remark. my query was not about tools using MR (everyone i think
> >> is
> >> > ok with those).
> >> > the topic was about: "are we ok with running MR jobs from Master and
> RSs
> >> > code?" since this will be the first time we do this
> >> >
> >> > Matteo
> >> >
> >> >
> >> > On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das <dd...@hortonworks.com>
> >> wrote:
> >> >
> >> > > Very much agree; for tools like ExportSnapshot / Backup / Restore,
> >> it's
> >> > > fine to be dependent on MR. MR is the right framework for such. We
> >> should
> >> > > also do compactions using MR (just saying :) )
> >> > > ________________________________________
> >> > > From: Ted Yu <yu...@gmail.com>
> >> > > Sent: Thursday, September 22, 2016 2:00 PM
> >> > > To: dev@hbase.apache.org
> >> > > Subject: Re: [DISCUSSION] MR jobs started by Master or RS
> >> > >
> >> > > I agree - backup / restore is in the same category as import /
> export.
> >> > >
> >> > > On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell <
> >> > andrew.purtell@gmail.com>
> >> > > wrote:
> >> > >
> >> > > > Backup is extra tooling around core in my opinion. Like import or
> >> > export.
> >> > > > Or the optional MOB tool. It's fine.
> >> > > >
> >> > > > > On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi <
> >> mbertozzi@apache.org>
> >> > > > wrote:
> >> > > > >
> >> > > > > What's the latest opinion around running MR jobs from hbase
> >> (Master
> >> > or
> >> > > > RS)?
> >> > > > >
> >> > > > > I remember in the past that there was discussion about not
> having
> >> MR
> >> > > has
> >> > > > > direct dependency of hbase.
> >> > > > >
> >> > > > > I think some of discussion where around MOB that had a MR job to
> >> > > compact,
> >> > > > > that later was transformed in a non-MR job to be merged, I think
> >> we
> >> > > had a
> >> > > > > similar discussion for log split/replay.
> >> > > > >
> >> > > > > the latest is the new Backup feature (HBASE-7912), that runs a
> MR
> >> job
> >> > > > from
> >> > > > > the master to copy data or restore data.
> >> > > > > (backup is also "not really core" as in.. if you don't use
> backup
> >> > > you'll
> >> > > > > not end up running MR jobs, but this was probably true for MOB
> as
> >> in
> >> > > "if
> >> > > > > you don't enable MOB you don't need MR")
> >> > > > >
> >> > > > > any thoughts? do we a rule that says "we don't want to have
> hbase
> >> run
> >> > > MR
> >> > > > > jobs, only tool started manually by the user can do that". or
> can
> >> we
> >> > > > start
> >> > > > > adding MR calls around without problems?
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>
Re: [DISCUSSION] MR jobs started by Master or RS
Posted by Vladimir Rodionov <vl...@gmail.com>.
Forgot WALPlayer :)
-Vlad
On Thu, Sep 22, 2016 at 4:21 PM, Vladimir Rodionov <vl...@gmail.com>
wrote:
> >> and
> >> backups too, but don't want to bother having to install and configure
> YARN
> >> just for that, as well as removing resources from HBase to give it to
>
> Any suggestions on how to do bulk data move with transformation from/to
> HBase cluster w/o MapReduce?
>
> Opposition to M/R does not make sense imo, as since we have a lot of tools
> in HBase which depend on MapReduce:
>
> CountRows
> CountCells
> Import
> Export
> ImportTsv
> ExportTsv
> CopyTable
> VerifyReplication
> ExportSnapshot
>
> and new backup create/restore of course.
>
>
> -Vlad
>
>
>
>
> On Thu, Sep 22, 2016 at 4:15 PM, Jean-Marc Spaggiari <
> jean-marc@spaggiari.org> wrote:
>
>> My 2¢: I have a strong preference for NOT having a dependency on MR
>> anywhere :( I run my HBase cluste without YARN. Just HBase and HDFS. I
>> like
>> all the features that we built. Would love to be able to use MOBs and
>> backups too, but don't want to bother having to install and configure YARN
>> just for that, as well as removing resources from HBase to give it to
>> yarn....
>>
>> JMS
>>
>> 2016-09-22 18:44 GMT-04:00 Matteo Bertozzi <th...@gmail.com>:
>>
>> > just a remark. my query was not about tools using MR (everyone i think
>> is
>> > ok with those).
>> > the topic was about: "are we ok with running MR jobs from Master and RSs
>> > code?" since this will be the first time we do this
>> >
>> > Matteo
>> >
>> >
>> > On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das <dd...@hortonworks.com>
>> wrote:
>> >
>> > > Very much agree; for tools like ExportSnapshot / Backup / Restore,
>> it's
>> > > fine to be dependent on MR. MR is the right framework for such. We
>> should
>> > > also do compactions using MR (just saying :) )
>> > > ________________________________________
>> > > From: Ted Yu <yu...@gmail.com>
>> > > Sent: Thursday, September 22, 2016 2:00 PM
>> > > To: dev@hbase.apache.org
>> > > Subject: Re: [DISCUSSION] MR jobs started by Master or RS
>> > >
>> > > I agree - backup / restore is in the same category as import / export.
>> > >
>> > > On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell <
>> > andrew.purtell@gmail.com>
>> > > wrote:
>> > >
>> > > > Backup is extra tooling around core in my opinion. Like import or
>> > export.
>> > > > Or the optional MOB tool. It's fine.
>> > > >
>> > > > > On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi <
>> mbertozzi@apache.org>
>> > > > wrote:
>> > > > >
>> > > > > What's the latest opinion around running MR jobs from hbase
>> (Master
>> > or
>> > > > RS)?
>> > > > >
>> > > > > I remember in the past that there was discussion about not having
>> MR
>> > > has
>> > > > > direct dependency of hbase.
>> > > > >
>> > > > > I think some of discussion where around MOB that had a MR job to
>> > > compact,
>> > > > > that later was transformed in a non-MR job to be merged, I think
>> we
>> > > had a
>> > > > > similar discussion for log split/replay.
>> > > > >
>> > > > > the latest is the new Backup feature (HBASE-7912), that runs a MR
>> job
>> > > > from
>> > > > > the master to copy data or restore data.
>> > > > > (backup is also "not really core" as in.. if you don't use backup
>> > > you'll
>> > > > > not end up running MR jobs, but this was probably true for MOB as
>> in
>> > > "if
>> > > > > you don't enable MOB you don't need MR")
>> > > > >
>> > > > > any thoughts? do we a rule that says "we don't want to have hbase
>> run
>> > > MR
>> > > > > jobs, only tool started manually by the user can do that". or can
>> we
>> > > > start
>> > > > > adding MR calls around without problems?
>> > > >
>> > >
>> >
>>
>
>
Re: [DISCUSSION] MR jobs started by Master or RS
Posted by Vladimir Rodionov <vl...@gmail.com>.
>> and
>> backups too, but don't want to bother having to install and configure
YARN
>> just for that, as well as removing resources from HBase to give it to
Any suggestions on how to do bulk data move with transformation from/to
HBase cluster w/o MapReduce?
Opposition to M/R does not make sense imo, as since we have a lot of tools
in HBase which depend on MapReduce:
CountRows
CountCells
Import
Export
ImportTsv
ExportTsv
CopyTable
VerifyReplication
ExportSnapshot
and new backup create/restore of course.
-Vlad
On Thu, Sep 22, 2016 at 4:15 PM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:
> My 2¢: I have a strong preference for NOT having a dependency on MR
> anywhere :( I run my HBase cluste without YARN. Just HBase and HDFS. I like
> all the features that we built. Would love to be able to use MOBs and
> backups too, but don't want to bother having to install and configure YARN
> just for that, as well as removing resources from HBase to give it to
> yarn....
>
> JMS
>
> 2016-09-22 18:44 GMT-04:00 Matteo Bertozzi <th...@gmail.com>:
>
> > just a remark. my query was not about tools using MR (everyone i think is
> > ok with those).
> > the topic was about: "are we ok with running MR jobs from Master and RSs
> > code?" since this will be the first time we do this
> >
> > Matteo
> >
> >
> > On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das <dd...@hortonworks.com>
> wrote:
> >
> > > Very much agree; for tools like ExportSnapshot / Backup / Restore, it's
> > > fine to be dependent on MR. MR is the right framework for such. We
> should
> > > also do compactions using MR (just saying :) )
> > > ________________________________________
> > > From: Ted Yu <yu...@gmail.com>
> > > Sent: Thursday, September 22, 2016 2:00 PM
> > > To: dev@hbase.apache.org
> > > Subject: Re: [DISCUSSION] MR jobs started by Master or RS
> > >
> > > I agree - backup / restore is in the same category as import / export.
> > >
> > > On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell <
> > andrew.purtell@gmail.com>
> > > wrote:
> > >
> > > > Backup is extra tooling around core in my opinion. Like import or
> > export.
> > > > Or the optional MOB tool. It's fine.
> > > >
> > > > > On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi <mbertozzi@apache.org
> >
> > > > wrote:
> > > > >
> > > > > What's the latest opinion around running MR jobs from hbase (Master
> > or
> > > > RS)?
> > > > >
> > > > > I remember in the past that there was discussion about not having
> MR
> > > has
> > > > > direct dependency of hbase.
> > > > >
> > > > > I think some of discussion where around MOB that had a MR job to
> > > compact,
> > > > > that later was transformed in a non-MR job to be merged, I think we
> > > had a
> > > > > similar discussion for log split/replay.
> > > > >
> > > > > the latest is the new Backup feature (HBASE-7912), that runs a MR
> job
> > > > from
> > > > > the master to copy data or restore data.
> > > > > (backup is also "not really core" as in.. if you don't use backup
> > > you'll
> > > > > not end up running MR jobs, but this was probably true for MOB as
> in
> > > "if
> > > > > you don't enable MOB you don't need MR")
> > > > >
> > > > > any thoughts? do we a rule that says "we don't want to have hbase
> run
> > > MR
> > > > > jobs, only tool started manually by the user can do that". or can
> we
> > > > start
> > > > > adding MR calls around without problems?
> > > >
> > >
> >
>
Re: [DISCUSSION] MR jobs started by Master or RS
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
My 2¢: I have a strong preference for NOT having a dependency on MR
anywhere :( I run my HBase cluste without YARN. Just HBase and HDFS. I like
all the features that we built. Would love to be able to use MOBs and
backups too, but don't want to bother having to install and configure YARN
just for that, as well as removing resources from HBase to give it to
yarn....
JMS
2016-09-22 18:44 GMT-04:00 Matteo Bertozzi <th...@gmail.com>:
> just a remark. my query was not about tools using MR (everyone i think is
> ok with those).
> the topic was about: "are we ok with running MR jobs from Master and RSs
> code?" since this will be the first time we do this
>
> Matteo
>
>
> On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das <dd...@hortonworks.com> wrote:
>
> > Very much agree; for tools like ExportSnapshot / Backup / Restore, it's
> > fine to be dependent on MR. MR is the right framework for such. We should
> > also do compactions using MR (just saying :) )
> > ________________________________________
> > From: Ted Yu <yu...@gmail.com>
> > Sent: Thursday, September 22, 2016 2:00 PM
> > To: dev@hbase.apache.org
> > Subject: Re: [DISCUSSION] MR jobs started by Master or RS
> >
> > I agree - backup / restore is in the same category as import / export.
> >
> > On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell <
> andrew.purtell@gmail.com>
> > wrote:
> >
> > > Backup is extra tooling around core in my opinion. Like import or
> export.
> > > Or the optional MOB tool. It's fine.
> > >
> > > > On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi <mb...@apache.org>
> > > wrote:
> > > >
> > > > What's the latest opinion around running MR jobs from hbase (Master
> or
> > > RS)?
> > > >
> > > > I remember in the past that there was discussion about not having MR
> > has
> > > > direct dependency of hbase.
> > > >
> > > > I think some of discussion where around MOB that had a MR job to
> > compact,
> > > > that later was transformed in a non-MR job to be merged, I think we
> > had a
> > > > similar discussion for log split/replay.
> > > >
> > > > the latest is the new Backup feature (HBASE-7912), that runs a MR job
> > > from
> > > > the master to copy data or restore data.
> > > > (backup is also "not really core" as in.. if you don't use backup
> > you'll
> > > > not end up running MR jobs, but this was probably true for MOB as in
> > "if
> > > > you don't enable MOB you don't need MR")
> > > >
> > > > any thoughts? do we a rule that says "we don't want to have hbase run
> > MR
> > > > jobs, only tool started manually by the user can do that". or can we
> > > start
> > > > adding MR calls around without problems?
> > >
> >
>
Re: [DISCUSSION] MR jobs started by Master or RS
Posted by Matteo Bertozzi <th...@gmail.com>.
just a remark. my query was not about tools using MR (everyone i think is
ok with those).
the topic was about: "are we ok with running MR jobs from Master and RSs
code?" since this will be the first time we do this
Matteo
On Thu, Sep 22, 2016 at 2:49 PM, Devaraj Das <dd...@hortonworks.com> wrote:
> Very much agree; for tools like ExportSnapshot / Backup / Restore, it's
> fine to be dependent on MR. MR is the right framework for such. We should
> also do compactions using MR (just saying :) )
> ________________________________________
> From: Ted Yu <yu...@gmail.com>
> Sent: Thursday, September 22, 2016 2:00 PM
> To: dev@hbase.apache.org
> Subject: Re: [DISCUSSION] MR jobs started by Master or RS
>
> I agree - backup / restore is in the same category as import / export.
>
> On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell <an...@gmail.com>
> wrote:
>
> > Backup is extra tooling around core in my opinion. Like import or export.
> > Or the optional MOB tool. It's fine.
> >
> > > On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi <mb...@apache.org>
> > wrote:
> > >
> > > What's the latest opinion around running MR jobs from hbase (Master or
> > RS)?
> > >
> > > I remember in the past that there was discussion about not having MR
> has
> > > direct dependency of hbase.
> > >
> > > I think some of discussion where around MOB that had a MR job to
> compact,
> > > that later was transformed in a non-MR job to be merged, I think we
> had a
> > > similar discussion for log split/replay.
> > >
> > > the latest is the new Backup feature (HBASE-7912), that runs a MR job
> > from
> > > the master to copy data or restore data.
> > > (backup is also "not really core" as in.. if you don't use backup
> you'll
> > > not end up running MR jobs, but this was probably true for MOB as in
> "if
> > > you don't enable MOB you don't need MR")
> > >
> > > any thoughts? do we a rule that says "we don't want to have hbase run
> MR
> > > jobs, only tool started manually by the user can do that". or can we
> > start
> > > adding MR calls around without problems?
> >
>
Re: [DISCUSSION] MR jobs started by Master or RS
Posted by Devaraj Das <dd...@hortonworks.com>.
Very much agree; for tools like ExportSnapshot / Backup / Restore, it's fine to be dependent on MR. MR is the right framework for such. We should also do compactions using MR (just saying :) )
________________________________________
From: Ted Yu <yu...@gmail.com>
Sent: Thursday, September 22, 2016 2:00 PM
To: dev@hbase.apache.org
Subject: Re: [DISCUSSION] MR jobs started by Master or RS
I agree - backup / restore is in the same category as import / export.
On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell <an...@gmail.com>
wrote:
> Backup is extra tooling around core in my opinion. Like import or export.
> Or the optional MOB tool. It's fine.
>
> > On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi <mb...@apache.org>
> wrote:
> >
> > What's the latest opinion around running MR jobs from hbase (Master or
> RS)?
> >
> > I remember in the past that there was discussion about not having MR has
> > direct dependency of hbase.
> >
> > I think some of discussion where around MOB that had a MR job to compact,
> > that later was transformed in a non-MR job to be merged, I think we had a
> > similar discussion for log split/replay.
> >
> > the latest is the new Backup feature (HBASE-7912), that runs a MR job
> from
> > the master to copy data or restore data.
> > (backup is also "not really core" as in.. if you don't use backup you'll
> > not end up running MR jobs, but this was probably true for MOB as in "if
> > you don't enable MOB you don't need MR")
> >
> > any thoughts? do we a rule that says "we don't want to have hbase run MR
> > jobs, only tool started manually by the user can do that". or can we
> start
> > adding MR calls around without problems?
>
Re: [DISCUSSION] MR jobs started by Master or RS
Posted by Ted Yu <yu...@gmail.com>.
I agree - backup / restore is in the same category as import / export.
On Thu, Sep 22, 2016 at 1:58 PM, Andrew Purtell <an...@gmail.com>
wrote:
> Backup is extra tooling around core in my opinion. Like import or export.
> Or the optional MOB tool. It's fine.
>
> > On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi <mb...@apache.org>
> wrote:
> >
> > What's the latest opinion around running MR jobs from hbase (Master or
> RS)?
> >
> > I remember in the past that there was discussion about not having MR has
> > direct dependency of hbase.
> >
> > I think some of discussion where around MOB that had a MR job to compact,
> > that later was transformed in a non-MR job to be merged, I think we had a
> > similar discussion for log split/replay.
> >
> > the latest is the new Backup feature (HBASE-7912), that runs a MR job
> from
> > the master to copy data or restore data.
> > (backup is also "not really core" as in.. if you don't use backup you'll
> > not end up running MR jobs, but this was probably true for MOB as in "if
> > you don't enable MOB you don't need MR")
> >
> > any thoughts? do we a rule that says "we don't want to have hbase run MR
> > jobs, only tool started manually by the user can do that". or can we
> start
> > adding MR calls around without problems?
>
Re: [DISCUSSION] MR jobs started by Master or RS
Posted by Andrew Purtell <an...@gmail.com>.
Backup is extra tooling around core in my opinion. Like import or export. Or the optional MOB tool. It's fine.
> On Sep 22, 2016, at 1:50 PM, Matteo Bertozzi <mb...@apache.org> wrote:
>
> What's the latest opinion around running MR jobs from hbase (Master or RS)?
>
> I remember in the past that there was discussion about not having MR has
> direct dependency of hbase.
>
> I think some of discussion where around MOB that had a MR job to compact,
> that later was transformed in a non-MR job to be merged, I think we had a
> similar discussion for log split/replay.
>
> the latest is the new Backup feature (HBASE-7912), that runs a MR job from
> the master to copy data or restore data.
> (backup is also "not really core" as in.. if you don't use backup you'll
> not end up running MR jobs, but this was probably true for MOB as in "if
> you don't enable MOB you don't need MR")
>
> any thoughts? do we a rule that says "we don't want to have hbase run MR
> jobs, only tool started manually by the user can do that". or can we start
> adding MR calls around without problems?