Re: [DISCUSSION] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues

[Devaraj Das <dd...@hortonworks.com>]

BTW I just wanted to bring to everyone's attention the work done over in
HADOOP-9421. Sanjay Radia has been telling me that a corresponding work
should be done in HBase as well. That will help keep the compatibility
story straight when the work to do with pluggable authentication mechanisms
happening elsewhere in the ecosystem takes shape..

Any thoughts?

Thanks
Devaraj


On Mon, Jun 24, 2013 at 8:45 PM, Stack <st...@duboce.net> wrote:

> (Changed the subject)
>
> On Mon, Jun 24, 2013 at 4:04 PM, Nick Dimiduk <nd...@gmail.com> wrote:
>
> > I want to see initial data type APIs ship out with 0.95.2. A patch for
> > ordered byte serialization is up (HBASE-8201) and is nearing
> steady-state.
> > However, sershe is the only person who's left feedback. I just posted an
> > early patch for the data type API itself (HBASE-8693). It should get some
> > eyes from all manor of interested parties, but I'll settle for folk from
> > Phoenix for now.
> >
> >
> It would be cool if Phoenix and Kiji fellows and any one else interested
> would weigh in and take a look see.
>
> This does not strike me as something we should hold up the release for
> though.  It looks like something that could go in at any time?
>
>
>
> > Should these tasks be escalated to criticals in order to grab attention?
> >
> >
> I don't think that works going by past experience (and I don't think this a
> blocker on 0.96)
>
>
>
> > Additional comments inline.
> >
> >
> ...
>
>
>
> > Namespaces is the long pole and progress seems slow.  Do we hold up the
> > > release for them?  How can we hurry this effort along?  Swat team
> > descends
> > > on Y!?
> > >
> >
> > It would be a shame to not get a decision on this in for 0.96.
> >
> >
> Agree.  We need to get 0.96 out though.  It has been too long.
>
>
>
> > + Is anyone testing?  Integration tests fail on ec2 build from time to
> time
> > > [2].  Our Elliott dug in on one of the failures a few days back and
> found
> > > legit issue w/ no retry on admin tasks (I heart hbase-it tests).  Our
> > unit
> > > test story is better [3] but there are still the odd failures.
> > >
> >
> > With the creation of the new list, noticing these issues is going to push
> > further back-burner. Nannying this stuff should retain focus. I'll
> > volunteer to track on these issues as I see them.
> >
> >
> Thank you Nick.
>
>
>
> > Notice though that some of the more recent failures are caused by lack of
> > disk space on the Jenkins build host.
> >
> >
> Oh.  Missed that.  Let me dig in.
>
>
> St.Ack
>

Re: [DISCUSSION] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues

[Stack <st...@duboce.net>]

On Mon, Jul 1, 2013 at 1:40 PM, Devaraj Das <dd...@hortonworks.com> wrote:

> BTW I just wanted to bring to everyone's attention the work done over in
> HADOOP-9421. Sanjay Radia has been telling me that a corresponding work
> should be done in HBase as well. That will help keep the compatibility
> story straight when the work to do with pluggable authentication mechanisms
> happening elsewhere in the ecosystem takes shape..
>
> Any thoughts?
>

There is no description of what was finally decided on?

Looks like client initiates and then server and client can negotiate?

Can't we up the rpc version when we want to add support for negotiated
protocols?  If so, this does not have to hold up 0.96?

St.Ack

Re: [DISCUSSION] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues

[Andrew Purtell <ap...@apache.org>]

On Mon, Jul 1, 2013 at 2:54 PM, Devaraj Das <dd...@hortonworks.com> wrote:

> However doing the equivalent patch
> for HADOOP-9421 in HBase will probably be easier and doable for 0.96...
>

If someone shows up with a patch equivalent to HADOOP-9421 for HBase real
soon now let's look at it. Maybe that is good enough and we will be able to
finesse common authentication on top of two divergent RPC stacks.

Let me throw out there a third way: Hadoop and HBase gets together right
now in a "protobuf RPC wire format working group" and solves at least the
wire side of the problem before one or the other ships something that has
to be set going forward. On the HBase side, that's happening real soon now,
perhaps as early as the end of the month.

-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Re: [DISCUSSION] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues

[Devaraj Das <dd...@hortonworks.com>]

On Mon, Jul 1, 2013 at 3:34 PM, Stack <st...@duboce.net> wrote:

> On Mon, Jul 1, 2013 at 2:54 PM, Devaraj Das <dd...@hortonworks.com> wrote:
>
> > > so we might look at using Hadoop RPC directly again?
> >
> > Totally agree with that thought, Andrew. I must admit that I don't know
> > about all the context on why the HBase RPC forked in the first place :-)
> >
> >
> Bug fixes and speed optimizations
>
>
>
> > Your suggestion on using the Hadoop RPC work here makes sense. I am not
> too
> > sure whether that is practical in the short term though (we would need to
> > put in Tracing framework calls in Hadoop RPC and so on). IMO this will
> > further delay the 0.96 shipping as well. However doing the equivalent
> patch
> > for HADOOP-9421 in HBase will probably be easier and doable for 0.96...
> >
> >
> That patch won't go in to hbase.  Can you sketch the back and forth finally
> decided upon in an issue?  If you do this, I could help on how you might
> get it in.
>
>
I meant to say that using Hadoop RPC with changes needed for HBase to work
makes sense (and, of course, not the patch directly from HADOOP-9421 :-) )

But I'll write up the proposal implemented in HADOOP-9421 in a HBASE jira
and we can go from there.


> St.Ack
>

Re: [DISCUSSION] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues

[Devaraj Das <dd...@hortonworks.com>]

On Mon, Jul 1, 2013 at 3:56 PM, Andrew Purtell <ap...@apache.org> wrote:

> On Mon, Jul 1, 2013 at 3:34 PM, Stack <st...@duboce.net> wrote:
>
> > That patch won't go in to hbase.
>
>
> What if we and Hadoop can come up with a common RPC connection setup, like
> PB-based SASL negotiation and a common version header? Then there are
> options post-singularity.
>
>
Hopefully, the HADOOP-9421 is a step in that direction. I am going to spend
some time on HADOOP-9421.


>
> --
> Best regards,
>
>    - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>

Re: [DISCUSSION] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues

[Andrew Purtell <ap...@apache.org>]

On Mon, Jul 1, 2013 at 3:34 PM, Stack <st...@duboce.net> wrote:

> That patch won't go in to hbase.


What if we and Hadoop can come up with a common RPC connection setup, like
PB-based SASL negotiation and a common version header? Then there are
options post-singularity.


-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Re: [DISCUSSION] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues

[Stack <st...@duboce.net>]

On Mon, Jul 1, 2013 at 2:54 PM, Devaraj Das <dd...@hortonworks.com> wrote:

> > so we might look at using Hadoop RPC directly again?
>
> Totally agree with that thought, Andrew. I must admit that I don't know
> about all the context on why the HBase RPC forked in the first place :-)
>
>
Bug fixes and speed optimizations



> Your suggestion on using the Hadoop RPC work here makes sense. I am not too
> sure whether that is practical in the short term though (we would need to
> put in Tracing framework calls in Hadoop RPC and so on). IMO this will
> further delay the 0.96 shipping as well. However doing the equivalent patch
> for HADOOP-9421 in HBase will probably be easier and doable for 0.96...
>
>
That patch won't go in to hbase.  Can you sketch the back and forth finally
decided upon in an issue?  If you do this, I could help on how you might
get it in.

St.Ack

Re: [DISCUSSION] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues

[Devaraj Das <dd...@hortonworks.com>]

> so we might look at using Hadoop RPC directly again?

Totally agree with that thought, Andrew. I must admit that I don't know
about all the context on why the HBase RPC forked in the first place :-)

Your suggestion on using the Hadoop RPC work here makes sense. I am not too
sure whether that is practical in the short term though (we would need to
put in Tracing framework calls in Hadoop RPC and so on). IMO this will
further delay the 0.96 shipping as well. However doing the equivalent patch
for HADOOP-9421 in HBase will probably be easier and doable for 0.96...

On Mon, Jul 1, 2013 at 1:54 PM, Andrew Purtell <ap...@apache.org> wrote:

> On Mon, Jul 1, 2013 at 1:40 PM, Devaraj Das <dd...@hortonworks.com> wrote:
>
> > HADOOP-9421
>
>
> How about turning that around? (smile) Any thoughts on removing the
> indirection in Hadoop RPC like Stack did with HBase PB RPC so we might look
> at using Hadoop RPC directly again? Seems if the goal is to support
> pluggable authentication mechanisms being developed in core then we should
> be taking that up through a core artifact, not duplicating the work over
> here?
>
>
> --
> Best regards,
>
>    - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>

Re: [DISCUSSION] Finishing up 0.96 --> WAS Re: 0.95 and 0.96 remaining issues

[Andrew Purtell <ap...@apache.org>]

On Mon, Jul 1, 2013 at 1:40 PM, Devaraj Das <dd...@hortonworks.com> wrote:

> HADOOP-9421


How about turning that around? (smile) Any thoughts on removing the
indirection in Hadoop RPC like Stack did with HBase PB RPC so we might look
at using Hadoop RPC directly again? Seems if the goal is to support
pluggable authentication mechanisms being developed in core then we should
be taking that up through a core artifact, not duplicating the work over
here?


-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)