You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@crunch.apache.org by Josh Wills <jw...@cloudera.com> on 2013/10/16 07:42:50 UTC
Thoughts on supporting HBase 0.96
Hey all,
To kill some time this afternoon, I took a pass at figuring out what
changes would be needed in Crunch to support HBase 0.96, which is going
through a few release candidates right now. I started out by building
against the 0.95.2 release, which has most of the API changes that I'm told
we can expect in 0.96.
The most consequential change I found is that many of the core HBase
classes we operate on-- Put, Delete, KeyValue, and Result-- will no longer
implement the Writable interface. Instead, the HBase team has added a
number of SerializationFactory classes for these types, which map the POJO
versions of those objects on to protocol buffers. This means that the
current trick of creating PTypes for HBase like this:
PType<Result> ptype = Writables.writables(Result.class);
won't work anymore in 0.96, i.e., the HBase data classes won't fit into
either of the existing type families.
The best solution I've come up with so far is to create a new,
HBase-specific PTypeFamily for supporting the way these classes are
serialized now. I'm not sure if there's a better approach here and/or how
complex this particular PTypeFamily implementation would need to be; I'm
very much open to ideas on how to proceed here.
J
--
Director of Data Science
Cloudera <http://www.cloudera.com>
Twitter: @josh_wills <http://twitter.com/josh_wills>
Re: Thoughts on supporting HBase 0.96
Posted by Josh Wills <jw...@cloudera.com>.
On Thu, Oct 17, 2013 at 7:06 AM, Gabriel Reid <ga...@gmail.com>wrote:
> On Thu, Oct 17, 2013 at 3:38 PM, Josh Wills <jw...@cloudera.com> wrote:
>
> > My feeling is that the consensus here is that adding a new PTypeFamily
> is a
> > bad idea. :)
> >
> > The other idea I had would be to add a way for the Source and Target to
> > indicate that they were reading input data directly from the Hadoop
> > serialization framework, and thus did not need the input/output PTypes to
> > perform any additional transforms via getInputMapFn/getOutputMapFn. We
> > would still need different PTypes for working with the HBase objects
> (along
> > the lines that Gabriel mentioned earlier in the thread), but this
> approach
> > would solve the core issue w/o requiring a new PTypeFamily.
> >
>
> Is there any chance that this is just as simple as (re)implementing the
> getConverter
> method in the HBase-related Source and Target impls?
>
I hope it's only marginally more complicated than that-- the calls to
PType.getInputMapFn and PType.getOutputMapFn happen inside of the planner,
so we'll need an option that controls whether or not they are applied for a
given Source/Target.
>
>
>
>
> >
> >
> > On Wed, Oct 16, 2013 at 7:17 PM, Micah Whitacre <mk...@gmail.com>
> wrote:
> >
> > > If we created a new PTypeFamily we'd need to build in support to the
> > Avros
> > > (and possibly Writables) class to support wrapping the HBaseTypeFamily
> > > types.
> > >
> > >
> > > On Wed, Oct 16, 2013 at 10:20 AM, Gabriel Reid <gabriel.reid@gmail.com
> > > >wrote:
> > >
> > > > Ok, makes sense. And yeah, going from a Put to bytes and then back
> to a
> > > > Put in order to write to HBase doesn't sound too awesome.
> > > >
> > > >
> > > > On Wed, Oct 16, 2013 at 5:10 PM, Josh Wills <jo...@gmail.com>
> > > wrote:
> > > >
> > > > > On Wed, Oct 16, 2013 at 8:02 AM, Gabriel Reid <
> > gabriel.reid@gmail.com
> > > > > >wrote:
> > > > >
> > > > > > On Wed, Oct 16, 2013 at 4:34 PM, Josh Wills <jwills@cloudera.com
> >
> > > > wrote:
> > > > > >
> > > > > > > On Wed, Oct 16, 2013 at 12:15 AM, Gabriel Reid <
> > > > gabriel.reid@gmail.com
> > > > > > > >wrote:
> > > > > > >
> > > > > > > > Wouldn't a derived PType (like in o.a.c.types.PTypes) be a
> > better
> > > > fit
> > > > > > > here?
> > > > > > > >
> > > > > > >
> > > > > > > That was my initial attempt, and in an ideal world, my
> preferred
> > > > > > solution--
> > > > > > > but I haven't figured out how to make it work. The question
> here
> > > is:
> > > > > what
> > > > > > > do I derive a KeyValue object to? What I really want, for
> > purposes
> > > of
> > > > > > > reading it/writing it to one of our HBase IO formats, is to map
> > it
> > > to
> > > > > > > itself, and not some subclass of Writable. Another option might
> > be
> > > an
> > > > > > > extension of WritableType to handle these special case
> formats--
> > > I'll
> > > > > > take
> > > > > > > a crack at getting that to work.
> > > > > > >
> > > > > >
> > > > > > I'm sure I'm just missing something obvious, but I don't totally
> > get
> > > > it.
> > > > > > What I had
> > > > > > in my head is that KeyValue, Put, Delete, Result, etc could all
> be
> > > > > derived
> > > > > > to byte
> > > > > > arrays, with the KeyValueSerialization, MutationSerialization,
> and
> > > > > > ResultSerialization
> > > > > > classes being used in the MapFns within the derived PType to go
> > > between
> > > > > the
> > > > > > type and its byte representation, i.e.
> > > > > >
> > > > > > public static PType<KeyValue> keyValue(PTypeFamily ptf) {
> > > > > > return ptf.derived(
> > > > > > KeyValue.class,
> > > > > > BYTES_TO_KEYVALUE_VIA_KVSERIALIZATION,
> > > > > > KEYVALUE_TO_BYTES_VIA_KVSERIALIZATION,
> > > > > > ptf.bytes());
> > > > > > }
> > > > > >
> > > > > > I'm guessing this is the same thing you're talking about, which I
> > > > assume
> > > > > > means that
> > > > > > I'm missing something simple as to why that wouldn't just work,
> but
> > > I'm
> > > > > not
> > > > > > sure
> > > > > > what it is that I'm missing.
> > > > > >
> > > > > >
> > > > > The rub is the Input and Output formats, which don't expect bytes--
> > > they
> > > > > expect either subclasses of the Mutation interface (Put or Delete),
> > or
> > > > > KeyValue (for HFile) or Result (for HTable) inputs. So we would
> need
> > to
> > > > > change the input and output formats so that they would take in
> bytes
> > as
> > > > > arguments and then convert them back to the objects that the HBase
> > APIs
> > > > > expect, so something like:
> > > > >
> > > > > getOutputMapFn() -> OutputFormat
> > > > > Put -> bytes() -> Put
> > > > >
> > > > > That isn't the end of the world, it's just a little odd. We'd need
> to
> > > do
> > > > > something similar on the Input format side as well, so like:
> > > > >
> > > > > InputFormat -> getInputMapFn()
> > > > > Result -> bytes() -> Result
> > > > >
> > > > >
> > > > >
> > > > > >
> > > > >
> > > > > > >
> > > > > > >
> > > > > > > > A whole new PTypeFamily sounds like a lot of work (unless
> maybe
> > > if
> > > > it
> > > > > > > was a
> > > > > > > > subclass of one of the existing ones), and I think there's
> > still
> > > a
> > > > > fair
> > > > > > > bit
> > > > > > > > of code
> > > > > > > > that assumes that Avro & Writable are the only two possible
> > > > > PTypeFamily
> > > > > > > > implementations.
> > > > > > > >
> > > > > > >
> > > > > > > For any kind of intermediate processing, that is still true.
> The
> > > > > > > HBaseTypeFamily would only ever really appear at the input or
> > > output
> > > > > for
> > > > > > a
> > > > > > > job.
> > > > > > >
> > > > > > >
> > > > > > True, although of course it would be nice if we wouldn't have
> that
> > > > > > limitation.
> > > > > >
> > > > > > - Gabriel
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Director of Data Science
> > Cloudera <http://www.cloudera.com>
> > Twitter: @josh_wills <http://twitter.com/josh_wills>
> >
>
--
Director of Data Science
Cloudera <http://www.cloudera.com>
Twitter: @josh_wills <http://twitter.com/josh_wills>
Re: Thoughts on supporting HBase 0.96
Posted by Gabriel Reid <ga...@gmail.com>.
On Thu, Oct 17, 2013 at 3:38 PM, Josh Wills <jw...@cloudera.com> wrote:
> My feeling is that the consensus here is that adding a new PTypeFamily is a
> bad idea. :)
>
> The other idea I had would be to add a way for the Source and Target to
> indicate that they were reading input data directly from the Hadoop
> serialization framework, and thus did not need the input/output PTypes to
> perform any additional transforms via getInputMapFn/getOutputMapFn. We
> would still need different PTypes for working with the HBase objects (along
> the lines that Gabriel mentioned earlier in the thread), but this approach
> would solve the core issue w/o requiring a new PTypeFamily.
>
Is there any chance that this is just as simple as (re)implementing the
getConverter
method in the HBase-related Source and Target impls?
>
>
> On Wed, Oct 16, 2013 at 7:17 PM, Micah Whitacre <mk...@gmail.com> wrote:
>
> > If we created a new PTypeFamily we'd need to build in support to the
> Avros
> > (and possibly Writables) class to support wrapping the HBaseTypeFamily
> > types.
> >
> >
> > On Wed, Oct 16, 2013 at 10:20 AM, Gabriel Reid <gabriel.reid@gmail.com
> > >wrote:
> >
> > > Ok, makes sense. And yeah, going from a Put to bytes and then back to a
> > > Put in order to write to HBase doesn't sound too awesome.
> > >
> > >
> > > On Wed, Oct 16, 2013 at 5:10 PM, Josh Wills <jo...@gmail.com>
> > wrote:
> > >
> > > > On Wed, Oct 16, 2013 at 8:02 AM, Gabriel Reid <
> gabriel.reid@gmail.com
> > > > >wrote:
> > > >
> > > > > On Wed, Oct 16, 2013 at 4:34 PM, Josh Wills <jw...@cloudera.com>
> > > wrote:
> > > > >
> > > > > > On Wed, Oct 16, 2013 at 12:15 AM, Gabriel Reid <
> > > gabriel.reid@gmail.com
> > > > > > >wrote:
> > > > > >
> > > > > > > Wouldn't a derived PType (like in o.a.c.types.PTypes) be a
> better
> > > fit
> > > > > > here?
> > > > > > >
> > > > > >
> > > > > > That was my initial attempt, and in an ideal world, my preferred
> > > > > solution--
> > > > > > but I haven't figured out how to make it work. The question here
> > is:
> > > > what
> > > > > > do I derive a KeyValue object to? What I really want, for
> purposes
> > of
> > > > > > reading it/writing it to one of our HBase IO formats, is to map
> it
> > to
> > > > > > itself, and not some subclass of Writable. Another option might
> be
> > an
> > > > > > extension of WritableType to handle these special case formats--
> > I'll
> > > > > take
> > > > > > a crack at getting that to work.
> > > > > >
> > > > >
> > > > > I'm sure I'm just missing something obvious, but I don't totally
> get
> > > it.
> > > > > What I had
> > > > > in my head is that KeyValue, Put, Delete, Result, etc could all be
> > > > derived
> > > > > to byte
> > > > > arrays, with the KeyValueSerialization, MutationSerialization, and
> > > > > ResultSerialization
> > > > > classes being used in the MapFns within the derived PType to go
> > between
> > > > the
> > > > > type and its byte representation, i.e.
> > > > >
> > > > > public static PType<KeyValue> keyValue(PTypeFamily ptf) {
> > > > > return ptf.derived(
> > > > > KeyValue.class,
> > > > > BYTES_TO_KEYVALUE_VIA_KVSERIALIZATION,
> > > > > KEYVALUE_TO_BYTES_VIA_KVSERIALIZATION,
> > > > > ptf.bytes());
> > > > > }
> > > > >
> > > > > I'm guessing this is the same thing you're talking about, which I
> > > assume
> > > > > means that
> > > > > I'm missing something simple as to why that wouldn't just work, but
> > I'm
> > > > not
> > > > > sure
> > > > > what it is that I'm missing.
> > > > >
> > > > >
> > > > The rub is the Input and Output formats, which don't expect bytes--
> > they
> > > > expect either subclasses of the Mutation interface (Put or Delete),
> or
> > > > KeyValue (for HFile) or Result (for HTable) inputs. So we would need
> to
> > > > change the input and output formats so that they would take in bytes
> as
> > > > arguments and then convert them back to the objects that the HBase
> APIs
> > > > expect, so something like:
> > > >
> > > > getOutputMapFn() -> OutputFormat
> > > > Put -> bytes() -> Put
> > > >
> > > > That isn't the end of the world, it's just a little odd. We'd need to
> > do
> > > > something similar on the Input format side as well, so like:
> > > >
> > > > InputFormat -> getInputMapFn()
> > > > Result -> bytes() -> Result
> > > >
> > > >
> > > >
> > > > >
> > > >
> > > > > >
> > > > > >
> > > > > > > A whole new PTypeFamily sounds like a lot of work (unless maybe
> > if
> > > it
> > > > > > was a
> > > > > > > subclass of one of the existing ones), and I think there's
> still
> > a
> > > > fair
> > > > > > bit
> > > > > > > of code
> > > > > > > that assumes that Avro & Writable are the only two possible
> > > > PTypeFamily
> > > > > > > implementations.
> > > > > > >
> > > > > >
> > > > > > For any kind of intermediate processing, that is still true. The
> > > > > > HBaseTypeFamily would only ever really appear at the input or
> > output
> > > > for
> > > > > a
> > > > > > job.
> > > > > >
> > > > > >
> > > > > True, although of course it would be nice if we wouldn't have that
> > > > > limitation.
> > > > >
> > > > > - Gabriel
> > > > >
> > > >
> > >
> >
>
>
>
> --
> Director of Data Science
> Cloudera <http://www.cloudera.com>
> Twitter: @josh_wills <http://twitter.com/josh_wills>
>
Re: Thoughts on supporting HBase 0.96
Posted by Josh Wills <jw...@cloudera.com>.
My feeling is that the consensus here is that adding a new PTypeFamily is a
bad idea. :)
The other idea I had would be to add a way for the Source and Target to
indicate that they were reading input data directly from the Hadoop
serialization framework, and thus did not need the input/output PTypes to
perform any additional transforms via getInputMapFn/getOutputMapFn. We
would still need different PTypes for working with the HBase objects (along
the lines that Gabriel mentioned earlier in the thread), but this approach
would solve the core issue w/o requiring a new PTypeFamily.
On Wed, Oct 16, 2013 at 7:17 PM, Micah Whitacre <mk...@gmail.com> wrote:
> If we created a new PTypeFamily we'd need to build in support to the Avros
> (and possibly Writables) class to support wrapping the HBaseTypeFamily
> types.
>
>
> On Wed, Oct 16, 2013 at 10:20 AM, Gabriel Reid <gabriel.reid@gmail.com
> >wrote:
>
> > Ok, makes sense. And yeah, going from a Put to bytes and then back to a
> > Put in order to write to HBase doesn't sound too awesome.
> >
> >
> > On Wed, Oct 16, 2013 at 5:10 PM, Josh Wills <jo...@gmail.com>
> wrote:
> >
> > > On Wed, Oct 16, 2013 at 8:02 AM, Gabriel Reid <gabriel.reid@gmail.com
> > > >wrote:
> > >
> > > > On Wed, Oct 16, 2013 at 4:34 PM, Josh Wills <jw...@cloudera.com>
> > wrote:
> > > >
> > > > > On Wed, Oct 16, 2013 at 12:15 AM, Gabriel Reid <
> > gabriel.reid@gmail.com
> > > > > >wrote:
> > > > >
> > > > > > Wouldn't a derived PType (like in o.a.c.types.PTypes) be a better
> > fit
> > > > > here?
> > > > > >
> > > > >
> > > > > That was my initial attempt, and in an ideal world, my preferred
> > > > solution--
> > > > > but I haven't figured out how to make it work. The question here
> is:
> > > what
> > > > > do I derive a KeyValue object to? What I really want, for purposes
> of
> > > > > reading it/writing it to one of our HBase IO formats, is to map it
> to
> > > > > itself, and not some subclass of Writable. Another option might be
> an
> > > > > extension of WritableType to handle these special case formats--
> I'll
> > > > take
> > > > > a crack at getting that to work.
> > > > >
> > > >
> > > > I'm sure I'm just missing something obvious, but I don't totally get
> > it.
> > > > What I had
> > > > in my head is that KeyValue, Put, Delete, Result, etc could all be
> > > derived
> > > > to byte
> > > > arrays, with the KeyValueSerialization, MutationSerialization, and
> > > > ResultSerialization
> > > > classes being used in the MapFns within the derived PType to go
> between
> > > the
> > > > type and its byte representation, i.e.
> > > >
> > > > public static PType<KeyValue> keyValue(PTypeFamily ptf) {
> > > > return ptf.derived(
> > > > KeyValue.class,
> > > > BYTES_TO_KEYVALUE_VIA_KVSERIALIZATION,
> > > > KEYVALUE_TO_BYTES_VIA_KVSERIALIZATION,
> > > > ptf.bytes());
> > > > }
> > > >
> > > > I'm guessing this is the same thing you're talking about, which I
> > assume
> > > > means that
> > > > I'm missing something simple as to why that wouldn't just work, but
> I'm
> > > not
> > > > sure
> > > > what it is that I'm missing.
> > > >
> > > >
> > > The rub is the Input and Output formats, which don't expect bytes--
> they
> > > expect either subclasses of the Mutation interface (Put or Delete), or
> > > KeyValue (for HFile) or Result (for HTable) inputs. So we would need to
> > > change the input and output formats so that they would take in bytes as
> > > arguments and then convert them back to the objects that the HBase APIs
> > > expect, so something like:
> > >
> > > getOutputMapFn() -> OutputFormat
> > > Put -> bytes() -> Put
> > >
> > > That isn't the end of the world, it's just a little odd. We'd need to
> do
> > > something similar on the Input format side as well, so like:
> > >
> > > InputFormat -> getInputMapFn()
> > > Result -> bytes() -> Result
> > >
> > >
> > >
> > > >
> > >
> > > > >
> > > > >
> > > > > > A whole new PTypeFamily sounds like a lot of work (unless maybe
> if
> > it
> > > > > was a
> > > > > > subclass of one of the existing ones), and I think there's still
> a
> > > fair
> > > > > bit
> > > > > > of code
> > > > > > that assumes that Avro & Writable are the only two possible
> > > PTypeFamily
> > > > > > implementations.
> > > > > >
> > > > >
> > > > > For any kind of intermediate processing, that is still true. The
> > > > > HBaseTypeFamily would only ever really appear at the input or
> output
> > > for
> > > > a
> > > > > job.
> > > > >
> > > > >
> > > > True, although of course it would be nice if we wouldn't have that
> > > > limitation.
> > > >
> > > > - Gabriel
> > > >
> > >
> >
>
--
Director of Data Science
Cloudera <http://www.cloudera.com>
Twitter: @josh_wills <http://twitter.com/josh_wills>
Re: Thoughts on supporting HBase 0.96
Posted by Micah Whitacre <mk...@gmail.com>.
If we created a new PTypeFamily we'd need to build in support to the Avros
(and possibly Writables) class to support wrapping the HBaseTypeFamily
types.
On Wed, Oct 16, 2013 at 10:20 AM, Gabriel Reid <ga...@gmail.com>wrote:
> Ok, makes sense. And yeah, going from a Put to bytes and then back to a
> Put in order to write to HBase doesn't sound too awesome.
>
>
> On Wed, Oct 16, 2013 at 5:10 PM, Josh Wills <jo...@gmail.com> wrote:
>
> > On Wed, Oct 16, 2013 at 8:02 AM, Gabriel Reid <gabriel.reid@gmail.com
> > >wrote:
> >
> > > On Wed, Oct 16, 2013 at 4:34 PM, Josh Wills <jw...@cloudera.com>
> wrote:
> > >
> > > > On Wed, Oct 16, 2013 at 12:15 AM, Gabriel Reid <
> gabriel.reid@gmail.com
> > > > >wrote:
> > > >
> > > > > Wouldn't a derived PType (like in o.a.c.types.PTypes) be a better
> fit
> > > > here?
> > > > >
> > > >
> > > > That was my initial attempt, and in an ideal world, my preferred
> > > solution--
> > > > but I haven't figured out how to make it work. The question here is:
> > what
> > > > do I derive a KeyValue object to? What I really want, for purposes of
> > > > reading it/writing it to one of our HBase IO formats, is to map it to
> > > > itself, and not some subclass of Writable. Another option might be an
> > > > extension of WritableType to handle these special case formats-- I'll
> > > take
> > > > a crack at getting that to work.
> > > >
> > >
> > > I'm sure I'm just missing something obvious, but I don't totally get
> it.
> > > What I had
> > > in my head is that KeyValue, Put, Delete, Result, etc could all be
> > derived
> > > to byte
> > > arrays, with the KeyValueSerialization, MutationSerialization, and
> > > ResultSerialization
> > > classes being used in the MapFns within the derived PType to go between
> > the
> > > type and its byte representation, i.e.
> > >
> > > public static PType<KeyValue> keyValue(PTypeFamily ptf) {
> > > return ptf.derived(
> > > KeyValue.class,
> > > BYTES_TO_KEYVALUE_VIA_KVSERIALIZATION,
> > > KEYVALUE_TO_BYTES_VIA_KVSERIALIZATION,
> > > ptf.bytes());
> > > }
> > >
> > > I'm guessing this is the same thing you're talking about, which I
> assume
> > > means that
> > > I'm missing something simple as to why that wouldn't just work, but I'm
> > not
> > > sure
> > > what it is that I'm missing.
> > >
> > >
> > The rub is the Input and Output formats, which don't expect bytes-- they
> > expect either subclasses of the Mutation interface (Put or Delete), or
> > KeyValue (for HFile) or Result (for HTable) inputs. So we would need to
> > change the input and output formats so that they would take in bytes as
> > arguments and then convert them back to the objects that the HBase APIs
> > expect, so something like:
> >
> > getOutputMapFn() -> OutputFormat
> > Put -> bytes() -> Put
> >
> > That isn't the end of the world, it's just a little odd. We'd need to do
> > something similar on the Input format side as well, so like:
> >
> > InputFormat -> getInputMapFn()
> > Result -> bytes() -> Result
> >
> >
> >
> > >
> >
> > > >
> > > >
> > > > > A whole new PTypeFamily sounds like a lot of work (unless maybe if
> it
> > > > was a
> > > > > subclass of one of the existing ones), and I think there's still a
> > fair
> > > > bit
> > > > > of code
> > > > > that assumes that Avro & Writable are the only two possible
> > PTypeFamily
> > > > > implementations.
> > > > >
> > > >
> > > > For any kind of intermediate processing, that is still true. The
> > > > HBaseTypeFamily would only ever really appear at the input or output
> > for
> > > a
> > > > job.
> > > >
> > > >
> > > True, although of course it would be nice if we wouldn't have that
> > > limitation.
> > >
> > > - Gabriel
> > >
> >
>
Re: Thoughts on supporting HBase 0.96
Posted by Gabriel Reid <ga...@gmail.com>.
Ok, makes sense. And yeah, going from a Put to bytes and then back to a
Put in order to write to HBase doesn't sound too awesome.
On Wed, Oct 16, 2013 at 5:10 PM, Josh Wills <jo...@gmail.com> wrote:
> On Wed, Oct 16, 2013 at 8:02 AM, Gabriel Reid <gabriel.reid@gmail.com
> >wrote:
>
> > On Wed, Oct 16, 2013 at 4:34 PM, Josh Wills <jw...@cloudera.com> wrote:
> >
> > > On Wed, Oct 16, 2013 at 12:15 AM, Gabriel Reid <gabriel.reid@gmail.com
> > > >wrote:
> > >
> > > > Wouldn't a derived PType (like in o.a.c.types.PTypes) be a better fit
> > > here?
> > > >
> > >
> > > That was my initial attempt, and in an ideal world, my preferred
> > solution--
> > > but I haven't figured out how to make it work. The question here is:
> what
> > > do I derive a KeyValue object to? What I really want, for purposes of
> > > reading it/writing it to one of our HBase IO formats, is to map it to
> > > itself, and not some subclass of Writable. Another option might be an
> > > extension of WritableType to handle these special case formats-- I'll
> > take
> > > a crack at getting that to work.
> > >
> >
> > I'm sure I'm just missing something obvious, but I don't totally get it.
> > What I had
> > in my head is that KeyValue, Put, Delete, Result, etc could all be
> derived
> > to byte
> > arrays, with the KeyValueSerialization, MutationSerialization, and
> > ResultSerialization
> > classes being used in the MapFns within the derived PType to go between
> the
> > type and its byte representation, i.e.
> >
> > public static PType<KeyValue> keyValue(PTypeFamily ptf) {
> > return ptf.derived(
> > KeyValue.class,
> > BYTES_TO_KEYVALUE_VIA_KVSERIALIZATION,
> > KEYVALUE_TO_BYTES_VIA_KVSERIALIZATION,
> > ptf.bytes());
> > }
> >
> > I'm guessing this is the same thing you're talking about, which I assume
> > means that
> > I'm missing something simple as to why that wouldn't just work, but I'm
> not
> > sure
> > what it is that I'm missing.
> >
> >
> The rub is the Input and Output formats, which don't expect bytes-- they
> expect either subclasses of the Mutation interface (Put or Delete), or
> KeyValue (for HFile) or Result (for HTable) inputs. So we would need to
> change the input and output formats so that they would take in bytes as
> arguments and then convert them back to the objects that the HBase APIs
> expect, so something like:
>
> getOutputMapFn() -> OutputFormat
> Put -> bytes() -> Put
>
> That isn't the end of the world, it's just a little odd. We'd need to do
> something similar on the Input format side as well, so like:
>
> InputFormat -> getInputMapFn()
> Result -> bytes() -> Result
>
>
>
> >
>
> > >
> > >
> > > > A whole new PTypeFamily sounds like a lot of work (unless maybe if it
> > > was a
> > > > subclass of one of the existing ones), and I think there's still a
> fair
> > > bit
> > > > of code
> > > > that assumes that Avro & Writable are the only two possible
> PTypeFamily
> > > > implementations.
> > > >
> > >
> > > For any kind of intermediate processing, that is still true. The
> > > HBaseTypeFamily would only ever really appear at the input or output
> for
> > a
> > > job.
> > >
> > >
> > True, although of course it would be nice if we wouldn't have that
> > limitation.
> >
> > - Gabriel
> >
>
Re: Thoughts on supporting HBase 0.96
Posted by Josh Wills <jo...@gmail.com>.
On Wed, Oct 16, 2013 at 8:02 AM, Gabriel Reid <ga...@gmail.com>wrote:
> On Wed, Oct 16, 2013 at 4:34 PM, Josh Wills <jw...@cloudera.com> wrote:
>
> > On Wed, Oct 16, 2013 at 12:15 AM, Gabriel Reid <gabriel.reid@gmail.com
> > >wrote:
> >
> > > Wouldn't a derived PType (like in o.a.c.types.PTypes) be a better fit
> > here?
> > >
> >
> > That was my initial attempt, and in an ideal world, my preferred
> solution--
> > but I haven't figured out how to make it work. The question here is: what
> > do I derive a KeyValue object to? What I really want, for purposes of
> > reading it/writing it to one of our HBase IO formats, is to map it to
> > itself, and not some subclass of Writable. Another option might be an
> > extension of WritableType to handle these special case formats-- I'll
> take
> > a crack at getting that to work.
> >
>
> I'm sure I'm just missing something obvious, but I don't totally get it.
> What I had
> in my head is that KeyValue, Put, Delete, Result, etc could all be derived
> to byte
> arrays, with the KeyValueSerialization, MutationSerialization, and
> ResultSerialization
> classes being used in the MapFns within the derived PType to go between the
> type and its byte representation, i.e.
>
> public static PType<KeyValue> keyValue(PTypeFamily ptf) {
> return ptf.derived(
> KeyValue.class,
> BYTES_TO_KEYVALUE_VIA_KVSERIALIZATION,
> KEYVALUE_TO_BYTES_VIA_KVSERIALIZATION,
> ptf.bytes());
> }
>
> I'm guessing this is the same thing you're talking about, which I assume
> means that
> I'm missing something simple as to why that wouldn't just work, but I'm not
> sure
> what it is that I'm missing.
>
>
The rub is the Input and Output formats, which don't expect bytes-- they
expect either subclasses of the Mutation interface (Put or Delete), or
KeyValue (for HFile) or Result (for HTable) inputs. So we would need to
change the input and output formats so that they would take in bytes as
arguments and then convert them back to the objects that the HBase APIs
expect, so something like:
getOutputMapFn() -> OutputFormat
Put -> bytes() -> Put
That isn't the end of the world, it's just a little odd. We'd need to do
something similar on the Input format side as well, so like:
InputFormat -> getInputMapFn()
Result -> bytes() -> Result
>
> >
> >
> > > A whole new PTypeFamily sounds like a lot of work (unless maybe if it
> > was a
> > > subclass of one of the existing ones), and I think there's still a fair
> > bit
> > > of code
> > > that assumes that Avro & Writable are the only two possible PTypeFamily
> > > implementations.
> > >
> >
> > For any kind of intermediate processing, that is still true. The
> > HBaseTypeFamily would only ever really appear at the input or output for
> a
> > job.
> >
> >
> True, although of course it would be nice if we wouldn't have that
> limitation.
>
> - Gabriel
>
Re: Thoughts on supporting HBase 0.96
Posted by Gabriel Reid <ga...@gmail.com>.
On Wed, Oct 16, 2013 at 4:34 PM, Josh Wills <jw...@cloudera.com> wrote:
> On Wed, Oct 16, 2013 at 12:15 AM, Gabriel Reid <gabriel.reid@gmail.com
> >wrote:
>
> > Wouldn't a derived PType (like in o.a.c.types.PTypes) be a better fit
> here?
> >
>
> That was my initial attempt, and in an ideal world, my preferred solution--
> but I haven't figured out how to make it work. The question here is: what
> do I derive a KeyValue object to? What I really want, for purposes of
> reading it/writing it to one of our HBase IO formats, is to map it to
> itself, and not some subclass of Writable. Another option might be an
> extension of WritableType to handle these special case formats-- I'll take
> a crack at getting that to work.
>
I'm sure I'm just missing something obvious, but I don't totally get it.
What I had
in my head is that KeyValue, Put, Delete, Result, etc could all be derived
to byte
arrays, with the KeyValueSerialization, MutationSerialization, and
ResultSerialization
classes being used in the MapFns within the derived PType to go between the
type and its byte representation, i.e.
public static PType<KeyValue> keyValue(PTypeFamily ptf) {
return ptf.derived(
KeyValue.class,
BYTES_TO_KEYVALUE_VIA_KVSERIALIZATION,
KEYVALUE_TO_BYTES_VIA_KVSERIALIZATION,
ptf.bytes());
}
I'm guessing this is the same thing you're talking about, which I assume
means that
I'm missing something simple as to why that wouldn't just work, but I'm not
sure
what it is that I'm missing.
>
>
> > A whole new PTypeFamily sounds like a lot of work (unless maybe if it
> was a
> > subclass of one of the existing ones), and I think there's still a fair
> bit
> > of code
> > that assumes that Avro & Writable are the only two possible PTypeFamily
> > implementations.
> >
>
> For any kind of intermediate processing, that is still true. The
> HBaseTypeFamily would only ever really appear at the input or output for a
> job.
>
>
True, although of course it would be nice if we wouldn't have that
limitation.
- Gabriel
Re: Thoughts on supporting HBase 0.96
Posted by Josh Wills <jw...@cloudera.com>.
On Wed, Oct 16, 2013 at 12:15 AM, Gabriel Reid <ga...@gmail.com>wrote:
> On Wed, Oct 16, 2013 at 8:46 AM, Josh Wills <jw...@cloudera.com> wrote:
>
> > On Tue, Oct 15, 2013 at 11:42 PM, Chao Shi <st...@live.com> wrote:
> >
> > > I don't understand why needs another PTypeFamily here. I think we can
> > > simply provide some pre-defined PTypes.
> > >
> > > interface HFilePTypes {
> > > static KEY_VALUE_PTYPE = xxx
> > > static PUT_PTYPE = xxx
> > > }
> > >
> >
> > Technically, every PType has to provide an implementation of the
> > PTypeFamily getFamily() method-- even if it's just returning a dummy
> > object.
> >
> >
> Wouldn't a derived PType (like in o.a.c.types.PTypes) be a better fit here?
>
That was my initial attempt, and in an ideal world, my preferred solution--
but I haven't figured out how to make it work. The question here is: what
do I derive a KeyValue object to? What I really want, for purposes of
reading it/writing it to one of our HBase IO formats, is to map it to
itself, and not some subclass of Writable. Another option might be an
extension of WritableType to handle these special case formats-- I'll take
a crack at getting that to work.
> A whole new PTypeFamily sounds like a lot of work (unless maybe if it was a
> subclass of one of the existing ones), and I think there's still a fair bit
> of code
> that assumes that Avro & Writable are the only two possible PTypeFamily
> implementations.
>
For any kind of intermediate processing, that is still true. The
HBaseTypeFamily would only ever really appear at the input or output for a
job.
>
> - Gabriel
>
--
Director of Data Science
Cloudera <http://www.cloudera.com>
Twitter: @josh_wills <http://twitter.com/josh_wills>
Re: Thoughts on supporting HBase 0.96
Posted by Gabriel Reid <ga...@gmail.com>.
On Wed, Oct 16, 2013 at 8:46 AM, Josh Wills <jw...@cloudera.com> wrote:
> On Tue, Oct 15, 2013 at 11:42 PM, Chao Shi <st...@live.com> wrote:
>
> > I don't understand why needs another PTypeFamily here. I think we can
> > simply provide some pre-defined PTypes.
> >
> > interface HFilePTypes {
> > static KEY_VALUE_PTYPE = xxx
> > static PUT_PTYPE = xxx
> > }
> >
>
> Technically, every PType has to provide an implementation of the
> PTypeFamily getFamily() method-- even if it's just returning a dummy
> object.
>
>
Wouldn't a derived PType (like in o.a.c.types.PTypes) be a better fit here?
A whole new PTypeFamily sounds like a lot of work (unless maybe if it was a
subclass of one of the existing ones), and I think there's still a fair bit
of code
that assumes that Avro & Writable are the only two possible PTypeFamily
implementations.
- Gabriel
Re: Thoughts on supporting HBase 0.96
Posted by Josh Wills <jw...@cloudera.com>.
On Tue, Oct 15, 2013 at 11:42 PM, Chao Shi <st...@live.com> wrote:
> Hi Josh,
>
> I don't understand why needs another PTypeFamily here. I think we can
> simply provide some pre-defined PTypes.
>
> interface HFilePTypes {
> static KEY_VALUE_PTYPE = xxx
> static PUT_PTYPE = xxx
> }
>
Technically, every PType has to provide an implementation of the
PTypeFamily getFamily() method-- even if it's just returning a dummy object.
>
> After a quick scan of the latest code, it seems they have dropped any
> Writable things and moved to protobuf. For example, there is
> RequestConverter class which converts Mutations (super class of Put and
> Delete) to the protobuf message. We can leverage this to implement our
> PType. But unfortunately, this piece of code does not exist in 0.94. So If
> we want to support both, we have to do similar thing as we did for hadoop 1
> and 2, which is very boring. (I have no experience in using HBase releases
> after 0.94, so what I said may be wrong.)
>
Totally agree that it's boring-- I'm happy to volunteer to do it. ;)
>
>
> 2013/10/16 Josh Wills <jw...@cloudera.com>
>
> > Hey all,
> >
> > To kill some time this afternoon, I took a pass at figuring out what
> > changes would be needed in Crunch to support HBase 0.96, which is going
> > through a few release candidates right now. I started out by building
> > against the 0.95.2 release, which has most of the API changes that I'm
> told
> > we can expect in 0.96.
> >
> > The most consequential change I found is that many of the core HBase
> > classes we operate on-- Put, Delete, KeyValue, and Result-- will no
> longer
> > implement the Writable interface. Instead, the HBase team has added a
> > number of SerializationFactory classes for these types, which map the
> POJO
> > versions of those objects on to protocol buffers. This means that the
> > current trick of creating PTypes for HBase like this:
> >
> > PType<Result> ptype = Writables.writables(Result.class);
> >
> > won't work anymore in 0.96, i.e., the HBase data classes won't fit into
> > either of the existing type families.
> >
> > The best solution I've come up with so far is to create a new,
> > HBase-specific PTypeFamily for supporting the way these classes are
> > serialized now. I'm not sure if there's a better approach here and/or how
> > complex this particular PTypeFamily implementation would need to be; I'm
> > very much open to ideas on how to proceed here.
> >
> > J
> >
> >
> > --
> > Director of Data Science
> > Cloudera <http://www.cloudera.com>
> > Twitter: @josh_wills <http://twitter.com/josh_wills>
> >
>
--
Director of Data Science
Cloudera <http://www.cloudera.com>
Twitter: @josh_wills <http://twitter.com/josh_wills>
Re: Thoughts on supporting HBase 0.96
Posted by Chao Shi <st...@live.com>.
Hi Josh,
I don't understand why needs another PTypeFamily here. I think we can
simply provide some pre-defined PTypes.
interface HFilePTypes {
static KEY_VALUE_PTYPE = xxx
static PUT_PTYPE = xxx
}
After a quick scan of the latest code, it seems they have dropped any
Writable things and moved to protobuf. For example, there is
RequestConverter class which converts Mutations (super class of Put and
Delete) to the protobuf message. We can leverage this to implement our
PType. But unfortunately, this piece of code does not exist in 0.94. So If
we want to support both, we have to do similar thing as we did for hadoop 1
and 2, which is very boring. (I have no experience in using HBase releases
after 0.94, so what I said may be wrong.)
2013/10/16 Josh Wills <jw...@cloudera.com>
> Hey all,
>
> To kill some time this afternoon, I took a pass at figuring out what
> changes would be needed in Crunch to support HBase 0.96, which is going
> through a few release candidates right now. I started out by building
> against the 0.95.2 release, which has most of the API changes that I'm told
> we can expect in 0.96.
>
> The most consequential change I found is that many of the core HBase
> classes we operate on-- Put, Delete, KeyValue, and Result-- will no longer
> implement the Writable interface. Instead, the HBase team has added a
> number of SerializationFactory classes for these types, which map the POJO
> versions of those objects on to protocol buffers. This means that the
> current trick of creating PTypes for HBase like this:
>
> PType<Result> ptype = Writables.writables(Result.class);
>
> won't work anymore in 0.96, i.e., the HBase data classes won't fit into
> either of the existing type families.
>
> The best solution I've come up with so far is to create a new,
> HBase-specific PTypeFamily for supporting the way these classes are
> serialized now. I'm not sure if there's a better approach here and/or how
> complex this particular PTypeFamily implementation would need to be; I'm
> very much open to ideas on how to proceed here.
>
> J
>
>
> --
> Director of Data Science
> Cloudera <http://www.cloudera.com>
> Twitter: @josh_wills <http://twitter.com/josh_wills>
>