You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@crunch.apache.org by Stephen Durfey <sj...@gmail.com> on 2018/03/13 19:10:39 UTC
Reading from HCat as Avro
I wasn't sure if this was the place to discuss this or on a JIRA. To
followup on the work for the HCatSource, I wrote some code to convert an
HCatRecord into a specific avro model. This way you can read from HCat, but
still deal with avro models instead of HCatRecord. It really isn't the
ideal path as HCat already does SerDe operations when it does schema
resolution. So, it's rather a rather inefficient path. At least with this
code it's all in memory pointer moving, rather than additional byte
serialization.
HCatalog could be used to just find the desired partitions, and then a
regular pipeline can be used to read in the avro models. However, you then
need to know where the data is located and its file format, which is one of
the biggest benefits of the HCatalog.
I'm not really sure if this conversion code in crunch is the right place
for it to live long term, as there isn't anything about it that is crunch
specific. Hive seems like the better place for it to live. However, I'm not
sure when they would get around to committing it, and it would likely be in
a version that is beyond what we support today in Crunch (hive 2.1). So,
maybe in crunch short term until it is accepted by hive?
If this code is valuable, I can open a JIRA and we can go from there.
Code:
https://github.com/sjdurfey/crunch/commit/f659dfe06f50862b9f674c1e2dd699a5c53b2b1f
The code to look at would be the HCatDecoder, HCatParser, and the various
HCatParse contexts.
Re: Reading from HCat as Avro
Posted by Josh Wills <jo...@gmail.com>.
Yeah, that makes a ton of sense; I don't know how much a non-generic,
Avro-only solution would be used outside of this specific context, where a
team has all of their data in Avro already. contrib/, perhaps?
J
On Tue, Mar 20, 2018 at 1:35 PM, Stephen Durfey <sj...@gmail.com> wrote:
> The problem I was trying to solve was not wanting to deal with HCatRecords
> (which are basically GenericRecords) in the M/R code. Particularly with all
> the code we (my team/org) have around today written against the specific
> record models. So, HCat can be used for its benefits (abstracting away on
> disk file format, data discoverability, a single schema for multiple tools
> to plug into), without having to change existing code to work with
> HCatRecords.
>
> Now, this does require the data on disk to be avro, so its a bit of an
> abstraction leak knowing that. It is also rather difficult to solve
> generically, as the storage handlers return the data differently (e.g. hdfs
> handler returns avro as is, but hbase handler returns maps of structs). So,
> there is already a bit of a leak when dealing with HCatRecords since you
> need to know where the data is coming from in order to know how to process
> it. So, its probably ok. My current solution only works for avro files on
> disk.
>
> An example of it working with the existing HCatSource:
> https://github.com/sjdurfey/crunch/blob/from-hcat-avro/
> crunch-hcatalog/src/main/java/org/apache/crunch/io/hcatalog/
> FromHCat.java#L162-L193
>
> On Tue, Mar 13, 2018 at 11:43 PM, Josh Wills <jo...@gmail.com> wrote:
>
> > The Crunch-specific argument in favor of it, at least from my
> perspective,
> > is that a) Crunch is the most Avro-friendly pipeline library around
> (almost
> > to a fault, you might argue), and b) a Hive/HCat dependency in Avro
> itself
> > would make even less sense, and if you're using Hive SerDe/HQL/UDF, then
> > you're already bought into Hive's data model anyway and an HCat -> Avro
> > translator wouldn't make much sense.
> >
> > Does the conversion buy me anything from a MapReduce perspective? One of
> my
> > favorite aspects of Avro has always been the super-fast serialization of
> > data during the shuffle of MR jobs, even compared to Thrift/Protobuf b/c
> > the schema allows you to skip so much id overhead, but since HCat has the
> > same schema benefits as Avro, the performance argument doesn't seem as
> > obviously compelling to me at first blush.
> >
> > Josh
> >
> > P.S. This is exactly the place to discuss this sort of thing!
> >
> > On Tue, Mar 13, 2018 at 12:10 PM, Stephen Durfey <sj...@gmail.com>
> > wrote:
> >
> > > I wasn't sure if this was the place to discuss this or on a JIRA. To
> > > followup on the work for the HCatSource, I wrote some code to convert
> an
> > > HCatRecord into a specific avro model. This way you can read from HCat,
> > but
> > > still deal with avro models instead of HCatRecord. It really isn't the
> > > ideal path as HCat already does SerDe operations when it does schema
> > > resolution. So, it's rather a rather inefficient path. At least with
> this
> > > code it's all in memory pointer moving, rather than additional byte
> > > serialization.
> > >
> > > HCatalog could be used to just find the desired partitions, and then a
> > > regular pipeline can be used to read in the avro models. However, you
> > then
> > > need to know where the data is located and its file format, which is
> one
> > of
> > > the biggest benefits of the HCatalog.
> > >
> > > I'm not really sure if this conversion code in crunch is the right
> place
> > > for it to live long term, as there isn't anything about it that is
> crunch
> > > specific. Hive seems like the better place for it to live. However, I'm
> > not
> > > sure when they would get around to committing it, and it would likely
> be
> > in
> > > a version that is beyond what we support today in Crunch (hive 2.1).
> So,
> > > maybe in crunch short term until it is accepted by hive?
> > >
> > > If this code is valuable, I can open a JIRA and we can go from there.
> > >
> > > Code:
> > > https://github.com/sjdurfey/crunch/commit/
> f659dfe06f50862b9f674c1e2dd699
> > > a5c53b2b1f
> > >
> > > The code to look at would be the HCatDecoder, HCatParser, and the
> various
> > > HCatParse contexts.
> > >
> >
>
Re: Reading from HCat as Avro
Posted by Stephen Durfey <sj...@gmail.com>.
The problem I was trying to solve was not wanting to deal with HCatRecords
(which are basically GenericRecords) in the M/R code. Particularly with all
the code we (my team/org) have around today written against the specific
record models. So, HCat can be used for its benefits (abstracting away on
disk file format, data discoverability, a single schema for multiple tools
to plug into), without having to change existing code to work with
HCatRecords.
Now, this does require the data on disk to be avro, so its a bit of an
abstraction leak knowing that. It is also rather difficult to solve
generically, as the storage handlers return the data differently (e.g. hdfs
handler returns avro as is, but hbase handler returns maps of structs). So,
there is already a bit of a leak when dealing with HCatRecords since you
need to know where the data is coming from in order to know how to process
it. So, its probably ok. My current solution only works for avro files on
disk.
An example of it working with the existing HCatSource:
https://github.com/sjdurfey/crunch/blob/from-hcat-avro/crunch-hcatalog/src/main/java/org/apache/crunch/io/hcatalog/FromHCat.java#L162-L193
On Tue, Mar 13, 2018 at 11:43 PM, Josh Wills <jo...@gmail.com> wrote:
> The Crunch-specific argument in favor of it, at least from my perspective,
> is that a) Crunch is the most Avro-friendly pipeline library around (almost
> to a fault, you might argue), and b) a Hive/HCat dependency in Avro itself
> would make even less sense, and if you're using Hive SerDe/HQL/UDF, then
> you're already bought into Hive's data model anyway and an HCat -> Avro
> translator wouldn't make much sense.
>
> Does the conversion buy me anything from a MapReduce perspective? One of my
> favorite aspects of Avro has always been the super-fast serialization of
> data during the shuffle of MR jobs, even compared to Thrift/Protobuf b/c
> the schema allows you to skip so much id overhead, but since HCat has the
> same schema benefits as Avro, the performance argument doesn't seem as
> obviously compelling to me at first blush.
>
> Josh
>
> P.S. This is exactly the place to discuss this sort of thing!
>
> On Tue, Mar 13, 2018 at 12:10 PM, Stephen Durfey <sj...@gmail.com>
> wrote:
>
> > I wasn't sure if this was the place to discuss this or on a JIRA. To
> > followup on the work for the HCatSource, I wrote some code to convert an
> > HCatRecord into a specific avro model. This way you can read from HCat,
> but
> > still deal with avro models instead of HCatRecord. It really isn't the
> > ideal path as HCat already does SerDe operations when it does schema
> > resolution. So, it's rather a rather inefficient path. At least with this
> > code it's all in memory pointer moving, rather than additional byte
> > serialization.
> >
> > HCatalog could be used to just find the desired partitions, and then a
> > regular pipeline can be used to read in the avro models. However, you
> then
> > need to know where the data is located and its file format, which is one
> of
> > the biggest benefits of the HCatalog.
> >
> > I'm not really sure if this conversion code in crunch is the right place
> > for it to live long term, as there isn't anything about it that is crunch
> > specific. Hive seems like the better place for it to live. However, I'm
> not
> > sure when they would get around to committing it, and it would likely be
> in
> > a version that is beyond what we support today in Crunch (hive 2.1). So,
> > maybe in crunch short term until it is accepted by hive?
> >
> > If this code is valuable, I can open a JIRA and we can go from there.
> >
> > Code:
> > https://github.com/sjdurfey/crunch/commit/f659dfe06f50862b9f674c1e2dd699
> > a5c53b2b1f
> >
> > The code to look at would be the HCatDecoder, HCatParser, and the various
> > HCatParse contexts.
> >
>
Re: Reading from HCat as Avro
Posted by Josh Wills <jo...@gmail.com>.
The Crunch-specific argument in favor of it, at least from my perspective,
is that a) Crunch is the most Avro-friendly pipeline library around (almost
to a fault, you might argue), and b) a Hive/HCat dependency in Avro itself
would make even less sense, and if you're using Hive SerDe/HQL/UDF, then
you're already bought into Hive's data model anyway and an HCat -> Avro
translator wouldn't make much sense.
Does the conversion buy me anything from a MapReduce perspective? One of my
favorite aspects of Avro has always been the super-fast serialization of
data during the shuffle of MR jobs, even compared to Thrift/Protobuf b/c
the schema allows you to skip so much id overhead, but since HCat has the
same schema benefits as Avro, the performance argument doesn't seem as
obviously compelling to me at first blush.
Josh
P.S. This is exactly the place to discuss this sort of thing!
On Tue, Mar 13, 2018 at 12:10 PM, Stephen Durfey <sj...@gmail.com> wrote:
> I wasn't sure if this was the place to discuss this or on a JIRA. To
> followup on the work for the HCatSource, I wrote some code to convert an
> HCatRecord into a specific avro model. This way you can read from HCat, but
> still deal with avro models instead of HCatRecord. It really isn't the
> ideal path as HCat already does SerDe operations when it does schema
> resolution. So, it's rather a rather inefficient path. At least with this
> code it's all in memory pointer moving, rather than additional byte
> serialization.
>
> HCatalog could be used to just find the desired partitions, and then a
> regular pipeline can be used to read in the avro models. However, you then
> need to know where the data is located and its file format, which is one of
> the biggest benefits of the HCatalog.
>
> I'm not really sure if this conversion code in crunch is the right place
> for it to live long term, as there isn't anything about it that is crunch
> specific. Hive seems like the better place for it to live. However, I'm not
> sure when they would get around to committing it, and it would likely be in
> a version that is beyond what we support today in Crunch (hive 2.1). So,
> maybe in crunch short term until it is accepted by hive?
>
> If this code is valuable, I can open a JIRA and we can go from there.
>
> Code:
> https://github.com/sjdurfey/crunch/commit/f659dfe06f50862b9f674c1e2dd699
> a5c53b2b1f
>
> The code to look at would be the HCatDecoder, HCatParser, and the various
> HCatParse contexts.
>