You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Sanjit Jhala <sj...@gmail.com> on 2010/05/20 01:37:02 UTC
SerDe and Rows
Hi,
I'm trying to write a StorageHandler for Hypertable, to facilitate
Hive-Hypertable integration. Looking at the documentation, it looks like the
SerDe interface deals with reading and writing abstract objects which are
the external data store's equivalent of (Hive) rows. Is this correct, or can
the interface be used to deal with sub-row objects (ie a rowkey + column)?
The reason I ask is that currently the Hypertable API only exposes Cells (a
row is essentially a collection of Cells with the same rowkey) and has no
explicit notion of a row.
-Sanjit
Re: Upgrade the hive version on Cloudera training VM
Posted by Sarah Sproehnle <sa...@cloudera.com>.
Hi Farah,
A new version of the Cloudera VM will be released in the next month or
so. In the meantime, it is fairly easy to upgrade Hive. Download and
extract hive-0.5.0-bin.tar.gz, then set HIVE_HOME and update your
path.
Cheers,
Sarah
On Fri, May 21, 2010 at 3:23 PM, Omer, Farah <fo...@microstrategy.com> wrote:
> Hi,
>
>
>
> I am working on the Cloudera’s Hadoop Training VM that has Hadoop and Hive
> installed in it. I basically connect via the Hive DOBC driver to the front
> end of MicroStrategy BI tool, to run reports. I saw that in my version of
> Hive, I am unable to run queries like:
>
>
>
> CREATE TABLE ZZTA41IFUZBMD000 AS
>
> SELECT a11.REGION_NBR REGION_NBR,
>
> sum(a11.CLE_SLS_DLR) WJXBFS1
>
> FROM region_division a11 LEFT SEMI JOIN lookup_store s21
>
> ON (s21.MARKET_NBR = 2 and s21.STORE_NBR = 1
>
> and s21.REGION_NBR = a11.REGION_NBR)
>
> group by a11.REGION_NBR;
>
>
>
> It errors out with the message of: mismatched input 'SEMI' expecting OUTER
> in join type specifier
>
>
>
> As I downloaded this VM from Cloudera long time ago the hive version may be
> older. I need to upgrade this to 0.5. What would be the best way to go about
> this? Does Cloudera have a VM with the latest hive in it? If not, then can I
> just upgrade, and how may I do so.
>
>
>
> Thanks very much.
>
>
>
> - Farah
>
>
--
get hadoop: cloudera.com/hadoop
online training: cloudera.com/hadoop-training
blog: cloudera.com/blog
twitter: twitter.com/cloudera
Upgrade the hive version on Cloudera training VM
Posted by "Omer, Farah" <fo...@microstrategy.com>.
Hi,
I am working on the Cloudera's Hadoop Training VM that has Hadoop and
Hive installed in it. I basically connect via the Hive DOBC driver to
the front end of MicroStrategy BI tool, to run reports. I saw that in my
version of Hive, I am unable to run queries like:
CREATE TABLE ZZTA41IFUZBMD000 AS
SELECT a11.REGION_NBR REGION_NBR,
sum(a11.CLE_SLS_DLR) WJXBFS1
FROM region_division a11 LEFT SEMI JOIN lookup_store s21
ON (s21.MARKET_NBR = 2 and s21.STORE_NBR = 1
and s21.REGION_NBR = a11.REGION_NBR)
group by a11.REGION_NBR;
It errors out with the message of: mismatched input 'SEMI' expecting
OUTER in join type specifier
As I downloaded this VM from Cloudera long time ago the hive version may
be older. I need to upgrade this to 0.5. What would be the best way to
go about this? Does Cloudera have a VM with the latest hive in it? If
not, then can I just upgrade, and how may I do so.
Thanks very much.
- Farah
Re: SerDe and Rows
Posted by John Sichi <js...@facebook.com>.
On Jun 3, 2010, at 12:36 PM, Sanjit Jhala wrote:
> I'm wondering why the Split class needs to extend FileSplit and also why the InputFormat needs to call FileInputFormat.getInputPaths(job) in getSplits. Is this because of legacy code that needs to be cleaned up or does it get used somewhere?
Both of these are due to legacy code needing cleanup (HIVE-1133). Currently some of the inputformat logic is based on physical paths where it should be based on logical data sources such as partitions instead. This is also the reason why currently we are forced to create an empty directory in the file system corresponding to the name of each non-native table (HIVE-1222).
JVS
Re: SerDe and Rows
Posted by Sanjit Jhala <sj...@gmail.com>.
I'm wondering why the Split class needs to extend FileSplit and also why the
InputFormat needs to call FileInputFormat.getInputPaths(job) in getSplits.
Is this because of legacy code that needs to be cleaned up or does it get
used somewhere?
-Sanjit
On Wed, Jun 2, 2010 at 12:59 PM, Edward Capriolo <ed...@gmail.com>wrote:
>
>
> On Wed, Jun 2, 2010 at 3:17 PM, Sanjit Jhala <sj...@gmail.com> wrote:
>
>> Thanks, that sounds great ! Would love to come to the meetup. Thanks for
>> all the work on the Storage Handlers, its really nifty stuff.
>> I'm getting close on the Hypertable storage handler and will definitely
>> send out pointers once its ready.
>>
>> -Sanjit
>>
>>
>> On Wed, Jun 2, 2010 at 11:56 AM, John Sichi <js...@facebook.com> wrote:
>>
>>> Based on some recent offline discussions, it looks like CloudEra will be
>>> taking the lead on driving the release process for 0.6, so expect to see
>>> some initial plans on that here soon.
>>>
>>> We're thinking of classifying new features and frameworks as stable vs
>>> experimental. For 0.6, items like storage handlers will definitely be
>>> classified as experimental, meaning they'll be there in the code, but
>>> expected to continue to evolve with breaking changes until they are declared
>>> stable in a subsequent release.
>>>
>>> We would also like to start holding monthly Hive developer meetups; it
>>> will be great if someone from hypertable can attend those--it's heartening
>>> to see so much interest in building up a storage handler ecosystem.
>>>
>>> I think the snapshot you reference is fine for trunk development work.
>>>
>>> Regarding thrift, here's info on the version currently being used:
>>>
>>> http://wiki.apache.org/hadoop/Hive/HowToContribute#Generating_Code
>>>
>>> JVS
>>>
>>> On Jun 2, 2010, at 11:19 AM, Sanjit Jhala wrote:
>>>
>>> Any idea when the next Hive release is scheduled for and whether the
>>> Storage Handler code will be included ?
>>>
>>> Also I'm currently using a snapshot from the trunk at commit:
>>>
>>> *commit bf7e3b9cc6c6ceced2dec70f0971ecc91fd0dcb3*
>>> *Author: Namit Jain <na...@apache.org>
>>> Date: Thu May 6 19:05:52 2010 +0000
>>>
>>> HIVE-1317. CombineHiveInputFormat throws exception when partition
>>> name contains special characte
>>> (Ning Zhang via namit)
>>>
>>> git-svn-id:
>>> https://svn.apache.org/repos/asf/hadoop/hive/trunk@94186013f79535-47bb-0310-9956-ff
>>> *
>>>
>>>
>>> Is this a reasonably stable commit or would you suggest another ? Also
>>> how do I figure out the corresponding Thrift version ?
>>>
>>> -Sanjit
>>>
>>>
>>>
>>>
>>> On Tue, Jun 1, 2010 at 5:36 PM, John Sichi <js...@facebook.com> wrote:
>>>
>>>> On Jun 1, 2010, at 4:45 PM, Sanjit Jhala wrote:
>>>>
>>>> > That looks cool. On a different note, it looks like the
>>>> HiveStorageHandler is based on the old Hadoop "mapred" interface. Any idea
>>>> when you plan to migrate to the "mapreduce" interface?
>>>>
>>>>
>>>> This one would be painful to do with shims, so I think it has to wait
>>>> until we drop support entirely for pre-0.20 Hadoop versions on Hive trunk.
>>>> For Facebook, we may be ready for that within a few months; I'm not sure
>>>> about other Hive users.
>>>>
>>>> JVS
>>>>
>>>>
>>>
>>>
>> IMHO...
>
> Trunk has lot of features that 5.0 does not have. All (most) of the
> development for hive happens on the trunk. Trunk changes 2-3 times a week so
> it is a moving target.
>
>
>
> Hive is all userspace code, anyone who understands that can have 100
> different versions of hive in their home directory configured to the same
> metastore and hdfs.
>
> I currently have hive 5.0 latest release installed on the system path.
> /usr/bin/hive6 -> another hive install of (trunk)
>
> This gives me the best of both worlds. Users can pick and chose the hive
> they want to run with. I am really not caught up in releases. They are good
> things but in general I can not wait for them.
>
> Edward
>
>
>
Re: SerDe and Rows
Posted by Edward Capriolo <ed...@gmail.com>.
On Wed, Jun 2, 2010 at 3:17 PM, Sanjit Jhala <sj...@gmail.com> wrote:
> Thanks, that sounds great ! Would love to come to the meetup. Thanks for
> all the work on the Storage Handlers, its really nifty stuff.
> I'm getting close on the Hypertable storage handler and will definitely
> send out pointers once its ready.
>
> -Sanjit
>
>
> On Wed, Jun 2, 2010 at 11:56 AM, John Sichi <js...@facebook.com> wrote:
>
>> Based on some recent offline discussions, it looks like CloudEra will be
>> taking the lead on driving the release process for 0.6, so expect to see
>> some initial plans on that here soon.
>>
>> We're thinking of classifying new features and frameworks as stable vs
>> experimental. For 0.6, items like storage handlers will definitely be
>> classified as experimental, meaning they'll be there in the code, but
>> expected to continue to evolve with breaking changes until they are declared
>> stable in a subsequent release.
>>
>> We would also like to start holding monthly Hive developer meetups; it
>> will be great if someone from hypertable can attend those--it's heartening
>> to see so much interest in building up a storage handler ecosystem.
>>
>> I think the snapshot you reference is fine for trunk development work.
>>
>> Regarding thrift, here's info on the version currently being used:
>>
>> http://wiki.apache.org/hadoop/Hive/HowToContribute#Generating_Code
>>
>> JVS
>>
>> On Jun 2, 2010, at 11:19 AM, Sanjit Jhala wrote:
>>
>> Any idea when the next Hive release is scheduled for and whether the
>> Storage Handler code will be included ?
>>
>> Also I'm currently using a snapshot from the trunk at commit:
>>
>> *commit bf7e3b9cc6c6ceced2dec70f0971ecc91fd0dcb3*
>> *Author: Namit Jain <na...@apache.org>
>> Date: Thu May 6 19:05:52 2010 +0000
>>
>> HIVE-1317. CombineHiveInputFormat throws exception when partition name
>> contains special characte
>> (Ning Zhang via namit)
>>
>> git-svn-id: https://svn.apache.org/repos/asf/hadoop/hive/trunk@94186013f79535-47bb-0310-9956-ff
>> *
>>
>>
>> Is this a reasonably stable commit or would you suggest another ? Also how
>> do I figure out the corresponding Thrift version ?
>>
>> -Sanjit
>>
>>
>>
>>
>> On Tue, Jun 1, 2010 at 5:36 PM, John Sichi <js...@facebook.com> wrote:
>>
>>> On Jun 1, 2010, at 4:45 PM, Sanjit Jhala wrote:
>>>
>>> > That looks cool. On a different note, it looks like the
>>> HiveStorageHandler is based on the old Hadoop "mapred" interface. Any idea
>>> when you plan to migrate to the "mapreduce" interface?
>>>
>>>
>>> This one would be painful to do with shims, so I think it has to wait
>>> until we drop support entirely for pre-0.20 Hadoop versions on Hive trunk.
>>> For Facebook, we may be ready for that within a few months; I'm not sure
>>> about other Hive users.
>>>
>>> JVS
>>>
>>>
>>
>>
> IMHO...
Trunk has lot of features that 5.0 does not have. All (most) of the
development for hive happens on the trunk. Trunk changes 2-3 times a week so
it is a moving target.
Hive is all userspace code, anyone who understands that can have 100
different versions of hive in their home directory configured to the same
metastore and hdfs.
I currently have hive 5.0 latest release installed on the system path.
/usr/bin/hive6 -> another hive install of (trunk)
This gives me the best of both worlds. Users can pick and chose the hive
they want to run with. I am really not caught up in releases. They are good
things but in general I can not wait for them.
Edward
Re: SerDe and Rows
Posted by Sanjit Jhala <sj...@gmail.com>.
Thanks, that sounds great ! Would love to come to the meetup. Thanks for all
the work on the Storage Handlers, its really nifty stuff.
I'm getting close on the Hypertable storage handler and will definitely send
out pointers once its ready.
-Sanjit
On Wed, Jun 2, 2010 at 11:56 AM, John Sichi <js...@facebook.com> wrote:
> Based on some recent offline discussions, it looks like CloudEra will be
> taking the lead on driving the release process for 0.6, so expect to see
> some initial plans on that here soon.
>
> We're thinking of classifying new features and frameworks as stable vs
> experimental. For 0.6, items like storage handlers will definitely be
> classified as experimental, meaning they'll be there in the code, but
> expected to continue to evolve with breaking changes until they are declared
> stable in a subsequent release.
>
> We would also like to start holding monthly Hive developer meetups; it will
> be great if someone from hypertable can attend those--it's heartening to see
> so much interest in building up a storage handler ecosystem.
>
> I think the snapshot you reference is fine for trunk development work.
>
> Regarding thrift, here's info on the version currently being used:
>
> http://wiki.apache.org/hadoop/Hive/HowToContribute#Generating_Code
>
> JVS
>
> On Jun 2, 2010, at 11:19 AM, Sanjit Jhala wrote:
>
> Any idea when the next Hive release is scheduled for and whether the
> Storage Handler code will be included ?
>
> Also I'm currently using a snapshot from the trunk at commit:
>
> *commit bf7e3b9cc6c6ceced2dec70f0971ecc91fd0dcb3*
> *Author: Namit Jain <na...@apache.org>
> Date: Thu May 6 19:05:52 2010 +0000
>
> HIVE-1317. CombineHiveInputFormat throws exception when partition name
> contains special characte
> (Ning Zhang via namit)
>
> git-svn-id: https://svn.apache.org/repos/asf/hadoop/hive/trunk@94186013f79535-47bb-0310-9956-ff
> *
>
>
> Is this a reasonably stable commit or would you suggest another ? Also how
> do I figure out the corresponding Thrift version ?
>
> -Sanjit
>
>
>
>
> On Tue, Jun 1, 2010 at 5:36 PM, John Sichi <js...@facebook.com> wrote:
>
>> On Jun 1, 2010, at 4:45 PM, Sanjit Jhala wrote:
>>
>> > That looks cool. On a different note, it looks like the
>> HiveStorageHandler is based on the old Hadoop "mapred" interface. Any idea
>> when you plan to migrate to the "mapreduce" interface?
>>
>>
>> This one would be painful to do with shims, so I think it has to wait
>> until we drop support entirely for pre-0.20 Hadoop versions on Hive trunk.
>> For Facebook, we may be ready for that within a few months; I'm not sure
>> about other Hive users.
>>
>> JVS
>>
>>
>
>
Re: SerDe and Rows
Posted by John Sichi <js...@facebook.com>.
Based on some recent offline discussions, it looks like CloudEra will be taking the lead on driving the release process for 0.6, so expect to see some initial plans on that here soon.
We're thinking of classifying new features and frameworks as stable vs experimental. For 0.6, items like storage handlers will definitely be classified as experimental, meaning they'll be there in the code, but expected to continue to evolve with breaking changes until they are declared stable in a subsequent release.
We would also like to start holding monthly Hive developer meetups; it will be great if someone from hypertable can attend those--it's heartening to see so much interest in building up a storage handler ecosystem.
I think the snapshot you reference is fine for trunk development work.
Regarding thrift, here's info on the version currently being used:
http://wiki.apache.org/hadoop/Hive/HowToContribute#Generating_Code
JVS
On Jun 2, 2010, at 11:19 AM, Sanjit Jhala wrote:
Any idea when the next Hive release is scheduled for and whether the Storage Handler code will be included ?
Also I'm currently using a snapshot from the trunk at commit:
commit bf7e3b9cc6c6ceced2dec70f0971ecc91fd0dcb3
Author: Namit Jain <na...@apache.org>>
Date: Thu May 6 19:05:52 2010 +0000
HIVE-1317. CombineHiveInputFormat throws exception when partition name contains special characte
(Ning Zhang via namit)
git-svn-id: https://svn.apache.org/repos/asf/hadoop/hive/trunk@941860 13f79535-47bb-0310-9956-ff
Is this a reasonably stable commit or would you suggest another ? Also how do I figure out the corresponding Thrift version ?
-Sanjit
On Tue, Jun 1, 2010 at 5:36 PM, John Sichi <js...@facebook.com>> wrote:
On Jun 1, 2010, at 4:45 PM, Sanjit Jhala wrote:
> That looks cool. On a different note, it looks like the HiveStorageHandler is based on the old Hadoop "mapred" interface. Any idea when you plan to migrate to the "mapreduce" interface?
This one would be painful to do with shims, so I think it has to wait until we drop support entirely for pre-0.20 Hadoop versions on Hive trunk. For Facebook, we may be ready for that within a few months; I'm not sure about other Hive users.
JVS
Re: SerDe and Rows
Posted by Sanjit Jhala <sj...@gmail.com>.
Any idea when the next Hive release is scheduled for and whether the Storage
Handler code will be included ?
Also I'm currently using a snapshot from the trunk at commit:
*commit bf7e3b9cc6c6ceced2dec70f0971ecc91fd0dcb3*
*Author: Namit Jain <na...@apache.org>
Date: Thu May 6 19:05:52 2010 +0000
HIVE-1317. CombineHiveInputFormat throws exception when partition name
contains special characte
(Ning Zhang via namit)
git-svn-id:
https://svn.apache.org/repos/asf/hadoop/hive/trunk@94186013f79535-47bb-0310-9956-ff
*
Is this a reasonably stable commit or would you suggest another ? Also how
do I figure out the corresponding Thrift version ?
-Sanjit
On Tue, Jun 1, 2010 at 5:36 PM, John Sichi <js...@facebook.com> wrote:
> On Jun 1, 2010, at 4:45 PM, Sanjit Jhala wrote:
>
> > That looks cool. On a different note, it looks like the
> HiveStorageHandler is based on the old Hadoop "mapred" interface. Any idea
> when you plan to migrate to the "mapreduce" interface?
>
>
> This one would be painful to do with shims, so I think it has to wait until
> we drop support entirely for pre-0.20 Hadoop versions on Hive trunk. For
> Facebook, we may be ready for that within a few months; I'm not sure about
> other Hive users.
>
> JVS
>
>
Re: SerDe and Rows
Posted by John Sichi <js...@facebook.com>.
On Jun 1, 2010, at 4:45 PM, Sanjit Jhala wrote:
> That looks cool. On a different note, it looks like the HiveStorageHandler is based on the old Hadoop "mapred" interface. Any idea when you plan to migrate to the "mapreduce" interface?
This one would be painful to do with shims, so I think it has to wait until we drop support entirely for pre-0.20 Hadoop versions on Hive trunk. For Facebook, we may be ready for that within a few months; I'm not sure about other Hive users.
JVS
Re: SerDe and Rows
Posted by Sanjit Jhala <sj...@gmail.com>.
That looks cool. On a different note, it looks like the HiveStorageHandler
is based on the old Hadoop "mapred" interface. Any idea when you plan to
migrate to the "mapreduce" interface?
-Sanjit
On Tue, Jun 1, 2010 at 12:26 PM, John Sichi <js...@facebook.com> wrote:
> On May 28, 2010, at 3:49 PM, Sanjit Jhala wrote:
>
> > John, theres some logic in the helper serialize method to serialize lists
> and structs. Is this used currently? I was under the impression that maps
> and primitives are the only types currently supported by the connector.
>
>
> Yes, this logic is working. I just now tested it interactively (see below)
> and will add a corresponding unit test when I work on HIVE-1245.
>
> I'm not sure what is going on with the JSON-vs-delimited stuff; in my test
> it looks like it is coming out as delimited based on what I see from the
> HBase side. There is a setUseJSONSerialize method but currently nothing
> invokes it; it would make sense to include this in the HIVE-1245 work as
> part of controlling how values are stored within HBase.
>
> JVS
>
> ----
>
> hive> CREATE TABLE complex(
> > key string,
> > a array<string>,
> > s struct<col1 : int, col2 : int>)
> > STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> > WITH SERDEPROPERTIES (
> > "hbase.columns.mapping" = "cf:a, cf:s"
> > );
> OK
> hive>
> > INSERT OVERWRITE TABLE complex
> > SELECT bar, array('x', 'y', 'z'), struct(100, 200)
> > FROM pokes
> > WHERE foo=497;
> ...
> OK
> hive>
> > SELECT * FROM complex;
> OK
> val_497 ["x","y","z"] {"col1":100,"col2":200}
>
> hbase(main):003:0> scan 'complex'
> ROW COLUMN+CELL
> val_497 column= cf:s, timestamp=1275419258650,
> value=100\x02200
> val_497 column=cf:a, timestamp=1275419258650,
> value=x\x02y\x02z
> 1 row(s) in 1.0250 seconds
>
>
Re: SerDe and Rows
Posted by John Sichi <js...@facebook.com>.
On May 28, 2010, at 3:49 PM, Sanjit Jhala wrote:
> John, theres some logic in the helper serialize method to serialize lists and structs. Is this used currently? I was under the impression that maps and primitives are the only types currently supported by the connector.
Yes, this logic is working. I just now tested it interactively (see below) and will add a corresponding unit test when I work on HIVE-1245.
I'm not sure what is going on with the JSON-vs-delimited stuff; in my test it looks like it is coming out as delimited based on what I see from the HBase side. There is a setUseJSONSerialize method but currently nothing invokes it; it would make sense to include this in the HIVE-1245 work as part of controlling how values are stored within HBase.
JVS
----
hive> CREATE TABLE complex(
> key string,
> a array<string>,
> s struct<col1 : int, col2 : int>)
> STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> WITH SERDEPROPERTIES (
> "hbase.columns.mapping" = "cf:a, cf:s"
> );
OK
hive>
> INSERT OVERWRITE TABLE complex
> SELECT bar, array('x', 'y', 'z'), struct(100, 200)
> FROM pokes
> WHERE foo=497;
...
OK
hive>
> SELECT * FROM complex;
OK
val_497 ["x","y","z"] {"col1":100,"col2":200}
hbase(main):003:0> scan 'complex'
ROW COLUMN+CELL
val_497 column= cf:s, timestamp=1275419258650, value=100\x02200
val_497 column=cf:a, timestamp=1275419258650, value=x\x02y\x02z
1 row(s) in 1.0250 seconds
Re: SerDe and Rows
Posted by Sanjit Jhala <sj...@gmail.com>.
John, theres some logic in the helper serialize method to serialize lists
and structs. Is this used currently? I was under the impression that maps
and primitives are the only types currently supported by the connector.
-Sanjit
On Thu, May 27, 2010 at 9:53 PM, Sanjit Jhala <sj...@gmail.com> wrote:
> I completely missed that. Being public threw me off.
>
> -Sanjit
>
>
> On Thu, May 27, 2010 at 9:43 PM, John Sichi <js...@facebook.com> wrote:
>
>> It is a helper method called by the main serialize method; it shouldn't
>> actually be public at all. Note that as an example, this code is good for
>> understanding the interactions, but it is not a good example for code
>> structure or performance; I'm working on refactoring it.
>>
>> JVS
>>
>> ________________________________________
>> From: Sanjit Jhala [sjhalaz@gmail.com]
>> Sent: Thursday, May 27, 2010 9:29 PM
>> To: hive-user@hadoop.apache.org
>> Subject: Re: SerDe and Rows
>>
>> Btw whats the purpose of the alternative serialize API in HBaseSerDe.java
>> ?
>> public static boolean serialize(ByteStream.Output out, Object obj,
>> ObjectInspector objInspector, byte[] separators, int level, Text
>> nullSequence, boolean escaped, byte escapeChar, boolean[] needsEscape)
>>
>> It doesn't look like this API is part of the SerDe interface and I'm
>> wondering where it gets called from?
>>
>> -Sanjit
>>
>> On Wed, May 26, 2010 at 9:38 AM, Sanjit Jhala <sjhalaz@gmail.com<mailto:
>> sjhalaz@gmail.com>> wrote:
>> Yes, I think you can do this as long as all inserts follow the scheme you
>> mention. In fact you can even think of having point in time row versioning
>> defined at some timestamp T as the collection of the latest versions of
>> cells with timestamps <= T. As you rightly point out, its all about the use
>> case :)
>>
>> -Sanjit
>>
>>
>>
>> On Tue, May 25, 2010 at 7:39 PM, John Sichi <jsichi@facebook.com<mailto:
>> jsichi@facebook.com>> wrote:
>> Sorry for my slow response; answers below.
>>
>> On May 20, 2010, at 5:23 PM, Sanjit Jhala wrote:
>>
>> > Thanks John, that does look quite interesting. It looks like in addition
>> to containing a bunch of cells, the row class needs to provide some
>> mechanism (eg a map) to efficiently lookup the cell corresponding to a given
>> qualified column (ie column family + qualifier). In the case where a Hive
>> column matches an entire column family, do you use this same map using the
>> property that the column family is a prefix of the map key or is there an
>> additional map that maps the column family to a set of qualifiers or
>> directly to a set of cells ?
>>
>> There is a separate map (LazyHBaseCellMap). LazyHBaseRow instantiates
>> this for Hive column values which correspond to HBase column families.
>>
>> > The wiki also indicates that in future multiple versions of a cell could
>> be exposed to the storage handler since Hive can deal with non-unique rows.
>> I can definitely see how you should be able to store non-unique Hive rows
>> in Hypertable (since Hypertable supports multi-versioned cells), however
>> since the fundamental unit of storage in the BigTable design is a cell, I
>> don't understand how you propose to map multiple cell versions back to
>> non-unique Hive rows. Maybe you're thinking of mapping them to a single Hive
>> row, where the columns are of the List type? And then maybe the query
>> language allows you to filter by the first, last or any value in the list?
>>
>>
>> Yeah, I realized this recently when I started thinking about it again :)
>>
>> Exposing per-cell timestamps is possible, and there are a number of ways
>> to do it, including the one you mention. But they're all unwieldy, so we
>> should probably defer them until there's a very good use case.
>>
>> A simpler scheme I'm thinking about is to map a Hive partition to a
>> particular timestamp. Then for queries, this will specify a point-in-time
>> (we would need to validate that only equality predicates are used on the
>> partition key since returning multiple versions of a row isn't well-defined
>> as you correctly point out). For inserts, all cells created would get the
>> same timestamp. Maybe this would cover the majority of use-cases?
>>
>> JVS
>>
>>
>>
>>
>
Re: SerDe and Rows
Posted by Sanjit Jhala <sj...@gmail.com>.
I completely missed that. Being public threw me off.
-Sanjit
On Thu, May 27, 2010 at 9:43 PM, John Sichi <js...@facebook.com> wrote:
> It is a helper method called by the main serialize method; it shouldn't
> actually be public at all. Note that as an example, this code is good for
> understanding the interactions, but it is not a good example for code
> structure or performance; I'm working on refactoring it.
>
> JVS
>
> ________________________________________
> From: Sanjit Jhala [sjhalaz@gmail.com]
> Sent: Thursday, May 27, 2010 9:29 PM
> To: hive-user@hadoop.apache.org
> Subject: Re: SerDe and Rows
>
> Btw whats the purpose of the alternative serialize API in HBaseSerDe.java ?
> public static boolean serialize(ByteStream.Output out, Object obj,
> ObjectInspector objInspector, byte[] separators, int level, Text
> nullSequence, boolean escaped, byte escapeChar, boolean[] needsEscape)
>
> It doesn't look like this API is part of the SerDe interface and I'm
> wondering where it gets called from?
>
> -Sanjit
>
> On Wed, May 26, 2010 at 9:38 AM, Sanjit Jhala <sjhalaz@gmail.com<mailto:
> sjhalaz@gmail.com>> wrote:
> Yes, I think you can do this as long as all inserts follow the scheme you
> mention. In fact you can even think of having point in time row versioning
> defined at some timestamp T as the collection of the latest versions of
> cells with timestamps <= T. As you rightly point out, its all about the use
> case :)
>
> -Sanjit
>
>
>
> On Tue, May 25, 2010 at 7:39 PM, John Sichi <jsichi@facebook.com<mailto:
> jsichi@facebook.com>> wrote:
> Sorry for my slow response; answers below.
>
> On May 20, 2010, at 5:23 PM, Sanjit Jhala wrote:
>
> > Thanks John, that does look quite interesting. It looks like in addition
> to containing a bunch of cells, the row class needs to provide some
> mechanism (eg a map) to efficiently lookup the cell corresponding to a given
> qualified column (ie column family + qualifier). In the case where a Hive
> column matches an entire column family, do you use this same map using the
> property that the column family is a prefix of the map key or is there an
> additional map that maps the column family to a set of qualifiers or
> directly to a set of cells ?
>
> There is a separate map (LazyHBaseCellMap). LazyHBaseRow instantiates this
> for Hive column values which correspond to HBase column families.
>
> > The wiki also indicates that in future multiple versions of a cell could
> be exposed to the storage handler since Hive can deal with non-unique rows.
> I can definitely see how you should be able to store non-unique Hive rows
> in Hypertable (since Hypertable supports multi-versioned cells), however
> since the fundamental unit of storage in the BigTable design is a cell, I
> don't understand how you propose to map multiple cell versions back to
> non-unique Hive rows. Maybe you're thinking of mapping them to a single Hive
> row, where the columns are of the List type? And then maybe the query
> language allows you to filter by the first, last or any value in the list?
>
>
> Yeah, I realized this recently when I started thinking about it again :)
>
> Exposing per-cell timestamps is possible, and there are a number of ways to
> do it, including the one you mention. But they're all unwieldy, so we
> should probably defer them until there's a very good use case.
>
> A simpler scheme I'm thinking about is to map a Hive partition to a
> particular timestamp. Then for queries, this will specify a point-in-time
> (we would need to validate that only equality predicates are used on the
> partition key since returning multiple versions of a row isn't well-defined
> as you correctly point out). For inserts, all cells created would get the
> same timestamp. Maybe this would cover the majority of use-cases?
>
> JVS
>
>
>
>
RE: SerDe and Rows
Posted by John Sichi <js...@facebook.com>.
It is a helper method called by the main serialize method; it shouldn't actually be public at all. Note that as an example, this code is good for understanding the interactions, but it is not a good example for code structure or performance; I'm working on refactoring it.
JVS
________________________________________
From: Sanjit Jhala [sjhalaz@gmail.com]
Sent: Thursday, May 27, 2010 9:29 PM
To: hive-user@hadoop.apache.org
Subject: Re: SerDe and Rows
Btw whats the purpose of the alternative serialize API in HBaseSerDe.java ?
public static boolean serialize(ByteStream.Output out, Object obj, ObjectInspector objInspector, byte[] separators, int level, Text nullSequence, boolean escaped, byte escapeChar, boolean[] needsEscape)
It doesn't look like this API is part of the SerDe interface and I'm wondering where it gets called from?
-Sanjit
On Wed, May 26, 2010 at 9:38 AM, Sanjit Jhala <sj...@gmail.com>> wrote:
Yes, I think you can do this as long as all inserts follow the scheme you mention. In fact you can even think of having point in time row versioning defined at some timestamp T as the collection of the latest versions of cells with timestamps <= T. As you rightly point out, its all about the use case :)
-Sanjit
On Tue, May 25, 2010 at 7:39 PM, John Sichi <js...@facebook.com>> wrote:
Sorry for my slow response; answers below.
On May 20, 2010, at 5:23 PM, Sanjit Jhala wrote:
> Thanks John, that does look quite interesting. It looks like in addition to containing a bunch of cells, the row class needs to provide some mechanism (eg a map) to efficiently lookup the cell corresponding to a given qualified column (ie column family + qualifier). In the case where a Hive column matches an entire column family, do you use this same map using the property that the column family is a prefix of the map key or is there an additional map that maps the column family to a set of qualifiers or directly to a set of cells ?
There is a separate map (LazyHBaseCellMap). LazyHBaseRow instantiates this for Hive column values which correspond to HBase column families.
> The wiki also indicates that in future multiple versions of a cell could be exposed to the storage handler since Hive can deal with non-unique rows. I can definitely see how you should be able to store non-unique Hive rows in Hypertable (since Hypertable supports multi-versioned cells), however since the fundamental unit of storage in the BigTable design is a cell, I don't understand how you propose to map multiple cell versions back to non-unique Hive rows. Maybe you're thinking of mapping them to a single Hive row, where the columns are of the List type? And then maybe the query language allows you to filter by the first, last or any value in the list?
Yeah, I realized this recently when I started thinking about it again :)
Exposing per-cell timestamps is possible, and there are a number of ways to do it, including the one you mention. But they're all unwieldy, so we should probably defer them until there's a very good use case.
A simpler scheme I'm thinking about is to map a Hive partition to a particular timestamp. Then for queries, this will specify a point-in-time (we would need to validate that only equality predicates are used on the partition key since returning multiple versions of a row isn't well-defined as you correctly point out). For inserts, all cells created would get the same timestamp. Maybe this would cover the majority of use-cases?
JVS
Re: SerDe and Rows
Posted by Sanjit Jhala <sj...@gmail.com>.
Btw whats the purpose of the alternative serialize API in HBaseSerDe.java ?*
public static boolean serialize(ByteStream.Output out, Object obj,
ObjectInspector objInspector, byte[] separators, int level, Text
nullSequence, boolean escaped, byte escapeChar, boolean[] needsEscape) *
It doesn't look like this API is part of the SerDe interface and I'm
wondering where it gets called from?
-Sanjit
On Wed, May 26, 2010 at 9:38 AM, Sanjit Jhala <sj...@gmail.com> wrote:
> Yes, I think you can do this as long as all inserts follow the scheme you
> mention. In fact you can even think of having point in time row versioning
> defined at some timestamp T as the collection of the latest versions of
> cells with timestamps <= T. As you rightly point out, its all about the use
> case :)
>
> -Sanjit
>
>
>
> On Tue, May 25, 2010 at 7:39 PM, John Sichi <js...@facebook.com> wrote:
>
>> Sorry for my slow response; answers below.
>>
>> On May 20, 2010, at 5:23 PM, Sanjit Jhala wrote:
>>
>> > Thanks John, that does look quite interesting. It looks like in addition
>> to containing a bunch of cells, the row class needs to provide some
>> mechanism (eg a map) to efficiently lookup the cell corresponding to a given
>> qualified column (ie column family + qualifier). In the case where a Hive
>> column matches an entire column family, do you use this same map using the
>> property that the column family is a prefix of the map key or is there an
>> additional map that maps the column family to a set of qualifiers or
>> directly to a set of cells ?
>>
>> There is a separate map (LazyHBaseCellMap). LazyHBaseRow instantiates
>> this for Hive column values which correspond to HBase column families.
>>
>> > The wiki also indicates that in future multiple versions of a cell could
>> be exposed to the storage handler since Hive can deal with non-unique rows.
>> I can definitely see how you should be able to store non-unique Hive rows
>> in Hypertable (since Hypertable supports multi-versioned cells), however
>> since the fundamental unit of storage in the BigTable design is a cell, I
>> don't understand how you propose to map multiple cell versions back to
>> non-unique Hive rows. Maybe you're thinking of mapping them to a single Hive
>> row, where the columns are of the List type? And then maybe the query
>> language allows you to filter by the first, last or any value in the list?
>>
>>
>> Yeah, I realized this recently when I started thinking about it again :)
>>
>> Exposing per-cell timestamps is possible, and there are a number of ways
>> to do it, including the one you mention. But they're all unwieldy, so we
>> should probably defer them until there's a very good use case.
>>
>> A simpler scheme I'm thinking about is to map a Hive partition to a
>> particular timestamp. Then for queries, this will specify a point-in-time
>> (we would need to validate that only equality predicates are used on the
>> partition key since returning multiple versions of a row isn't well-defined
>> as you correctly point out). For inserts, all cells created would get the
>> same timestamp. Maybe this would cover the majority of use-cases?
>>
>> JVS
>>
>>
>
Re: SerDe and Rows
Posted by Sanjit Jhala <sj...@gmail.com>.
Yes, I think you can do this as long as all inserts follow the scheme you
mention. In fact you can even think of having point in time row versioning
defined at some timestamp T as the collection of the latest versions of
cells with timestamps <= T. As you rightly point out, its all about the use
case :)
-Sanjit
On Tue, May 25, 2010 at 7:39 PM, John Sichi <js...@facebook.com> wrote:
> Sorry for my slow response; answers below.
>
> On May 20, 2010, at 5:23 PM, Sanjit Jhala wrote:
>
> > Thanks John, that does look quite interesting. It looks like in addition
> to containing a bunch of cells, the row class needs to provide some
> mechanism (eg a map) to efficiently lookup the cell corresponding to a given
> qualified column (ie column family + qualifier). In the case where a Hive
> column matches an entire column family, do you use this same map using the
> property that the column family is a prefix of the map key or is there an
> additional map that maps the column family to a set of qualifiers or
> directly to a set of cells ?
>
> There is a separate map (LazyHBaseCellMap). LazyHBaseRow instantiates this
> for Hive column values which correspond to HBase column families.
>
> > The wiki also indicates that in future multiple versions of a cell could
> be exposed to the storage handler since Hive can deal with non-unique rows.
> I can definitely see how you should be able to store non-unique Hive rows
> in Hypertable (since Hypertable supports multi-versioned cells), however
> since the fundamental unit of storage in the BigTable design is a cell, I
> don't understand how you propose to map multiple cell versions back to
> non-unique Hive rows. Maybe you're thinking of mapping them to a single Hive
> row, where the columns are of the List type? And then maybe the query
> language allows you to filter by the first, last or any value in the list?
>
>
> Yeah, I realized this recently when I started thinking about it again :)
>
> Exposing per-cell timestamps is possible, and there are a number of ways to
> do it, including the one you mention. But they're all unwieldy, so we
> should probably defer them until there's a very good use case.
>
> A simpler scheme I'm thinking about is to map a Hive partition to a
> particular timestamp. Then for queries, this will specify a point-in-time
> (we would need to validate that only equality predicates are used on the
> partition key since returning multiple versions of a row isn't well-defined
> as you correctly point out). For inserts, all cells created would get the
> same timestamp. Maybe this would cover the majority of use-cases?
>
> JVS
>
>
Re: SerDe and Rows
Posted by John Sichi <js...@facebook.com>.
Sorry for my slow response; answers below.
On May 20, 2010, at 5:23 PM, Sanjit Jhala wrote:
> Thanks John, that does look quite interesting. It looks like in addition to containing a bunch of cells, the row class needs to provide some mechanism (eg a map) to efficiently lookup the cell corresponding to a given qualified column (ie column family + qualifier). In the case where a Hive column matches an entire column family, do you use this same map using the property that the column family is a prefix of the map key or is there an additional map that maps the column family to a set of qualifiers or directly to a set of cells ?
There is a separate map (LazyHBaseCellMap). LazyHBaseRow instantiates this for Hive column values which correspond to HBase column families.
> The wiki also indicates that in future multiple versions of a cell could be exposed to the storage handler since Hive can deal with non-unique rows. I can definitely see how you should be able to store non-unique Hive rows in Hypertable (since Hypertable supports multi-versioned cells), however since the fundamental unit of storage in the BigTable design is a cell, I don't understand how you propose to map multiple cell versions back to non-unique Hive rows. Maybe you're thinking of mapping them to a single Hive row, where the columns are of the List type? And then maybe the query language allows you to filter by the first, last or any value in the list?
Yeah, I realized this recently when I started thinking about it again :)
Exposing per-cell timestamps is possible, and there are a number of ways to do it, including the one you mention. But they're all unwieldy, so we should probably defer them until there's a very good use case.
A simpler scheme I'm thinking about is to map a Hive partition to a particular timestamp. Then for queries, this will specify a point-in-time (we would need to validate that only equality predicates are used on the partition key since returning multiple versions of a row isn't well-defined as you correctly point out). For inserts, all cells created would get the same timestamp. Maybe this would cover the majority of use-cases?
JVS
Re: SerDe and Rows
Posted by Sanjit Jhala <sj...@gmail.com>.
Thanks John, that does look quite interesting. It looks like in addition to
containing a bunch of cells, the row class needs to provide some mechanism
(eg a map) to efficiently lookup the cell corresponding to a given qualified
column (ie column family + qualifier). In the case where a Hive column
matches an entire column family, do you use this same map using the property
that the column family is a prefix of the map key or is there an additional
map that maps the column family to a set of qualifiers or directly to a set
of cells ?
The wiki also indicates that in future multiple versions of a cell could be
exposed to the storage handler since Hive can deal with non-unique rows. I
can definitely see how you should be able to store non-unique Hive rows in
Hypertable (since Hypertable supports multi-versioned cells), however since
the fundamental unit of storage in the BigTable design is a cell, I don't
understand how you propose to map multiple cell versions back to non-unique
Hive rows. Maybe you're thinking of mapping them to a single Hive row, where
the columns are of the List type? And then maybe the query language allows
you to filter by the first, last or any value in the list?
-Sanjit
On Wed, May 19, 2010 at 6:26 PM, John Sichi <js...@facebook.com> wrote:
> That's correct. In order to map the data into the relational world, the
> storage handler will need to put together references for all of the cells
> for a given row and return a reference to that. Take a look at the HBase
> handler to see how to do that in a lazy fashion if that makes sense for you.
>
> JVS
>
> On May 19, 2010, at 6:00 PM, Sanjit Jhala wrote:
>
> Thinking about this a bit more I realize the Input and Output formats have
> to have some notion of rows for any kind of filtering, grouping etc to work.
>
> -Sanjit
>
> On Wed, May 19, 2010 at 4:37 PM, Sanjit Jhala <sj...@gmail.com> wrote:
>
>> Hi,
>>
>> I'm trying to write a StorageHandler for Hypertable, to facilitate
>> Hive-Hypertable integration. Looking at the documentation, it looks like the
>> SerDe interface deals with reading and writing abstract objects which are
>> the external data store's equivalent of (Hive) rows. Is this correct, or can
>> the interface be used to deal with sub-row objects (ie a rowkey + column)?
>> The reason I ask is that currently the Hypertable API only exposes Cells (a
>> row is essentially a collection of Cells with the same rowkey) and has no
>> explicit notion of a row.
>>
>> -Sanjit
>>
>
>
>
Re: SerDe and Rows
Posted by John Sichi <js...@facebook.com>.
That's correct. In order to map the data into the relational world, the storage handler will need to put together references for all of the cells for a given row and return a reference to that. Take a look at the HBase handler to see how to do that in a lazy fashion if that makes sense for you.
JVS
On May 19, 2010, at 6:00 PM, Sanjit Jhala wrote:
Thinking about this a bit more I realize the Input and Output formats have to have some notion of rows for any kind of filtering, grouping etc to work.
-Sanjit
On Wed, May 19, 2010 at 4:37 PM, Sanjit Jhala <sj...@gmail.com>> wrote:
Hi,
I'm trying to write a StorageHandler for Hypertable, to facilitate Hive-Hypertable integration. Looking at the documentation, it looks like the SerDe interface deals with reading and writing abstract objects which are the external data store's equivalent of (Hive) rows. Is this correct, or can the interface be used to deal with sub-row objects (ie a rowkey + column)? The reason I ask is that currently the Hypertable API only exposes Cells (a row is essentially a collection of Cells with the same rowkey) and has no explicit notion of a row.
-Sanjit
Re: SerDe and Rows
Posted by Sanjit Jhala <sj...@gmail.com>.
Thinking about this a bit more I realize the Input and Output formats have
to have some notion of rows for any kind of filtering, grouping etc to work.
-Sanjit
On Wed, May 19, 2010 at 4:37 PM, Sanjit Jhala <sj...@gmail.com> wrote:
> Hi,
>
> I'm trying to write a StorageHandler for Hypertable, to facilitate
> Hive-Hypertable integration. Looking at the documentation, it looks like the
> SerDe interface deals with reading and writing abstract objects which are
> the external data store's equivalent of (Hive) rows. Is this correct, or can
> the interface be used to deal with sub-row objects (ie a rowkey + column)?
> The reason I ask is that currently the Hypertable API only exposes Cells (a
> row is essentially a collection of Cells with the same rowkey) and has no
> explicit notion of a row.
>
> -Sanjit
>