You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by "kulkarni.swarnim@gmail.com" <ku...@gmail.com> on 2012/05/17 17:29:26 UTC
Multiple SerDe per table name
Does hive currently support multiple SerDe s to be defined per table name?
Looking through the code and documentation, it seems like it doesn't as
only one could be specified through the ROW FORMAT SERDE but just wanted to
be sure.
--
Swarnim
Re: Multiple SerDe per table name
Posted by Edward Capriolo <ed...@gmail.com>.
No. The common approach is Serde's can take parameters (or you can use
table parameters) or they could discover the schema themselves
programatically (reflection, parsing) and act appropriately.
On 5/18/12, kulkarni.swarnim@gmail.com <ku...@gmail.com> wrote:
> Considering a case where we have multi HBase columns in an HBase table,
> each containing data of a different structure would warrant a need for
> multiple SerDe to map them to a single Hive table. Correct?
>
> On Thu, May 17, 2012 at 11:45 AM, Edward Capriolo
> <ed...@gmail.com>wrote:
>
>> This does not work. A Deserializer's role is to turn the value which
>> came form the InputFormat into something hive can use as column data.
>> In essence the Deserializer creates the columns so I do not see a
>> logical way to have more then one.
>>
>> On Thu, May 17, 2012 at 11:53 AM, kulkarni.swarnim@gmail.com
>> <ku...@gmail.com> wrote:
>> > I was thinking more from a perspective of specifying a SerDe per column
>> > name.
>> >
>> > On Thu, May 17, 2012 at 10:38 AM, Mark Grover <mg...@oanda.com>
>> > wrote:
>> >>
>> >> Hi Swarnim,
>> >> What's your use case?
>> >> If you use multiple SerDe's, when you are writing to the table, how
>> would
>> >> you want Hive to decide which one to use?
>> >>
>> >> Mark
>> >>
>> >> Mark Grover, Business Intelligence Analyst
>> >> OANDA Corporation
>> >>
>> >> www: oanda.com www: fxtrade.com
>> >>
>> >> ----- Original Message -----
>> >> From: "kulkarni swarnim" <ku...@gmail.com>
>> >> To: user@hive.apache.org
>> >> Sent: Thursday, May 17, 2012 11:29:26 AM
>> >> Subject: Multiple SerDe per table name
>> >>
>> >> Does hive currently support multiple SerDe s to be defined per table
>> name?
>> >> Looking through the code and documentation, it seems like it doesn't
>> >> as
>> only
>> >> one could be specified through the ROW FORMAT SERDE but just wanted to
>> be
>> >> sure.
>> >>
>> >>
>> >> --
>> >> Swarnim
>> >
>> >
>> >
>> >
>> > --
>> > Swarnim
>>
>
>
>
> --
> Swarnim
>
Re: Multiple SerDe per table name
Posted by "kulkarni.swarnim@gmail.com" <ku...@gmail.com>.
Considering a case where we have multi HBase columns in an HBase table,
each containing data of a different structure would warrant a need for
multiple SerDe to map them to a single Hive table. Correct?
On Thu, May 17, 2012 at 11:45 AM, Edward Capriolo <ed...@gmail.com>wrote:
> This does not work. A Deserializer's role is to turn the value which
> came form the InputFormat into something hive can use as column data.
> In essence the Deserializer creates the columns so I do not see a
> logical way to have more then one.
>
> On Thu, May 17, 2012 at 11:53 AM, kulkarni.swarnim@gmail.com
> <ku...@gmail.com> wrote:
> > I was thinking more from a perspective of specifying a SerDe per column
> > name.
> >
> > On Thu, May 17, 2012 at 10:38 AM, Mark Grover <mg...@oanda.com> wrote:
> >>
> >> Hi Swarnim,
> >> What's your use case?
> >> If you use multiple SerDe's, when you are writing to the table, how
> would
> >> you want Hive to decide which one to use?
> >>
> >> Mark
> >>
> >> Mark Grover, Business Intelligence Analyst
> >> OANDA Corporation
> >>
> >> www: oanda.com www: fxtrade.com
> >>
> >> ----- Original Message -----
> >> From: "kulkarni swarnim" <ku...@gmail.com>
> >> To: user@hive.apache.org
> >> Sent: Thursday, May 17, 2012 11:29:26 AM
> >> Subject: Multiple SerDe per table name
> >>
> >> Does hive currently support multiple SerDe s to be defined per table
> name?
> >> Looking through the code and documentation, it seems like it doesn't as
> only
> >> one could be specified through the ROW FORMAT SERDE but just wanted to
> be
> >> sure.
> >>
> >>
> >> --
> >> Swarnim
> >
> >
> >
> >
> > --
> > Swarnim
>
--
Swarnim
Re: Multiple SerDe per table name
Posted by Edward Capriolo <ed...@gmail.com>.
This does not work. A Deserializer's role is to turn the value which
came form the InputFormat into something hive can use as column data.
In essence the Deserializer creates the columns so I do not see a
logical way to have more then one.
On Thu, May 17, 2012 at 11:53 AM, kulkarni.swarnim@gmail.com
<ku...@gmail.com> wrote:
> I was thinking more from a perspective of specifying a SerDe per column
> name.
>
> On Thu, May 17, 2012 at 10:38 AM, Mark Grover <mg...@oanda.com> wrote:
>>
>> Hi Swarnim,
>> What's your use case?
>> If you use multiple SerDe's, when you are writing to the table, how would
>> you want Hive to decide which one to use?
>>
>> Mark
>>
>> Mark Grover, Business Intelligence Analyst
>> OANDA Corporation
>>
>> www: oanda.com www: fxtrade.com
>>
>> ----- Original Message -----
>> From: "kulkarni swarnim" <ku...@gmail.com>
>> To: user@hive.apache.org
>> Sent: Thursday, May 17, 2012 11:29:26 AM
>> Subject: Multiple SerDe per table name
>>
>> Does hive currently support multiple SerDe s to be defined per table name?
>> Looking through the code and documentation, it seems like it doesn't as only
>> one could be specified through the ROW FORMAT SERDE but just wanted to be
>> sure.
>>
>>
>> --
>> Swarnim
>
>
>
>
> --
> Swarnim
Re: Multiple SerDe per table name
Posted by Carl Steinbach <ca...@cloudera.com>.
In Hive 0.8 and later versions you have the ability to specify the SerDe on
a per-partition basis, but that's as granular as it gets.
Thanks.
Carl
On Thu, May 17, 2012 at 8:53 AM, kulkarni.swarnim@gmail.com <
kulkarni.swarnim@gmail.com> wrote:
> I was thinking more from a perspective of specifying a SerDe per column
> name.
>
>
> On Thu, May 17, 2012 at 10:38 AM, Mark Grover <mg...@oanda.com> wrote:
>
>> Hi Swarnim,
>> What's your use case?
>> If you use multiple SerDe's, when you are writing to the table, how would
>> you want Hive to decide which one to use?
>>
>> Mark
>>
>> Mark Grover, Business Intelligence Analyst
>> OANDA Corporation
>>
>> www: oanda.com www: fxtrade.com
>>
>> ----- Original Message -----
>> From: "kulkarni swarnim" <ku...@gmail.com>
>> To: user@hive.apache.org
>> Sent: Thursday, May 17, 2012 11:29:26 AM
>> Subject: Multiple SerDe per table name
>>
>> Does hive currently support multiple SerDe s to be defined per table
>> name? Looking through the code and documentation, it seems like it doesn't
>> as only one could be specified through the ROW FORMAT SERDE but just wanted
>> to be sure.
>>
>>
>> --
>> Swarnim
>>
>
>
>
> --
> Swarnim
>
Re: Multiple SerDe per table name
Posted by "kulkarni.swarnim@gmail.com" <ku...@gmail.com>.
I was thinking more from a perspective of specifying a SerDe per column
name.
On Thu, May 17, 2012 at 10:38 AM, Mark Grover <mg...@oanda.com> wrote:
> Hi Swarnim,
> What's your use case?
> If you use multiple SerDe's, when you are writing to the table, how would
> you want Hive to decide which one to use?
>
> Mark
>
> Mark Grover, Business Intelligence Analyst
> OANDA Corporation
>
> www: oanda.com www: fxtrade.com
>
> ----- Original Message -----
> From: "kulkarni swarnim" <ku...@gmail.com>
> To: user@hive.apache.org
> Sent: Thursday, May 17, 2012 11:29:26 AM
> Subject: Multiple SerDe per table name
>
> Does hive currently support multiple SerDe s to be defined per table name?
> Looking through the code and documentation, it seems like it doesn't as
> only one could be specified through the ROW FORMAT SERDE but just wanted to
> be sure.
>
>
> --
> Swarnim
>
--
Swarnim
Re: Multiple SerDe per table name
Posted by Mark Grover <mg...@oanda.com>.
Hi Swarnim,
What's your use case?
If you use multiple SerDe's, when you are writing to the table, how would you want Hive to decide which one to use?
Mark
Mark Grover, Business Intelligence Analyst
OANDA Corporation
www: oanda.com www: fxtrade.com
----- Original Message -----
From: "kulkarni swarnim" <ku...@gmail.com>
To: user@hive.apache.org
Sent: Thursday, May 17, 2012 11:29:26 AM
Subject: Multiple SerDe per table name
Does hive currently support multiple SerDe s to be defined per table name? Looking through the code and documentation, it seems like it doesn't as only one could be specified through the ROW FORMAT SERDE but just wanted to be sure.
--
Swarnim