You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Josh Spiegel <jo...@gmail.com> on 2012/10/11 04:21:35 UTC
Creating a table with a custom InputFormat
Hi,
I want to create a table with a custom InputFormat. For example, something
like this:
CREATE TABLE xxx (blah string)
STORED AS
INPUTFORMAT 'org.foo.MyInputFormat'
OUTPUTFORMAT 'org.foo.MyOutputFormat'
;
I also want to be able specify some configuration values that will be
available to the InputFormat (possibly via the JobConf). Is this
possible?
It looks like "stored by"/HiveStorageHandler might make this possible by
allowing me to push TBLPROPERTIES into the JobConf. But I would like to
avoid a non-native table if possible.
Thanks,
Josh
Re: Creating a table with a custom InputFormat
Posted by Josh Spiegel <jo...@gmail.com>.
I want to specify the configuration properties when the table is created.
This bug looks like it enables specifying a configuration when the table is
referenced.
Thanks again,
Josh
On Wed, Oct 10, 2012 at 8:33 PM, Edward Capriolo <ed...@gmail.com>wrote:
> This is something similar. it may be the issue I remember.
>
> https://reviews.facebook.net/D2499
>
> In any case we could pass ALL the information from the properties,
> Input paths, would be a way to make hive semi pig-like.
>
>
>
> On Wed, Oct 10, 2012 at 11:26 PM, Josh Spiegel <jo...@gmail.com> wrote:
> > Thanks. Do you have the bug number? I couldn't find it.
> >
> > On Wed, Oct 10, 2012 at 7:49 PM, Edward Capriolo <ed...@gmail.com>
> > wrote:
> >>
> >> No this is not we have a ticket open to propage tbl properies matching
> >> a pattern to the input format but not complete yet.
> >>
> >> On Wed, Oct 10, 2012 at 10:21 PM, Josh Spiegel <jo...@gmail.com>
> wrote:
> >> > Hi,
> >> >
> >> > I want to create a table with a custom InputFormat. For example,
> >> > something
> >> > like this:
> >> >
> >> > CREATE TABLE xxx (blah string)
> >> > STORED AS
> >> > INPUTFORMAT 'org.foo.MyInputFormat'
> >> > OUTPUTFORMAT 'org.foo.MyOutputFormat'
> >> > ;
> >> >
> >> > I also want to be able specify some configuration values that will be
> >> > available to the InputFormat (possibly via the JobConf). Is this
> >> > possible?
> >> >
> >> > It looks like "stored by"/HiveStorageHandler might make this possible
> by
> >> > allowing me to push TBLPROPERTIES into the JobConf. But I would like
> to
> >> > avoid a non-native table if possible.
> >> >
> >> > Thanks,
> >> > Josh
> >
> >
>
Re: Creating a table with a custom InputFormat
Posted by Edward Capriolo <ed...@gmail.com>.
This is something similar. it may be the issue I remember.
https://reviews.facebook.net/D2499
In any case we could pass ALL the information from the properties,
Input paths, would be a way to make hive semi pig-like.
On Wed, Oct 10, 2012 at 11:26 PM, Josh Spiegel <jo...@gmail.com> wrote:
> Thanks. Do you have the bug number? I couldn't find it.
>
> On Wed, Oct 10, 2012 at 7:49 PM, Edward Capriolo <ed...@gmail.com>
> wrote:
>>
>> No this is not we have a ticket open to propage tbl properies matching
>> a pattern to the input format but not complete yet.
>>
>> On Wed, Oct 10, 2012 at 10:21 PM, Josh Spiegel <jo...@gmail.com> wrote:
>> > Hi,
>> >
>> > I want to create a table with a custom InputFormat. For example,
>> > something
>> > like this:
>> >
>> > CREATE TABLE xxx (blah string)
>> > STORED AS
>> > INPUTFORMAT 'org.foo.MyInputFormat'
>> > OUTPUTFORMAT 'org.foo.MyOutputFormat'
>> > ;
>> >
>> > I also want to be able specify some configuration values that will be
>> > available to the InputFormat (possibly via the JobConf). Is this
>> > possible?
>> >
>> > It looks like "stored by"/HiveStorageHandler might make this possible by
>> > allowing me to push TBLPROPERTIES into the JobConf. But I would like to
>> > avoid a non-native table if possible.
>> >
>> > Thanks,
>> > Josh
>
>
Re: Creating a table with a custom InputFormat
Posted by Josh Spiegel <jo...@gmail.com>.
Thanks. Do you have the bug number? I couldn't find it.
On Wed, Oct 10, 2012 at 7:49 PM, Edward Capriolo <ed...@gmail.com>wrote:
> No this is not we have a ticket open to propage tbl properies matching
> a pattern to the input format but not complete yet.
>
> On Wed, Oct 10, 2012 at 10:21 PM, Josh Spiegel <jo...@gmail.com> wrote:
> > Hi,
> >
> > I want to create a table with a custom InputFormat. For example,
> something
> > like this:
> >
> > CREATE TABLE xxx (blah string)
> > STORED AS
> > INPUTFORMAT 'org.foo.MyInputFormat'
> > OUTPUTFORMAT 'org.foo.MyOutputFormat'
> > ;
> >
> > I also want to be able specify some configuration values that will be
> > available to the InputFormat (possibly via the JobConf). Is this
> possible?
> >
> > It looks like "stored by"/HiveStorageHandler might make this possible by
> > allowing me to push TBLPROPERTIES into the JobConf. But I would like to
> > avoid a non-native table if possible.
> >
> > Thanks,
> > Josh
>
Re: Creating a table with a custom InputFormat
Posted by Edward Capriolo <ed...@gmail.com>.
No this is not we have a ticket open to propage tbl properies matching
a pattern to the input format but not complete yet.
On Wed, Oct 10, 2012 at 10:21 PM, Josh Spiegel <jo...@gmail.com> wrote:
> Hi,
>
> I want to create a table with a custom InputFormat. For example, something
> like this:
>
> CREATE TABLE xxx (blah string)
> STORED AS
> INPUTFORMAT 'org.foo.MyInputFormat'
> OUTPUTFORMAT 'org.foo.MyOutputFormat'
> ;
>
> I also want to be able specify some configuration values that will be
> available to the InputFormat (possibly via the JobConf). Is this possible?
>
> It looks like "stored by"/HiveStorageHandler might make this possible by
> allowing me to push TBLPROPERTIES into the JobConf. But I would like to
> avoid a non-native table if possible.
>
> Thanks,
> Josh