You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by P Reeder <p_...@persistentsys.com> on 2013/10/28 20:48:06 UTC

SerDe for Fixed Columns?

Hi! Sorry to ask such a basic question, but

How do I import data from a text file with fixed columns (no delimiter)? I.e, column 1 is characters 1-6, column 2 is characters 7-11, column 3 is character 12, column 4 is character 13, and so forth?  I'd be surprised if no one has written a SerDe for this.

The manual talks about a MetadataTypedColumnsetSerDe, a ThriftSerDe, a DynamicSerDe, and SerDes for JSON, Avro and ORC.  I also searched Bing for "site:mail-archives.apache.org/mod_mbox/hive-user/ serde fixed", but that didn't appear to have anything relevant.

FWIW, the dataset is the National Immunization Survey: http://www.cdc.gov/nchs/nis/data_files.htm

DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.


Re: SerDe for Fixed Columns?

Posted by Roberto Congiu <ro...@openx.com>.
Also, you could use the regexp serde using a regexp like
(.{10})(.{3})(.{4}) etc (1st column -> 10 characters, 2nd col -> 3
characters, etc).


On Mon, Oct 28, 2013 at 1:03 PM, Nitin Pawar <ni...@gmail.com>wrote:

> I am not aware of any such serde.
>
> Till others reply you can wait
> or if you are in hurry then load the data in a table as single column and
> then use substring method to select and load data into proper table.
>
>
> On Tue, Oct 29, 2013 at 1:18 AM, P Reeder <p_...@persistentsys.com>wrote:
>
>>  Hi! Sorry to ask such a basic question, but ****
>>
>> ** **
>>
>> How do I import data from a text file with fixed columns (no delimiter)?
>> I.e, column 1 is characters 1-6, column 2 is characters 7-11, column 3 is
>> character 12, column 4 is character 13, and so forth?  I’d be surprised if
>> no one has written a SerDe for this.  ****
>>
>> ** **
>>
>> The manual talks about a MetadataTypedColumnsetSerDe, a ThriftSerDe, a
>> DynamicSerDe, and SerDes for JSON, Avro and ORC.  I also searched Bing for
>> “site:mail-archives.apache.org/mod_mbox/hive-user/ serde fixed”, but
>> that didn’t appear to have anything relevant.****
>>
>> ** **
>>
>> FWIW, the dataset is the National Immunization Survey:
>> http://www.cdc.gov/nchs/nis/data_files.htm****
>>
>> DISCLAIMER ========== This e-mail may contain privileged and confidential
>> information which is the property of Persistent Systems Ltd. It is intended
>> only for the use of the individual or entity to which it is addressed. If
>> you are not the intended recipient, you are not authorized to read, retain,
>> copy, print, distribute or use this message. If you have received this
>> communication in error, please notify the sender and delete all copies of
>> this message. Persistent Systems Ltd. does not accept any liability for
>> virus infected mails.
>>
>
>
>
> --
> Nitin Pawar
>



-- 
----------------------------------------------------------
Good judgement comes with experience.
Experience comes with bad judgement.
----------------------------------------------------------
Roberto Congiu - Data Engineer - OpenX
tel: +1 626 466 1141

Re: SerDe for Fixed Columns?

Posted by Nitin Pawar <ni...@gmail.com>.
I am not aware of any such serde.

Till others reply you can wait
or if you are in hurry then load the data in a table as single column and
then use substring method to select and load data into proper table.


On Tue, Oct 29, 2013 at 1:18 AM, P Reeder <p_...@persistentsys.com>wrote:

>  Hi! Sorry to ask such a basic question, but ****
>
> ** **
>
> How do I import data from a text file with fixed columns (no delimiter)?
> I.e, column 1 is characters 1-6, column 2 is characters 7-11, column 3 is
> character 12, column 4 is character 13, and so forth?  I’d be surprised if
> no one has written a SerDe for this.  ****
>
> ** **
>
> The manual talks about a MetadataTypedColumnsetSerDe, a ThriftSerDe, a
> DynamicSerDe, and SerDes for JSON, Avro and ORC.  I also searched Bing for
> “site:mail-archives.apache.org/mod_mbox/hive-user/ serde fixed”, but that
> didn’t appear to have anything relevant.****
>
> ** **
>
> FWIW, the dataset is the National Immunization Survey:
> http://www.cdc.gov/nchs/nis/data_files.htm****
>
> DISCLAIMER ========== This e-mail may contain privileged and confidential
> information which is the property of Persistent Systems Ltd. It is intended
> only for the use of the individual or entity to which it is addressed. If
> you are not the intended recipient, you are not authorized to read, retain,
> copy, print, distribute or use this message. If you have received this
> communication in error, please notify the sender and delete all copies of
> this message. Persistent Systems Ltd. does not accept any liability for
> virus infected mails.
>



-- 
Nitin Pawar