You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by wd <wd...@wdicc.com> on 2011/04/28 06:33:27 UTC
Load gzip files into hive
hi,
I've tried to load gzip files into hive to save disk space, but failed.
hive> load data local inpath 'tmp_b.20110426.gz' into table raw_logs
partition ( dt=20110426 );
Copying data from file:/home/wd/t/tmp_b.20110426.gz
Copying file: file:/home/wd/t/tmp_b.20110426.gz
Loading data to table default.raw_logs partition (dt=20110426)
Failed with exception Wrong file format. Please check the file's format.
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.MoveTask
The raw_logs table is created by:
create table raw_logs ( ............) partitioned by ( dt int ) STORED AS
SEQUENCEFILE;
Is there something wrong? The error is same both in hive 0.5 and 0.7.
Re: Load gzip files into hive
Posted by wd <wd...@wdicc.com>.
Thanks for your help
2011/4/28 Loren Siebert <lo...@siebert.org>
> You have the file type as sequence file, but you are trying to load a GZip
> file. Won’t that only work if the table is defined as a text file?
>
I've think sequence = gzip file before, and now I realized it's not.
It's work when table is defined as text file.
>
> Hive isn’t doing anything on your behalf when you do LOAD DATA. It’s
> syntactic sugar for copying a file into a HDFS location. From there, if you
> want a RCFile table or a sequence file table or whatever, you can select
> from the raw_logs table into the new table (e.g., raw_logs_rcfile) that you
> have defined in the different format.
>
> So, this is the only way I can put data into a table defined as sequence
file? Can I generate the RCFile use a unix command or some tools ?
>
> On Apr 27, 2011, at 9:33 PM, wd wrote:
>
> hi,
>
> I've tried to load gzip files into hive to save disk space, but failed.
>
> hive> load data local inpath 'tmp_b.20110426.gz' into table raw_logs
> partition ( dt=20110426 );
> Copying data from file:/home/wd/t/tmp_b.20110426.gz
> Copying file: file:/home/wd/t/tmp_b.20110426.gz
> Loading data to table default.raw_logs partition (dt=20110426)
> Failed with exception Wrong file format. Please check the file's format.
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.MoveTask
>
> The raw_logs table is created by:
> create table raw_logs ( ............) partitioned by ( dt int ) STORED AS
> SEQUENCEFILE;
>
> Is there something wrong? The error is same both in hive 0.5 and 0.7.
>
>
>
Re: Load gzip files into hive
Posted by Loren Siebert <lo...@siebert.org>.
You have the file type as sequence file, but you are trying to load a GZip file. Won’t that only work if the table is defined as a text file?
Hive isn’t doing anything on your behalf when you do LOAD DATA. It’s syntactic sugar for copying a file into a HDFS location. From there, if you want a RCFile table or a sequence file table or whatever, you can select from the raw_logs table into the new table (e.g., raw_logs_rcfile) that you have defined in the different format.
On Apr 27, 2011, at 9:33 PM, wd wrote:
> hi,
>
> I've tried to load gzip files into hive to save disk space, but failed.
>
> hive> load data local inpath 'tmp_b.20110426.gz' into table raw_logs partition ( dt=20110426 );
> Copying data from file:/home/wd/t/tmp_b.20110426.gz
> Copying file: file:/home/wd/t/tmp_b.20110426.gz
> Loading data to table default.raw_logs partition (dt=20110426)
> Failed with exception Wrong file format. Please check the file's format.
> FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask
>
> The raw_logs table is created by:
> create table raw_logs ( ............) partitioned by ( dt int ) STORED AS SEQUENCEFILE;
>
> Is there something wrong? The error is same both in hive 0.5 and 0.7.