You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Fan Yang <ya...@ebupt.com> on 2010/04/06 04:18:42 UTC
Problems of loading data from hdfs
Hi
When I try to load data into table from hdfs, it seems successful and gives
no warnings. But using select clause can't get any result from the table.
However, after I load data from local file which is download from hdfs by
using 'hadoop dfs get' command, I can select out records.
Note, these two files are exactly the same and no errors were given during
the procedure.
Thanks in advance.
Fan
Re: Problems of loading data from hdfs
Posted by Zheng Shao <zs...@gmail.com>.
There is no way to go around that right now.
Note that the partition column is actually not in the data - it's
appended to each row at runtime.
Hive uses the directory structure to figure out the partition values.
On Mon, Apr 5, 2010 at 9:24 PM, Fan Yang <ya...@ebupt.com> wrote:
> I’ve found out the root cause is not the data and table format. It’s because
> of the name of file on hdfs. I used “.data” as the file name and load data
> command ignore the file when its name started with ‘.’
>
>
>
> Anyway, thanks very much.
>
>
>
> BTW, when create a table with partitions. It seems the partition column must
> be the last column of the table. How can I use some other column which is
> not the last one to mark partition?
>
>
>
> Fan
>
>
>
> From: zhu weimin [mailto:xim-shu@tsm.kddilabs.jp]
> Sent: Tuesday, April 06, 2010 10:53 AM
> To: hive-user@hadoop.apache.org
> Subject: RE: Problems of loading data from hdfs
>
>
>
> I think your data probably terminated by comma character.
>
> If is that, recreate table like the following
>
> CREATE TABLE table_name{
>
> …
>
> }
>
> ROW FORMAT DELIMITED
>
> FIELDS TERMINATED BY ',';
>
>
>
> If is not, review your table describe and data samples
>
>
>
> Zhu weimin
>
>
>
> From: Fan Yang [mailto:yangfan@ebupt.com]
> Sent: Tuesday, April 06, 2010 11:19 AM
> To: hive-user@hadoop.apache.org
> Subject: Problems of loading data from hdfs
>
>
>
> Hi
>
>
>
> When I try to load data into table from hdfs, it seems successful and gives
> no warnings. But using select clause can’t get any result from the table.
>
> However, after I load data from local file which is download from hdfs by
> using ‘hadoop dfs get’ command, I can select out records.
>
> Note, these two files are exactly the same and no errors were given during
> the procedure.
>
>
>
> Thanks in advance.
>
>
>
> Fan
--
Yours,
Zheng
http://www.linkedin.com/in/zshao
RE: Problems of loading data from hdfs
Posted by Fan Yang <ya...@ebupt.com>.
I've found out the root cause is not the data and table format. It's because
of the name of file on hdfs. I used ".data" as the file name and load data
command ignore the file when its name started with '.'
Anyway, thanks very much.
BTW, when create a table with partitions. It seems the partition column must
be the last column of the table. How can I use some other column which is
not the last one to mark partition?
Fan
From: zhu weimin [mailto:xim-shu@tsm.kddilabs.jp]
Sent: Tuesday, April 06, 2010 10:53 AM
To: hive-user@hadoop.apache.org
Subject: RE: Problems of loading data from hdfs
I think your data probably terminated by comma character.
If is that, recreate table like the following
CREATE TABLE table_name{
.
}
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ',';
If is not, review your table describe and data samples
Zhu weimin
From: Fan Yang [mailto:yangfan@ebupt.com]
Sent: Tuesday, April 06, 2010 11:19 AM
To: hive-user@hadoop.apache.org
Subject: Problems of loading data from hdfs
Hi
When I try to load data into table from hdfs, it seems successful and gives
no warnings. But using select clause can't get any result from the table.
However, after I load data from local file which is download from hdfs by
using 'hadoop dfs get' command, I can select out records.
Note, these two files are exactly the same and no errors were given during
the procedure.
Thanks in advance.
Fan
RE: Problems of loading data from hdfs
Posted by zhu weimin <xi...@tsm.kddilabs.jp>.
I think your data probably terminated by comma character.
If is that, recreate table like the following
CREATE TABLE table_name{
.
}
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ',';
If is not, review your table describe and data samples
Zhu weimin
From: Fan Yang [mailto:yangfan@ebupt.com]
Sent: Tuesday, April 06, 2010 11:19 AM
To: hive-user@hadoop.apache.org
Subject: Problems of loading data from hdfs
Hi
When I try to load data into table from hdfs, it seems successful and gives
no warnings. But using select clause can't get any result from the table.
However, after I load data from local file which is download from hdfs by
using 'hadoop dfs get' command, I can select out records.
Note, these two files are exactly the same and no errors were given during
the procedure.
Thanks in advance.
Fan