You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Jeremy Chow <co...@gmail.com> on 2009/01/12 10:19:51 UTC
Can hive load a table from a SequenceFile?
hi list,
Let's assume that we have a SequenceFile with a format like this,
<LongWritable key, Text value>.
All of the keys will discard when loading into a hive table. Each value,
which is delimited by tabs, will be transformed into a row of a hive table.
Can hive load a table from this kind of files? How can I implement it ?
Thanks,
Jeremy
--
My research interests are distributed systems, parallel computing and
bytecode based virtual machine.
http://coderplay.javaeye.com
Re: Can hive load a table from a SequenceFile?
Posted by Prasad Chakka <pr...@facebook.com>.
Jeremy,
Josh had same problem and we figured out that if you have full hdfs path in hive.metastore.warehouse.dir then below queries would work. It is bug that the query isn't working without the full path. I suppose this workaround can be used for now?
Prasad
hadoop-site.xml
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
hive-default.xml
<property>
<name>hive.metastore.warehouse.dir</name>
<value>hdfs://localhost:9000/user/hive/warehouse</value>
<description>location of default database for the warehouse</description>
</property>
________________________________
From: Prasad Chakka <pr...@facebook.com>
Date: Wed, 14 Jan 2009 10:01:24 -0800
To: <hi...@hadoop.apache.org>
Conversation: Can hive load a table from a SequenceFile?
Subject: Re: Can hive load a table from a SequenceFile?
Jeremy, could post the namenode and the metastore.warehouse variables that are in hadoop-site.xml and hive-default.xml files. I have not seen this problem in my setup so I want to see why it is different in my set up.
Thanks,
Prasad
________________________________
From: Joydeep Sen Sarma <js...@facebook.com>
Reply-To: <hi...@hadoop.apache.org>
Date: Wed, 14 Jan 2009 09:00:38 -0800
To: <hi...@hadoop.apache.org>
Subject: RE: Can hive load a table from a SequenceFile?
Hey Jeremy -
Looks like this was more trouble than it should have been. Can u help us by filing a couple of Jiras on expected behavior:
1. should 'location ..' clause in create table force people to specify uri? Or should it use fs.default.name from hadoop configuration and tell user that it's doing so?
2. load data command - should it use scheme/authority of the target table instead of using the fs.default.name from hadoop config?
Would fixing these issues have helped?
Thanks
Joydeep
________________________________
From: Jeremy Chow [mailto:coderplay@gmail.com]
Sent: Tuesday, January 13, 2009 9:42 PM
To: hive-user@hadoop.apache.org
Subject: Re: Can hive load a table from a SequenceFile?
Hi all,
It seems like a bug of hive. The URI of table is valued by this code in Hive.java,
table.setDataLocation(new URI(tTable.getSd().getLocation()));
and the tTable.getSd().getLocation() is a string without prefix like "file://" and "hdfs://" , it will causes that the scheme of URI object be null.
Jeremy
--
My research interests are distributed systems, parallel computing and bytecode based virtual machine.
http://coderplay.javaeye.com
Re: Can hive load a table from a SequenceFile?
Posted by Prasad Chakka <pr...@facebook.com>.
Jeremy, could post the namenode and the metastore.warehouse variables that are in hadoop-site.xml and hive-default.xml files. I have not seen this problem in my setup so I want to see why it is different in my set up.
Thanks,
Prasad
________________________________
From: Joydeep Sen Sarma <js...@facebook.com>
Reply-To: <hi...@hadoop.apache.org>
Date: Wed, 14 Jan 2009 09:00:38 -0800
To: <hi...@hadoop.apache.org>
Subject: RE: Can hive load a table from a SequenceFile?
Hey Jeremy -
Looks like this was more trouble than it should have been. Can u help us by filing a couple of Jiras on expected behavior:
1. should 'location ..' clause in create table force people to specify uri? Or should it use fs.default.name from hadoop configuration and tell user that it's doing so?
2. load data command - should it use scheme/authority of the target table instead of using the fs.default.name from hadoop config?
Would fixing these issues have helped?
Thanks
Joydeep
________________________________
From: Jeremy Chow [mailto:coderplay@gmail.com]
Sent: Tuesday, January 13, 2009 9:42 PM
To: hive-user@hadoop.apache.org
Subject: Re: Can hive load a table from a SequenceFile?
Hi all,
It seems like a bug of hive. The URI of table is valued by this code in Hive.java,
table.setDataLocation(new URI(tTable.getSd().getLocation()));
and the tTable.getSd().getLocation() is a string without prefix like "file://" and "hdfs://" , it will causes that the scheme of URI object be null.
Jeremy
--
My research interests are distributed systems, parallel computing and bytecode based virtual machine.
http://coderplay.javaeye.com
RE: Can hive load a table from a SequenceFile?
Posted by Joydeep Sen Sarma <js...@facebook.com>.
Hey Jeremy -
Looks like this was more trouble than it should have been. Can u help us by filing a couple of Jiras on expected behavior:
1. should 'location ..' clause in create table force people to specify uri? Or should it use fs.default.name from hadoop configuration and tell user that it's doing so?
2. load data command - should it use scheme/authority of the target table instead of using the fs.default.name from hadoop config?
Would fixing these issues have helped?
Thanks
Joydeep
________________________________
From: Jeremy Chow [mailto:coderplay@gmail.com]
Sent: Tuesday, January 13, 2009 9:42 PM
To: hive-user@hadoop.apache.org
Subject: Re: Can hive load a table from a SequenceFile?
Hi all,
It seems like a bug of hive. The URI of table is valued by this code in Hive.java,
table.setDataLocation(new URI(tTable.getSd().getLocation()));
and the tTable.getSd().getLocation() is a string without prefix like "file://" and "hdfs://" , it will causes that the scheme of URI object be null.
Jeremy
--
My research interests are distributed systems, parallel computing and bytecode based virtual machine.
http://coderplay.javaeye.com
Re: Can hive load a table from a SequenceFile?
Posted by Jeremy Chow <co...@gmail.com>.
Hi all,
It will be okay when creating a table in this way, describing its
LOCATION,
CREATE TABLE log_data(log STRING, time STRING)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '"'
STORED AS SEQUENCEFILE
LOCATION 'hdfs://test162.sqa:9000/user/hive/warehouse/log_data';
JPox will retrieve the full path of this table, including its prefix
"hdfs://test162.sqa:9000".
--
My research interests are distributed systems, parallel computing and
bytecode based virtual machine.
http://coderplay.javaeye.com
Re: Can hive load a table from a SequenceFile?
Posted by Jeremy Chow <co...@gmail.com>.
And the original table path is from the ObjectStore, see this line in
ObjectStore.java
mtbl = (MTable) query.execute(table.trim(), db.trim());
Jeremy
--
My research interests are distributed systems, parallel computing and
bytecode based virtual machine.
http://coderplay.javaeye.com
Re: Can hive load a table from a SequenceFile?
Posted by Jeremy Chow <co...@gmail.com>.
Hi all,
It seems like a bug of hive. The URI of table is valued by this code in
Hive.java,
table.setDataLocation(new URI(tTable.getSd().getLocation()));
and the tTable.getSd().getLocation() is a string without prefix like
"file://" and "hdfs://" , it will causes that the scheme of URI object be
null.
Jeremy
--
My research interests are distributed systems, parallel computing and
bytecode based virtual machine.
http://coderplay.javaeye.com
Re: Can hive load a table from a SequenceFile?
Posted by Josh Ferguson <jo...@besquared.net>.
This is indeed the same problem stopping me from being able to insert
directly from a custom map/reduce job.
Josh F.
On Jan 13, 2009, at 7:00 PM, Jeremy Chow wrote:
> Hi ,
> I've rebuild the cluster yesterday, but errors still happened. Then
> I scaned the hive.log, and found a error report,
> 2009-01-14 10:38:42,770 ERROR parse.LoadSemanticAnalyzer
> (LoadSemanticAnalyzer.java:applyConstraints(152)) - Move from:
> hdfs://test.sqa:9000/user/zhoumin/weblog to: /user/hive/warehouse/
> log_data is not valid
>
> By digging into the source code, I found that the value of
> fromURI.getScheme(), toURI.getScheme() are "hdfs" and null
> respectively.
>
> so, how can I solve it?
>
> On Tue, Jan 13, 2009 at 4:09 PM, Zheng Shao <zs...@gmail.com> wrote:
> The problem is that the specification of file system ("hdfs://master:
> 9000/") is different as the one specified in hadoop-site.xml.
> Can you make sure they match?
>
> Zheng
>
>
> On Tue, Jan 13, 2009 at 12:04 AM, Jeremy Chow <co...@gmail.com>
> wrote:
> Hi,
> this is the result:
>
> $ bin/hadoop dfs -ls /user/zhoumin/log/web_log2009010606
> Found 1 items
> -rw-r--r-- 3 hadoop supergroup 204198066 2009-01-12 16:36 /user/
> zhoumin/log/web_log2009010606
>
>
> On Tue, Jan 13, 2009 at 3:43 PM, Raghu Murthy <ra...@facebook.com>
> wrote:
> Can you paste the output of the following command?
> $ hadoop dfs -ls /user/zhoumin/log/web_log2009010606
>
>
> On 1/12/09 11:40 PM, "Jeremy Chow" <co...@gmail.com> wrote:
>
> > I've tried it, but failed.
> >
> > hive> LOAD DATA INPATH 'hdfs://master:9000/user/zhoumin/log/
> web_log2009010606'
> > OVERWRITE INTO TABLE log_data;
> > FAILED: Error in semantic analysis: line 1:18 Path is not legal
> > 'hdfs://master:9000/user/zhoumin/log/web_log2009010606': Cannot
> load data
> > across filesystems, use load data local
> > Time taken: 2.641 seconds
> >
> >
> > On Tue, Jan 13, 2009 at 11:04 AM, Joydeep Sen Sarma <jssarma@facebook.com
> >
> > wrote:
> >> Please give a full uri like hdfs://xxx.yyy.zzz:9000/user/S
> >>
> >>
> >>
> >> Where xxx.yyy.zzz is the same namenode/hdfs instance where u are
> planning to
> >> store the hive tables.
> >>
> >>
> >>
> >>
> >> From: Jeremy Chow [mailto:coderplay@gmail.com]
> >> Sent: Monday, January 12, 2009 6:17 PM
> >>
> >> To: hive-user@hadoop.apache.org
> >> Subject: Re: Can hive load a table from a SequenceFile?
> >>
> >>
> >>
> >> Hey Joydeep,
> >>
> >> - load data infile yyy into table xxx
> >>
> >> hive> LOAD DATA INPATH '/user/zhoumin/log/web_log2009010606'
> OVERWRITE INTO
> >> TABLE log_data;
> >> FAILED: Error in semantic analysis: line 1:18 Path is not legal
> >> '/user/zhoumin/log/web_log2009010606': Cannot load data across
> filesystems,
> >> use load data local
> >> Time taken: 0.039 seconds
> >>
> >> It failed, that file existed in hdfs, but can not be loaded into
> hive.
> >>
> >> --
> >> My research interests are distributed systems, parallel computing
> and
> >> bytecode
> >> based virtual machine.
> >>
> >> http://coderplay.javaeye.com
> >
> >
>
>
>
>
> --
> My research interests are distributed systems, parallel computing
> and bytecode based virtual machine.
>
> http://coderplay.javaeye.com
>
>
>
> --
> Yours,
> Zheng
>
>
>
> --
> My research interests are distributed systems, parallel computing
> and bytecode based virtual machine.
>
> http://coderplay.javaeye.com
Re: Can hive load a table from a SequenceFile?
Posted by Jeremy Chow <co...@gmail.com>.
Hi ,
I've rebuild the cluster yesterday, but errors still happened. Then I scaned
the hive.log, and found a error report,
2009-01-14 10:38:42,770 ERROR parse.LoadSemanticAnalyzer
(LoadSemanticAnalyzer.java:applyConstraints(152)) - Move from:
hdfs://test.sqa:9000/user/zhoumin/weblog to: /user/hive/warehouse/log_data
is not valid
By digging into the source code, I found that the value of
fromURI.getScheme(), toURI.getScheme() are "hdfs" and null respectively.
so, how can I solve it?
On Tue, Jan 13, 2009 at 4:09 PM, Zheng Shao <zs...@gmail.com> wrote:
> The problem is that the specification of file system
> ("hdfs://master:9000/") is different as the one specified in
> hadoop-site.xml.
> Can you make sure they match?
>
> Zheng
>
>
> On Tue, Jan 13, 2009 at 12:04 AM, Jeremy Chow <co...@gmail.com> wrote:
>
>> Hi,
>> this is the result:
>>
>> $ bin/hadoop dfs -ls /user/zhoumin/log/web_log2009010606
>> Found 1 items
>> -rw-r--r-- 3 hadoop supergroup 204198066 2009-01-12 16:36
>> /user/zhoumin/log/web_log2009010606
>>
>>
>> On Tue, Jan 13, 2009 at 3:43 PM, Raghu Murthy <ra...@facebook.com> wrote:
>>
>>> Can you paste the output of the following command?
>>> $ hadoop dfs -ls /user/zhoumin/log/web_log2009010606
>>>
>>>
>>> On 1/12/09 11:40 PM, "Jeremy Chow" <co...@gmail.com> wrote:
>>>
>>> > I've tried it, but failed.
>>> >
>>> > hive> LOAD DATA INPATH
>>> 'hdfs://master:9000/user/zhoumin/log/web_log2009010606'
>>> > OVERWRITE INTO TABLE log_data;
>>> > FAILED: Error in semantic analysis: line 1:18 Path is not legal
>>> > 'hdfs://master:9000/user/zhoumin/log/web_log2009010606': Cannot load
>>> data
>>> > across filesystems, use load data local
>>> > Time taken: 2.641 seconds
>>> >
>>> >
>>> > On Tue, Jan 13, 2009 at 11:04 AM, Joydeep Sen Sarma <
>>> jssarma@facebook.com>
>>> > wrote:
>>> >> Please give a full uri like hdfs://xxx.yyy.zzz:9000/user/S
>>> >>
>>> >>
>>> >>
>>> >> Where xxx.yyy.zzz is the same namenode/hdfs instance where u are
>>> planning to
>>> >> store the hive tables.
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> From: Jeremy Chow [mailto:coderplay@gmail.com]
>>> >> Sent: Monday, January 12, 2009 6:17 PM
>>> >>
>>> >> To: hive-user@hadoop.apache.org
>>> >> Subject: Re: Can hive load a table from a SequenceFile?
>>> >>
>>> >>
>>> >>
>>> >> Hey Joydeep,
>>> >>
>>> >> - load data infile yyy into table xxx
>>> >>
>>> >> hive> LOAD DATA INPATH '/user/zhoumin/log/web_log2009010606' OVERWRITE
>>> INTO
>>> >> TABLE log_data;
>>> >> FAILED: Error in semantic analysis: line 1:18 Path is not legal
>>> >> '/user/zhoumin/log/web_log2009010606': Cannot load data across
>>> filesystems,
>>> >> use load data local
>>> >> Time taken: 0.039 seconds
>>> >>
>>> >> It failed, that file existed in hdfs, but can not be loaded into hive.
>>> >>
>>> >> --
>>> >> My research interests are distributed systems, parallel computing and
>>> >> bytecode
>>> >> based virtual machine.
>>> >>
>>> >> http://coderplay.javaeye.com
>>> >
>>> >
>>>
>>>
>>
>>
>> --
>> My research interests are distributed systems, parallel computing and
>> bytecode based virtual machine.
>>
>> http://coderplay.javaeye.com
>>
>
>
>
> --
> Yours,
> Zheng
>
--
My research interests are distributed systems, parallel computing and
bytecode based virtual machine.
http://coderplay.javaeye.com
Re: Can hive load a table from a SequenceFile?
Posted by Zheng Shao <zs...@gmail.com>.
The problem is that the specification of file system ("hdfs://master:9000/")
is different as the one specified in hadoop-site.xml.
Can you make sure they match?
Zheng
On Tue, Jan 13, 2009 at 12:04 AM, Jeremy Chow <co...@gmail.com> wrote:
> Hi,
> this is the result:
>
> $ bin/hadoop dfs -ls /user/zhoumin/log/web_log2009010606
> Found 1 items
> -rw-r--r-- 3 hadoop supergroup 204198066 2009-01-12 16:36
> /user/zhoumin/log/web_log2009010606
>
>
> On Tue, Jan 13, 2009 at 3:43 PM, Raghu Murthy <ra...@facebook.com> wrote:
>
>> Can you paste the output of the following command?
>> $ hadoop dfs -ls /user/zhoumin/log/web_log2009010606
>>
>>
>> On 1/12/09 11:40 PM, "Jeremy Chow" <co...@gmail.com> wrote:
>>
>> > I've tried it, but failed.
>> >
>> > hive> LOAD DATA INPATH
>> 'hdfs://master:9000/user/zhoumin/log/web_log2009010606'
>> > OVERWRITE INTO TABLE log_data;
>> > FAILED: Error in semantic analysis: line 1:18 Path is not legal
>> > 'hdfs://master:9000/user/zhoumin/log/web_log2009010606': Cannot load
>> data
>> > across filesystems, use load data local
>> > Time taken: 2.641 seconds
>> >
>> >
>> > On Tue, Jan 13, 2009 at 11:04 AM, Joydeep Sen Sarma <
>> jssarma@facebook.com>
>> > wrote:
>> >> Please give a full uri like hdfs://xxx.yyy.zzz:9000/user/S
>> >>
>> >>
>> >>
>> >> Where xxx.yyy.zzz is the same namenode/hdfs instance where u are
>> planning to
>> >> store the hive tables.
>> >>
>> >>
>> >>
>> >>
>> >> From: Jeremy Chow [mailto:coderplay@gmail.com]
>> >> Sent: Monday, January 12, 2009 6:17 PM
>> >>
>> >> To: hive-user@hadoop.apache.org
>> >> Subject: Re: Can hive load a table from a SequenceFile?
>> >>
>> >>
>> >>
>> >> Hey Joydeep,
>> >>
>> >> - load data infile yyy into table xxx
>> >>
>> >> hive> LOAD DATA INPATH '/user/zhoumin/log/web_log2009010606' OVERWRITE
>> INTO
>> >> TABLE log_data;
>> >> FAILED: Error in semantic analysis: line 1:18 Path is not legal
>> >> '/user/zhoumin/log/web_log2009010606': Cannot load data across
>> filesystems,
>> >> use load data local
>> >> Time taken: 0.039 seconds
>> >>
>> >> It failed, that file existed in hdfs, but can not be loaded into hive.
>> >>
>> >> --
>> >> My research interests are distributed systems, parallel computing and
>> >> bytecode
>> >> based virtual machine.
>> >>
>> >> http://coderplay.javaeye.com
>> >
>> >
>>
>>
>
>
> --
> My research interests are distributed systems, parallel computing and
> bytecode based virtual machine.
>
> http://coderplay.javaeye.com
>
--
Yours,
Zheng
Re: Can hive load a table from a SequenceFile?
Posted by Jeremy Chow <co...@gmail.com>.
Hi,
this is the result:
$ bin/hadoop dfs -ls /user/zhoumin/log/web_log2009010606
Found 1 items
-rw-r--r-- 3 hadoop supergroup 204198066 2009-01-12 16:36
/user/zhoumin/log/web_log2009010606
On Tue, Jan 13, 2009 at 3:43 PM, Raghu Murthy <ra...@facebook.com> wrote:
> Can you paste the output of the following command?
> $ hadoop dfs -ls /user/zhoumin/log/web_log2009010606
>
>
> On 1/12/09 11:40 PM, "Jeremy Chow" <co...@gmail.com> wrote:
>
> > I've tried it, but failed.
> >
> > hive> LOAD DATA INPATH
> 'hdfs://master:9000/user/zhoumin/log/web_log2009010606'
> > OVERWRITE INTO TABLE log_data;
> > FAILED: Error in semantic analysis: line 1:18 Path is not legal
> > 'hdfs://master:9000/user/zhoumin/log/web_log2009010606': Cannot load data
> > across filesystems, use load data local
> > Time taken: 2.641 seconds
> >
> >
> > On Tue, Jan 13, 2009 at 11:04 AM, Joydeep Sen Sarma <
> jssarma@facebook.com>
> > wrote:
> >> Please give a full uri like hdfs://xxx.yyy.zzz:9000/user/S
> >>
> >>
> >>
> >> Where xxx.yyy.zzz is the same namenode/hdfs instance where u are
> planning to
> >> store the hive tables.
> >>
> >>
> >>
> >>
> >> From: Jeremy Chow [mailto:coderplay@gmail.com]
> >> Sent: Monday, January 12, 2009 6:17 PM
> >>
> >> To: hive-user@hadoop.apache.org
> >> Subject: Re: Can hive load a table from a SequenceFile?
> >>
> >>
> >>
> >> Hey Joydeep,
> >>
> >> - load data infile yyy into table xxx
> >>
> >> hive> LOAD DATA INPATH '/user/zhoumin/log/web_log2009010606' OVERWRITE
> INTO
> >> TABLE log_data;
> >> FAILED: Error in semantic analysis: line 1:18 Path is not legal
> >> '/user/zhoumin/log/web_log2009010606': Cannot load data across
> filesystems,
> >> use load data local
> >> Time taken: 0.039 seconds
> >>
> >> It failed, that file existed in hdfs, but can not be loaded into hive.
> >>
> >> --
> >> My research interests are distributed systems, parallel computing and
> >> bytecode
> >> based virtual machine.
> >>
> >> http://coderplay.javaeye.com
> >
> >
>
>
--
My research interests are distributed systems, parallel computing and
bytecode based virtual machine.
http://coderplay.javaeye.com
Re: Can hive load a table from a SequenceFile?
Posted by Raghu Murthy <ra...@facebook.com>.
Can you paste the output of the following command?
$ hadoop dfs -ls /user/zhoumin/log/web_log2009010606
On 1/12/09 11:40 PM, "Jeremy Chow" <co...@gmail.com> wrote:
> I've tried it, but failed.
>
> hive> LOAD DATA INPATH 'hdfs://master:9000/user/zhoumin/log/web_log2009010606'
> OVERWRITE INTO TABLE log_data;
> FAILED: Error in semantic analysis: line 1:18 Path is not legal
> 'hdfs://master:9000/user/zhoumin/log/web_log2009010606': Cannot load data
> across filesystems, use load data local
> Time taken: 2.641 seconds
>
>
> On Tue, Jan 13, 2009 at 11:04 AM, Joydeep Sen Sarma <js...@facebook.com>
> wrote:
>> Please give a full uri like hdfs://xxx.yyy.zzz:9000/user/S
>>
>>
>>
>> Where xxx.yyy.zzz is the same namenode/hdfs instance where u are planning to
>> store the hive tables.
>>
>>
>>
>>
>> From: Jeremy Chow [mailto:coderplay@gmail.com]
>> Sent: Monday, January 12, 2009 6:17 PM
>>
>> To: hive-user@hadoop.apache.org
>> Subject: Re: Can hive load a table from a SequenceFile?
>>
>>
>>
>> Hey Joydeep,
>>
>> - load data infile yyy into table xxx
>>
>> hive> LOAD DATA INPATH '/user/zhoumin/log/web_log2009010606' OVERWRITE INTO
>> TABLE log_data;
>> FAILED: Error in semantic analysis: line 1:18 Path is not legal
>> '/user/zhoumin/log/web_log2009010606': Cannot load data across filesystems,
>> use load data local
>> Time taken: 0.039 seconds
>>
>> It failed, that file existed in hdfs, but can not be loaded into hive.
>>
>> --
>> My research interests are distributed systems, parallel computing and
>> bytecode
>> based virtual machine.
>>
>> http://coderplay.javaeye.com
>
>
Re: Can hive load a table from a SequenceFile?
Posted by Jeremy Chow <co...@gmail.com>.
I've tried it, but failed.
hive> LOAD DATA INPATH
'hdfs://master:9000/user/zhoumin/log/web_log2009010606' OVERWRITE INTO TABLE
log_data;
FAILED: Error in semantic analysis: line 1:18 Path is not legal
'hdfs://master:9000/user/zhoumin/log/web_log2009010606': Cannot load data
across filesystems, use load data local
Time taken: 2.641 seconds
On Tue, Jan 13, 2009 at 11:04 AM, Joydeep Sen Sarma <js...@facebook.com>wrote:
> Please give a full uri – like hdfs://xxx.yyy.zzz:9000/user/…
>
>
>
> Where xxx.yyy.zzz is the same namenode/hdfs instance where u are planning
> to store the hive tables.
>
>
> ------------------------------
>
> *From:* Jeremy Chow [mailto:coderplay@gmail.com]
> *Sent:* Monday, January 12, 2009 6:17 PM
> *To:* hive-user@hadoop.apache.org
> *Subject:* Re: Can hive load a table from a SequenceFile?
>
>
>
> Hey Joydeep,
>
> - load data infile yyy into table xxx
>
> hive> LOAD DATA INPATH '/user/zhoumin/log/web_log2009010606' OVERWRITE INTO
> TABLE log_data;
> FAILED: Error in semantic analysis: line 1:18 Path is not legal
> '/user/zhoumin/log/web_log2009010606': Cannot load data across filesystems,
> use load data local
> Time taken: 0.039 seconds
>
> It failed, that file existed in hdfs, but can not be loaded into hive.
>
> --
> My research interests are distributed systems, parallel computing and
> bytecode based virtual machine.
>
> http://coderplay.javaeye.com
>
--
My research interests are distributed systems, parallel computing and
bytecode based virtual machine.
http://coderplay.javaeye.com
RE: Can hive load a table from a SequenceFile?
Posted by Joydeep Sen Sarma <js...@facebook.com>.
Please give a full uri - like hdfs://xxx.yyy.zzz:9000/user/...
Where xxx.yyy.zzz is the same namenode/hdfs instance where u are planning to store the hive tables.
________________________________
From: Jeremy Chow [mailto:coderplay@gmail.com]
Sent: Monday, January 12, 2009 6:17 PM
To: hive-user@hadoop.apache.org
Subject: Re: Can hive load a table from a SequenceFile?
Hey Joydeep,
- load data infile yyy into table xxx
hive> LOAD DATA INPATH '/user/zhoumin/log/web_log2009010606' OVERWRITE INTO TABLE log_data;
FAILED: Error in semantic analysis: line 1:18 Path is not legal '/user/zhoumin/log/web_log2009010606': Cannot load data across filesystems, use load data local
Time taken: 0.039 seconds
It failed, that file existed in hdfs, but can not be loaded into hive.
--
My research interests are distributed systems, parallel computing and bytecode based virtual machine.
http://coderplay.javaeye.com
Re: Can hive load a table from a SequenceFile?
Posted by Jeremy Chow <co...@gmail.com>.
Hey Joydeep,
- load data infile yyy into table xxx
hive> LOAD DATA INPATH '/user/zhoumin/log/web_log2009010606' OVERWRITE INTO
TABLE log_data;
FAILED: Error in semantic analysis: line 1:18 Path is not legal
'/user/zhoumin/log/web_log2009010606': Cannot load data across filesystems,
use load data local
Time taken: 0.039 seconds
It failed, that file existed in hdfs, but can not be loaded into hive.
--
My research interests are distributed systems, parallel computing and
bytecode based virtual machine.
http://coderplay.javaeye.com
RE: Can hive load a table from a SequenceFile?
Posted by Joydeep Sen Sarma <js...@facebook.com>.
If u have a file of this type already - loading it into Hive is trivial.
- create table xxx (<schema>) ... stored as sequencefile
- load data infile yyy into table xxx
assuming yyy is already in hdfs. See the wiki for additional create table documentation: http://wiki.apache.org/hadoop/Hive/UserGuide , http://wiki.apache.org/hadoop/Hive/HiveQL
________________________________
From: Jeremy Chow [mailto:coderplay@gmail.com]
Sent: Monday, January 12, 2009 1:20 AM
To: hive-user@hadoop.apache.org
Subject: Can hive load a table from a SequenceFile?
hi list,
Let's assume that we have a SequenceFile with a format like this, <LongWritable key, Text value>.
All of the keys will discard when loading into a hive table. Each value, which is delimited by tabs, will be transformed into a row of a hive table.
Can hive load a table from this kind of files? How can I implement it ?
Thanks,
Jeremy
--
My research interests are distributed systems, parallel computing and bytecode based virtual machine.
http://coderplay.javaeye.com