You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Pradeep Kamath <pr...@yahoo-inc.com> on 2010/06/17 02:22:53 UTC
alter table add partition error
Hi,
I am trying to create an external table against already existing
data in sequencefile format. However I have written a custom Serde to
interpret the data. I am able to create the table fine but get the
exception shown in the session output below when I try to add partition
- any help would be greatly appreciated.
Thanks,
Pradeep
== session output ===
[pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER
TABLE mytable add partition(datestamp = '20091101', srcid = '10',action)
location '/user/pradeepk/mytable/20091101/10';"
10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException:
get_partition failed: unknown result
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
[pradeepk@chargesize:~/dev/howl]
== session output ===
/tmp/pradeepk/hive.log has:
2010-06-16 17:09:00,841 ERROR exec.DDLTask
(SessionState.java:printError(269)) - FAILED: Error in metadata:
org.apache.thrift.TApplicationException: get_partition failed: unknown
result
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.thrift.TApplicationException: get_partition failed: unknown
result
at
org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
at
org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
at
org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: org.apache.thrift.TApplicationException: get_partition
failed: unknown result
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get
_partition(ThriftHiveMetastore.java:931)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_part
ition(ThriftHiveMetastore.java:899)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMe
taStoreClient.java:500)
at
org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
... 15 more
The thrift server messages are:
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table :
db=default tbl=mytable
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store
with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize
called
10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition :
db=default tbl=mytable
Re: alter table add partition error
Posted by Edward Capriolo <ed...@gmail.com>.
On Fri, Jun 18, 2010 at 1:49 PM, Ning Zhang <nz...@facebook.com> wrote:
> Pradeep,
>
> I ran the commands you provided and it succeeded with the expected
> behavior.
>
> One possibility is that there are multiple versions of libthrift.jar in
> your CLASSPATH (hadoop & hive). Can you check in the Hadoop & Hive CLASSPATH
> so that no other libthrift.jar is there? What is in
> hive_config_without_thrift?
>
> Thanks,
> Ning
>
> On Jun 18, 2010, at 10:19 AM, Pradeep Kamath wrote:
>
> I think there are two separate issues here – I want to open a jira for
> the first one since I am now able to reproduce it even with text format with
> builtin Serdes. Essentially this is a bug in the thrift code (not sure if it
> is in the client or server) since the same alter table statement works fine
> when the hive client does not use thrift. Here are the details:
>
>
>
> *cat create_dummy.sql*
>
> CREATE external TABLE if not exists dummy (
>
>
>
> partition_name string
>
> ,partition_id int
>
> )
>
> PARTITIONED BY ( datestamp string, srcid string, action string, testid
> string )
>
> row format delimited
>
> stored as textfile
>
> location '/user/pradeepk/dummy';
>
>
>
> *hive -f create_dummy.sql*
>
> 10/06/18 10:13:36 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
> found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use
> core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of
> core-default.xml, mapred-default.xml and hdfs-default.xml respectively
>
> Hive history
> file=/tmp/pradeepk/hive_job_log_pradeepk_201006181013_184583537.txt
>
> OK
>
> Time taken: 0.627 seconds
>
> * *
>
> *hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid =
> '100',action='view',testid='10') location
> '/user/pradeepk/dummy/20100602/100/view/10';"*
>
> 10/06/18 10:14:11 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
> found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use
> core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of
> core-default.xml, mapred-default.xml and hdfs-default.xml respectively
>
> Hive history
> file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_700722546.txt
>
> *FAILED: Error in metadata: org.apache.thrift.TApplicationException:
> get_partition failed: unknown result*
>
> *FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask*
>
>
>
> *hive --config hive_conf_without_thrift -e "ALTER TABLE dummy add
> partition(datestamp = '20100602', srcid = '100',action='view',testid='10')
> location '/user/pradeepk/dummy/20100602/100/view/10';"*
>
> 10/06/18 10:14:31 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
> found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use
> core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of
> core-default.xml, mapred-default.xml and hdfs-default.xml respectively
>
> Hive history
> file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_598649843.txt
>
> *OK*
>
> Time taken: 5.849 seconds
>
>
>
> Is there some thrift setting I am missing or is this a bug? – If it is the
> latter, I can open a jira with the above details.
>
>
>
> Thanks,
>
> Pradeep
>
>
>
>
> ------------------------------
>
> *From:* Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
> *Sent:* Thursday, June 17, 2010 1:25 PM
> *To:* hive-user@hadoop.apache.org
> *Subject:* RE: alter table add partition error
>
>
>
> Here are the create table and alter table statements:
>
> CREATE external TABLE if not exists mytable (
>
>
>
> bc string
>
> ,src_spaceid string
>
> ,srcpvid string
>
> ,dstpvid string
>
> ,dst_spaceid string
>
> ,page_params map<string, string>
>
> ,clickinfo map<string, string>
>
> ,viewinfo array<map<string, string>>
>
>
>
> )
>
> PARTITIONED BY ( datestamp string, srcid string, action string, testid
> string )
>
> row format serde 'com.yahoo.mySerde’
>
> stored as inputformat 'org.apache.hadoop.mapred.SequenceFileInputFormat'
> outputformat 'org.apache.hadoop.mapred.SequenceFileOutputFormat'
>
> location '/user/pradeepk/mytable’;
>
>
>
> hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add
> partition(datestamp = '20091101', srcid =
> '19174',action='click',testid='NOTESTID') location
> '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"
>
>
>
> I get the following error:
>
> Hive history
> file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
>
> *FAILED: Error in metadata: org.apache.thrift.TApplicationException:
> get_partition failed: unknown result*
>
> *FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask*
>
> If I don’t use thrift and use a hive-site.xml to directly talk to the db,
> the alter table seems to succeed:
>
> hive --auxpath ult-serde.jar *--config hive_conf_without_thrift* -e "ALTER
> TABLE mytable add partition(datestamp = '20091101', srcid =
> '19174',action='click',testid='NOTESTID') location
> '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"
>
>
>
> However I get errors when I try to run a query:
>
> [pradeepk@chargesize:~/dev]*hive --auxpath ult-serde.jar --config
> hive_conf_without_thrift -e "select src_spaceid from ult_search_austria_ult
> where datestamp='20091101';"*
>
> 10/06/17 13:22:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
> found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use
> core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of
> core-default.xml, mapred-default.xml and hdfs-default.xml respectively
>
> Hive history
> file=/tmp/pradeepk/hive_job_log_pradeepk_201006171322_1913647383.txt
>
> Total MapReduce jobs = 1
>
> Launching Job 1 out of 1
>
> Number of reduce tasks is set to 0 since there's no reduce operator
>
> java.lang.IllegalArgumentException: Can not create a Path from an empty
> string
>
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)
>
> at org.apache.hadoop.fs.Path.<init>(Path.java:90)
>
> at org.apache.hadoop.fs.Path.<init>(Path.java:50)
>
> at
> org.apache.hadoop.mapred.JobClient.copyRemoteFiles(JobClient.java:523)
>
> at
> org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient.java:603)
>
> at
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)
>
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
>
> at
> org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:684)
>
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
>
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
>
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
>
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
>
>
>
> Any help is much appreciated.
>
>
>
> Pradeep
>
>
> ------------------------------
>
> *From:* Ashish Thusoo [mailto:athusoo@facebook.com]
> *Sent:* Thursday, June 17, 2010 11:15 AM
> *To:* hive-user@hadoop.apache.org
> *Subject:* RE: alter table add partition error
>
>
>
> hmm... Can you send the exact command and also the create table command for
> this table.
>
>
>
> Ashish
>
>
> ------------------------------
>
> *From:* Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
> *Sent:* Thursday, June 17, 2010 9:09 AM
> *To:* hive-user@hadoop.apache.org
> *Subject:* RE: alter table add partition error
>
> Sorry – that was a cut-paste error – I don’t have the action part – so I am
> specifying key-value pairs. Since what I am trying to do seems like a basic
> operation, I am wondering if it’s something to do with my Serde –
> unfortunately the error I see gives me no clue of what could be wrong – any
> help would be greatly appreciated!
>
>
>
> Thanks,
>
> Pradeep
>
>
> ------------------------------
>
> *From:* yq he [mailto:hhh.heyq@gmail.com]
> *Sent:* Wednesday, June 16, 2010 5:54 PM
> *To:* hive-user@hadoop.apache.org
> *Subject:* Re: alter table add partition error
>
>
>
> Hi Pradeep,
>
>
>
> partition definition need to be key-value pairs. partition key `action`
> seems missed the value.
>
>
>
> Thanks
>
> Yongqiang
>
> On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath <pr...@yahoo-inc.com>
> wrote:
>
> Hi,
>
> I am trying to create an external table against already existing data
> in sequencefile format. However I have written a custom Serde to interpret
> the data. I am able to create the table fine but get the exception shown in
> the session output below when I try to add partition – any help would be
> greatly appreciated.
>
>
>
> Thanks,
>
> Pradeep
>
>
>
> == session output ===
>
>
>
> [pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER
> TABLE mytable add partition(datestamp = '20091101', srcid = ‘10’,action)
> location '/user/pradeepk/mytable/20091101/10’;"
>
> 10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
> found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use
> core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of
> core-default.xml, mapred-default.xml and hdfs-default.xml respectively
>
> Hive history
> file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
>
> *FAILED: Error in metadata: org.apache.thrift.TApplicationException:
> get_partition failed: unknown result*
>
> *FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask*
>
> [pradeepk@chargesize:~/dev/howl]
>
>
>
> == session output ===
>
>
>
> /tmp/pradeepk/hive.log has:
>
> 2010-06-16 17:09:00,841 ERROR exec.DDLTask
> (SessionState.java:printError(269)) - FAILED: Error in metadata:
> org.apache.thrift.TApplicationException: get_partition failed: unknown
> result
>
> org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.thrift.TApplicationException: get_partition failed: unknown
> result
>
> at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
>
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)
>
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)
>
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
>
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
>
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
>
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
>
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
>
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
>
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
>
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>
> at java.lang.reflect.Method.invoke(Method.java:597)
>
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>
> *Caused by: org.apache.thrift.TApplicationException: get_partition failed:
> unknown result*
>
> * at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition(ThriftHiveMetastore.java:931)
> *
>
> * at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition(ThriftHiveMetastore.java:899)
> *
>
> * at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:500)
> *
>
> * at
> org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)*
>
> * ... 15 more*
>
>
>
> The thrift server messages are:
>
> 10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table : db=default
> tbl=mytable
>
> 10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store with
> implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
>
> 10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize
> called
>
> 10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore
>
> 10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition :
> db=default tbl=mytable
>
>
>
>
>
>
>
>
>
Also I find that serde's and input formats should be put in hive_home/auxlib
as well as hive_home/lib. auxlib directory is not created by default but if
it exists the hive shell bootstrap picks it up.
hive.aux.jars.path not used - get a SerDe does not exist error
Posted by Karthik <ka...@yahoo.com>.
I have my custom SerDe classes (as jar files) under /home/hadoop/hive/lib folder and I have set "hive.aux.jars.path" property in my "hive-site.xml" file to this location (value: "file:///home/hadoop/hive/lib/")
When I create a able (or query an existing table), I get a SerDe does not exist error, althought, it works fine if I set the value to full absolute path of the file name like this: "file:///home/hadoop/hive/lib/myserde.jar", but again, my HWI complains with same error even if I set full path name of the JAR.
It also works well if I run "hive --auxpath /home/hadoop/hive/lib".
I need to provide multiple JAR files under auxpath. I use Hive 0.5+20. Should I not relay on "hive.aux.jars.path" set in hive-site.xml and only use --auxpath param?
Please help.
Regards,
Karthik.
RE: alter table add partition error
Posted by John Sichi <js...@facebook.com>.
I don't know who owns that server, but I happened to have an old copy of the tarball lying around, so I've put it somewhere you can download it:
http://ansible.eigenbase.org/download/thrift-instant-r790732.tar.gz
For the long run we'll figure out what's up with the latest thrift and how to upgrade.
JVS
________________________________________
From: Pradeep Kamath [pradeepk@yahoo-inc.com]
Sent: Wednesday, June 23, 2010 9:03 AM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
In the second step of
http://wiki.apache.org/hadoop/Hive/HowToContribute#Generating_Code
“Use the approved version of thrift. This is currently thrift-instant-r790732, which you can obtain from http://instant.thrift-rpc.org”
Looks like that site is down since it is not responding to requests – is there an alternate url I can try? Also, is the approved version still thrift-instant-r790732?
________________________________
From: Paul Yang [mailto:pyang@facebook.com]
Sent: Tuesday, June 22, 2010 6:06 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Hey Pradeep,
Yep, that’s on the right path :) Since you made a change to hive_metastore.thrift, you should run the thrift compiler by calling ‘ant thriftif’ within trunk/metastore. But before you can do that, the thrift compiler has to be in your path. JVS has a nice write up on how to do this:
http://wiki.apache.org/hadoop/Hive/HowToContribute#Generating_Code
Also, I’ve set up a JIRA for this issue at:
https://issues.apache.org/jira/browse/HIVE-1428
Can you post your patch there? It makes it a lot easier to track issues. Thanks!
-Paul
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Tuesday, June 22, 2010 10:22 AM
To: hive-user@hadoop.apache.org
Subject: Re: alter table add partition error
Hi Paul,
I have attached a draft of a patch (incomplete). The problem I am facing is that to make this change I seem to need to change the thrift idl (hive_metastore.thrift). Looking at the code structure it seems like the generate java files are checked in rather than generated through the build (is this so that the build does not depend on the thrift compiler?). I tried downloading and building thrift but hit issues in the make install - wanted to check with you if the patch looks correct so far or if I am on the wrong path. Also if the direction is right, I need some guidance on generating the java files from the modified idl.
Thanks,
Pradeep
Paul Yang wrote:
Well, the section of code you are referring to is automatically generated by the thrift compiler so we can’t modify it. As the inability to return null is a limitation of Thrift, we can work around that issue by throwing an exception instead.
Check out this thread:
http://publists.facebook.com/pipermail/thrift/2007-September/000085.html
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Monday, June 21, 2010 5:46 PM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: alter table add partition error
Hi Paul,
Yes I don’t mind working on a patch though I have no knowledge of how thrift code works. I was also confused by your suggestion for a fix. I thought the fix would be in ThriftHiveMetaStore.Client.recv_get_partition() – to first check if(result.o1 != null) throw result.o1, then to just return result.success and not throw the “unknown result” exception. I tried something along these lines and the alter table worked. Am I missing something?
Thanks,
Pradeep
________________________________
From: Paul Yang [mailto:pyang@facebook.com]
Sent: Monday, June 21, 2010 2:35 PM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: alter table add partition error
Ah okay. So what’s going on is that getPartition() is supposed to return null if the specified partition doesn’t exist. But because Thrift can’t handle null return values, we are seeing this exception. This isn’t a problem with a local metastore because Thrift isn’t used.
One solution to this problem would be modify ObjectStore.getPartition() to throw a NoSuchObjectException instead of returning null for non-existent partitions. Then Hive.getPartition() can catch the exception and return null to retain the original behavior.
Pradeep, are you interested in working on a patch for this problem?
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Monday, June 21, 2010 2:11 PM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: alter table add partition error
I tried debugging in code a little more. Here is what I found:
The code in ThriftHiveMetaStore eventually makes a call – get_partition() passing the partition key values for the partition I am trying to add using alter table. I assume this is to check that the partition doesn’t already exist.
I added a debug line in the following code:
public Partition recv_get_partition() throws MetaException, TException
{
TMessage msg = iprot_.readMessageBegin();
if (msg.type == TMessageType.EXCEPTION) {
TApplicationException x = TApplicationException.read(iprot_);
iprot_.readMessageEnd();
throw x;
}
get_partition_result result = new get_partition_result();
result.read(iprot_);
System.err.println("XXX: result:" + result);
iprot_.readMessageEnd();
if (result.isSetSuccess()) {
return result.success;
}
if (result.o1 != null) {
throw result.o1;
}
throw new TApplicationException(TApplicationException.MISSING_RESULT, "get_partition failed: unknown result");
}
I also put a debug statements in the read() method:
public void read(TProtocol iprot) throws TException {
TField field;
iprot.readStructBegin();
System.err.println("XXX: Reading TProtocol object:");
while (true)
{
field = iprot.readFieldBegin();
System.err.println("XXX: field just read:" + field);
if (field.type == TType.STOP) {
break;
I got
XXX: Reading TProtocol object:
XXX: field just read:<TField name:'' type:0 field-id:0>
XXX: result:get_partition_result(success:null, o1:null)
The field read in the thrift response message is of type “STOP” and with id of type SUCCESS. This seems right since there are no existing partitions. But the way the rest of the code handles this, results in the exception.
Any pointers?
TIA,
Pradeep
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Friday, June 18, 2010 2:51 PM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: alter table add partition error
Looks like the standalone script works fine against the existing partition:
./ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name default dummy datestamp=20100602/srcid=100/action=view/testid=10
Partition(parameters={'transient_lastDdlTime': '1276881277'}, tableName='dummy', createTime=1276881277, lastAccessTime=0, values=['20100602', '100', 'view', '10'], dbName='default', sd=StorageDescriptor(outputFormat='org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat', sortCols=[], inputFormat='org.apache.hadoop.mapred.TextInputFormat', cols=[FieldSchema(comment=None, type='string', name='partition_name'), FieldSchema(comment=None, type='int', name='partition_id')], compressed=False, bucketCols=[], numBuckets=-1, parameters={}, serdeInfo=SerDeInfo(serializationLib='org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe', name=None, parameters={'serialization.format': '1'}), location='hdfs://wilbur21.labs.corp.sp1.yahoo.com/user/pradeepk/dummy/20100602/100/view/10'))
[pradeepk@chargesize:~/dev/howl/src/metastore/src/gen-py/hive_metastore]
However when I tried to add another partition with the hive cli using thrift:
hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid = '100',action='click',testid='10') location '/user/pradeepk/dummy/20100602/100/click/10';"
10/06/18 14:49:13 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181449_1158492515.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
tail -30 /tmp/pradeepk/hive.log
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
2010-06-18 14:49:14,124 ERROR exec.DDLTask (SessionState.java:printError(277)) - FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.TApplicationException: get_partition failed: unknown result
at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
at org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:255)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:169)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:633)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:506)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:384)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:267)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: org.apache.thrift.TApplicationException: get_partition failed: unknown result
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition(ThriftHiveMetastore.java:931)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition(ThriftHiveMetastore.java:899)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:500)
at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
... 15 more
2010-06-18 14:49:14,124 ERROR ql.Driver (SessionState.java:printError(277)) - FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
________________________________
From: Paul Yang [mailto:pyang@facebook.com]
Sent: Friday, June 18, 2010 2:19 PM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: alter table add partition error
Looks like the thrift python libraries aren’t in your path. run:
export PYTHONPATH=<path-to-trunk>/build/dist/lib/py/
before trying the remote command
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Friday, June 18, 2010 1:38 PM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: alter table add partition error
Sorry, don’t know python – so couldn’t make sense out of the following error when I run the command you suggested:
[src/metastore/src/gen-py/hive_metastore]./ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name default dummy datestamp=20100602/srcid=100/action=view/testid=10
Traceback (most recent call last):
File "./ThriftHiveMetastore-remote", line 11, in ?
from thrift.transport import TTransport
ImportError: No module named thrift.transport
Thanks,
Pradeep
________________________________
From: Paul Yang [mailto:pyang@facebook.com]
Sent: Friday, June 18, 2010 12:10 PM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: alter table add partition error
Hi Pradeep,
In trunk/metastore/src/gen-py/hive_metastore/, there is a script called ThriftHiveMetastore-remote that can be used to test out the thrift server independent of the CLI. As a quick test to narrow down the problem, after the partition is created, can you try running
ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name default dummy datestamp=20100602/srcid=100/action=view/testid=10
Cheers,
Paul
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Friday, June 18, 2010 11:26 AM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: alter table add partition error
Hi Ning,
I have pasted below [1] the values for HADOOP_CLASSPATH and CLASSPATH from an “env” command printed just before the following line in execHiveCmd.sh
exec $HADOOP jar ${HIVE_LIB}/hive-cli-*.jar $CLASS $HIVE_OPTS "$@"
There is only one occurrence of libthrift.jar whose cksum matches the one from lib/libthrift.jar from a hive checkout build.
The hive_config_without_thrift is a conf directory containing a hive-site.xml with jdo settings to connect to the database directly. By default I have hive set up to use a hive-site.xml [2] which connects to a thrift server running on localhost.
Any pointers appreciated!
Thanks,
Pradeep
[1]
HADOOP_CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libexec/hive_client/lib/derby.jar:/home/y/libex ec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.jar:/home/y/libexec/hi ve_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:
CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libexec/hive_client/lib/derby.jar:/home/y/libex ec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.jar:/home/y/libexec/hi ve_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:
[2]
cat /home/y/libexec/hive_client/conf/hive-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<!-- Hive Configuration can either be stored in this file or in the hadoop configuration files -->
<!-- that are implied by Hadoop setup variables. -->
<!-- Aside from Hadoop setup variables - this file is provided as a convenience so that Hive -->
<!-- users do not have to edit hadoop configuration files (that may be managed as a centralized -->
<!-- resource). -->
<!-- Hive Execution Parameters -->
<property>
<name>hive.metastore.uris</name>
<value>thrift://localhost:9080</value>
<description>host and port for the thrift metastore server</description>
</property>
<property>
<name>hive.metastore.local</name>
<value>false</value>
<description>controls whether to connect to remove metastore server or open a new metastore server in Hive Client JVM</description>
</property>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/user/pradeepk/hive/warehouse</value>
<description>location of default database for the warehouse</description>
</property>
</configuration>
________________________________
From: Ning Zhang [mailto:nzhang@facebook.com]
Sent: Friday, June 18, 2010 10:50 AM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: alter table add partition error
Pradeep,
I ran the commands you provided and it succeeded with the expected behavior.
One possibility is that there are multiple versions of libthrift.jar in your CLASSPATH (hadoop & hive). Can you check in the Hadoop & Hive CLASSPATH so that no other libthrift.jar is there? What is in hive_config_without_thrift?
Thanks,
Ning
On Jun 18, 2010, at 10:19 AM, Pradeep Kamath wrote:
I think there are two separate issues here – I want to open a jira for the first one since I am now able to reproduce it even with text format with builtin Serdes. Essentially this is a bug in the thrift code (not sure if it is in the client or server) since the same alter table statement works fine when the hive client does not use thrift. Here are the details:
cat create_dummy.sql
CREATE external TABLE if not exists dummy (
partition_name string
,partition_id int
)
PARTITIONED BY ( datestamp string, srcid string, action string, testid string )
row format delimited
stored as textfile
location '/user/pradeepk/dummy';
hive -f create_dummy.sql
10/06/18 10:13:36 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181013_184583537.txt
OK
Time taken: 0.627 seconds
hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid = '100',action='view',testid='10') location '/user/pradeepk/dummy/20100602/100/view/10';"
10/06/18 10:14:11 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_700722546.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
hive --config hive_conf_without_thrift -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid = '100',action='view',testid='10') location '/user/pradeepk/dummy/20100602/100/view/10';"
10/06/18 10:14:31 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_598649843.txt
OK
Time taken: 5.849 seconds
Is there some thrift setting I am missing or is this a bug? – If it is the latter, I can open a jira with the above details.
Thanks,
Pradeep
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Thursday, June 17, 2010 1:25 PM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: alter table add partition error
Here are the create table and alter table statements:
CREATE external TABLE if not exists mytable (
bc string
,src_spaceid string
,srcpvid string
,dstpvid string
,dst_spaceid string
,page_params map<string, string>
,clickinfo map<string, string>
,viewinfo array<map<string, string>>
)
PARTITIONED BY ( datestamp string, srcid string, action string, testid string )
row format serde 'com.yahoo.mySerde’
stored as inputformat 'org.apache.hadoop.mapred.SequenceFileInputFormat' outputformat 'org.apache.hadoop.mapred.SequenceFileOutputFormat'
location '/user/pradeepk/mytable’;
hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '19174',action='click',testid='NOTESTID') location '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"
I get the following error:
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
If I don’t use thrift and use a hive-site.xml to directly talk to the db, the alter table seems to succeed:
hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '19174',action='click',testid='NOTESTID') location '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"
However I get errors when I try to run a query:
[pradeepk@chargesize:~/dev]hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "select src_spaceid from ult_search_austria_ult where datestamp='20091101';"
10/06/17 13:22:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006171322_1913647383.txt
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
java.lang.IllegalArgumentException: Can not create a Path from an empty string
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)
at org.apache.hadoop.fs.Path.<init>(Path.java:90)
at org.apache.hadoop.fs.Path.<init>(Path.java:50)
at org.apache.hadoop.mapred.JobClient.copyRemoteFiles(JobClient.java:523)
at org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient.java:603)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:684)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
Any help is much appreciated.
Pradeep
________________________________
From: Ashish Thusoo [mailto:athusoo@facebook.com]
Sent: Thursday, June 17, 2010 11:15 AM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: alter table add partition error
hmm... Can you send the exact command and also the create table command for this table.
Ashish
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Thursday, June 17, 2010 9:09 AM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: alter table add partition error
Sorry – that was a cut-paste error – I don’t have the action part – so I am specifying key-value pairs. Since what I am trying to do seems like a basic operation, I am wondering if it’s something to do with my Serde – unfortunately the error I see gives me no clue of what could be wrong – any help would be greatly appreciated!
Thanks,
Pradeep
________________________________
From: yq he [mailto:hhh.heyq@gmail.com]
Sent: Wednesday, June 16, 2010 5:54 PM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: alter table add partition error
Hi Pradeep,
partition definition need to be key-value pairs. partition key `action` seems missed the value.
Thanks
Yongqiang
On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath <pr...@yahoo-inc.com>> wrote:
Hi,
I am trying to create an external table against already existing data in sequencefile format. However I have written a custom Serde to interpret the data. I am able to create the table fine but get the exception shown in the session output below when I try to add partition – any help would be greatly appreciated.
Thanks,
Pradeep
== session output ===
[pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = ‘10’,action) location '/user/pradeepk/mytable/20091101/10’;"
10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
[pradeepk@chargesize:~/dev/howl]
== session output ===
/tmp/pradeepk/hive.log has:
2010-06-16 17:09:00,841 ERROR exec.DDLTask (SessionState.java:printError(269)) - FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.TApplicationException: get_partition failed: unknown result
at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
at org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: org.apache.thrift.TApplicationException: get_partition failed: unknown result
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition(ThriftHiveMetastore.java:931)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition(ThriftHiveMetastore.java:899)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:500)
at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
... 15 more
The thrift server messages are:
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table : db=default tbl=mytable
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize called
10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition : db=default tbl=mytable
RE: alter table add partition error
Posted by Pradeep Kamath <pr...@yahoo-inc.com>.
I am having difficulty starting a thrift service instance in the junit
test for this patch -
https://issues.apache.org/jira/browse/HIVE-1428?focusedCommentId=1288361
4&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpane
l#action_12883614 - any pointers would be appreciated.
Thanks,
Pradeep
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Friday, June 25, 2010 1:37 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
I have posted a patch for this - not sure how to write unit tests for it
- so patch doesn't have tests.
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Wednesday, June 23, 2010 9:04 AM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
In the second step of
http://wiki.apache.org/hadoop/Hive/HowToContribute#Generating_Code
"Use the approved version of thrift. This is currently
thrift-instant-r790732, which you can obtain from
http://instant.thrift-rpc.org"
Looks like that site is down since it is not responding to requests - is
there an alternate url I can try? Also, is the approved version still
thrift-instant-r790732?
________________________________
From: Paul Yang [mailto:pyang@facebook.com]
Sent: Tuesday, June 22, 2010 6:06 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Hey Pradeep,
Yep, that's on the right path :) Since you made a change to
hive_metastore.thrift, you should run the thrift compiler by calling
'ant thriftif' within trunk/metastore. But before you can do that, the
thrift compiler has to be in your path. JVS has a nice write up on how
to do this:
http://wiki.apache.org/hadoop/Hive/HowToContribute#Generating_Code
Also, I've set up a JIRA for this issue at:
https://issues.apache.org/jira/browse/HIVE-1428
Can you post your patch there? It makes it a lot easier to track issues.
Thanks!
-Paul
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Tuesday, June 22, 2010 10:22 AM
To: hive-user@hadoop.apache.org
Subject: Re: alter table add partition error
Hi Paul,
I have attached a draft of a patch (incomplete). The problem I am
facing is that to make this change I seem to need to change the thrift
idl (hive_metastore.thrift). Looking at the code structure it seems like
the generate java files are checked in rather than generated through the
build (is this so that the build does not depend on the thrift
compiler?). I tried downloading and building thrift but hit issues in
the make install - wanted to check with you if the patch looks correct
so far or if I am on the wrong path. Also if the direction is right, I
need some guidance on generating the java files from the modified idl.
Thanks,
Pradeep
Paul Yang wrote:
Well, the section of code you are referring to is automatically
generated by the thrift compiler so we can't modify it. As the inability
to return null is a limitation of Thrift, we can work around that issue
by throwing an exception instead.
Check out this thread:
http://publists.facebook.com/pipermail/thrift/2007-September/000085.html
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Monday, June 21, 2010 5:46 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Hi Paul,
Yes I don't mind working on a patch though I have no knowledge of how
thrift code works. I was also confused by your suggestion for a fix. I
thought the fix would be in
ThriftHiveMetaStore.Client.recv_get_partition() - to first check
if(result.o1 != null) throw result.o1, then to just return
result.success and not throw the "unknown result" exception. I tried
something along these lines and the alter table worked. Am I missing
something?
Thanks,
Pradeep
________________________________
From: Paul Yang [mailto:pyang@facebook.com]
Sent: Monday, June 21, 2010 2:35 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Ah okay. So what's going on is that getPartition() is supposed to return
null if the specified partition doesn't exist. But because Thrift can't
handle null return values, we are seeing this exception. This isn't a
problem with a local metastore because Thrift isn't used.
One solution to this problem would be modify ObjectStore.getPartition()
to throw a NoSuchObjectException instead of returning null for
non-existent partitions. Then Hive.getPartition() can catch the
exception and return null to retain the original behavior.
Pradeep, are you interested in working on a patch for this problem?
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Monday, June 21, 2010 2:11 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
I tried debugging in code a little more. Here is what I found:
The code in ThriftHiveMetaStore eventually makes a call -
get_partition() passing the partition key values for the partition I am
trying to add using alter table. I assume this is to check that the
partition doesn't already exist.
I added a debug line in the following code:
public Partition recv_get_partition() throws MetaException, TException
{
TMessage msg = iprot_.readMessageBegin();
if (msg.type == TMessageType.EXCEPTION) {
TApplicationException x = TApplicationException.read(iprot_);
iprot_.readMessageEnd();
throw x;
}
get_partition_result result = new get_partition_result();
result.read(iprot_);
System.err.println("XXX: result:" + result);
iprot_.readMessageEnd();
if (result.isSetSuccess()) {
return result.success;
}
if (result.o1 != null) {
throw result.o1;
}
throw new
TApplicationException(TApplicationException.MISSING_RESULT,
"get_partition failed: unknown result");
}
I also put a debug statements in the read() method:
public void read(TProtocol iprot) throws TException {
TField field;
iprot.readStructBegin();
System.err.println("XXX: Reading TProtocol object:");
while (true)
{
field = iprot.readFieldBegin();
System.err.println("XXX: field just read:" + field);
if (field.type == TType.STOP) {
break;
I got
XXX: Reading TProtocol object:
XXX: field just read:<TField name:'' type:0 field-id:0>
XXX: result:get_partition_result(success:null, o1:null)
The field read in the thrift response message is of type "STOP" and with
id of type SUCCESS. This seems right since there are no existing
partitions. But the way the rest of the code handles this, results in
the exception.
Any pointers?
TIA,
Pradeep
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Friday, June 18, 2010 2:51 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Looks like the standalone script works fine against the existing
partition:
./ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name
default dummy datestamp=20100602/srcid=100/action=view/testid=10
Partition(parameters={'transient_lastDdlTime': '1276881277'},
tableName='dummy', createTime=1276881277, lastAccessTime=0,
values=['20100602', '100', 'view', '10'], dbName='default',
sd=StorageDescriptor(outputFormat='org.apache.hadoop.hive.ql.io.HiveIgno
reKeyTextOutputFormat', sortCols=[],
inputFormat='org.apache.hadoop.mapred.TextInputFormat',
cols=[FieldSchema(comment=None, type='string', name='partition_name'),
FieldSchema(comment=None, type='int', name='partition_id')],
compressed=False, bucketCols=[], numBuckets=-1, parameters={},
serdeInfo=SerDeInfo(serializationLib='org.apache.hadoop.hive.serde2.lazy
.LazySimpleSerDe', name=None, parameters={'serialization.format': '1'}),
location='hdfs://wilbur21.labs.corp.sp1.yahoo.com/user/pradeepk/dummy/20
100602/100/view/10'))
[pradeepk@chargesize:~/dev/howl/src/metastore/src/gen-py/hive_metastore]
However when I tried to add another partition with the hive cli using
thrift:
hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid
= '100',action='click',testid='10') location
'/user/pradeepk/dummy/20100602/100/click/10';"
10/06/18 14:49:13 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006181449_1158492515.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException:
get_partition failed: unknown result
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
tail -30 /tmp/pradeepk/hive.log
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
2010-06-18 14:49:14,124 ERROR exec.DDLTask
(SessionState.java:printError(277)) - FAILED: Error in metadata:
org.apache.thrift.TApplicationException: get_partition failed: unknown
result
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.thrift.TApplicationException: get_partition failed: unknown
result
at
org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
at
org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:255)
at
org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:169)
at
org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:633)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:506)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:384)
at
org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:267)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: org.apache.thrift.TApplicationException: get_partition
failed: unknown result
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get
_partition(ThriftHiveMetastore.java:931)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_part
ition(ThriftHiveMetastore.java:899)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMe
taStoreClient.java:500)
at
org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
... 15 more
2010-06-18 14:49:14,124 ERROR ql.Driver
(SessionState.java:printError(277)) - FAILED: Execution Error, return
code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
________________________________
From: Paul Yang [mailto:pyang@facebook.com]
Sent: Friday, June 18, 2010 2:19 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Looks like the thrift python libraries aren't in your path. run:
export PYTHONPATH=<path-to-trunk>/build/dist/lib/py/
before trying the remote command
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Friday, June 18, 2010 1:38 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Sorry, don't know python - so couldn't make sense out of the following
error when I run the command you suggested:
[src/metastore/src/gen-py/hive_metastore]./ThriftHiveMetastore-remote -h
localhost:9080 get_partition_by_name default dummy
datestamp=20100602/srcid=100/action=view/testid=10
Traceback (most recent call last):
File "./ThriftHiveMetastore-remote", line 11, in ?
from thrift.transport import TTransport
ImportError: No module named thrift.transport
Thanks,
Pradeep
________________________________
From: Paul Yang [mailto:pyang@facebook.com]
Sent: Friday, June 18, 2010 12:10 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Hi Pradeep,
In trunk/metastore/src/gen-py/hive_metastore/, there is a script called
ThriftHiveMetastore-remote that can be used to test out the thrift
server independent of the CLI. As a quick test to narrow down the
problem, after the partition is created, can you try running
ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name
default dummy datestamp=20100602/srcid=100/action=view/testid=10
Cheers,
Paul
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Friday, June 18, 2010 11:26 AM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Hi Ning,
I have pasted below [1] the values for HADOOP_CLASSPATH and CLASSPATH
from an "env" command printed just before the following line in
execHiveCmd.sh
exec $HADOOP jar ${HIVE_LIB}/hive-cli-*.jar $CLASS $HIVE_OPTS "$@"
There is only one occurrence of libthrift.jar whose cksum matches the
one from lib/libthrift.jar from a hive checkout build.
The hive_config_without_thrift is a conf directory containing a
hive-site.xml with jdo settings to connect to the database directly. By
default I have hive set up to use a hive-site.xml [2] which connects to
a thrift server running on localhost.
Any pointers appreciated!
Thanks,
Pradeep
[1]
HADOOP_CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_c
lient/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.
1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home
/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_cl
ient/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/c
ommons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.j
ar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/lib
exec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_
client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucl
eus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus
-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1
.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home
/y/libexec/hive_client/lib/derby.jar:/home/y/libex
ec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbas
e-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.ja
r:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hiv
e_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_
contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/
libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_clie
nt/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.
0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/l
ibexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/
lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.
6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/
y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/l
ib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libe
xec/hive_client/lib/libfb303.jar:/home/y/libexec/hi
ve_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15
.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_cli
ent/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtem
plate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y
/libexec/hive_client/lib/zookeeper-3.2.2.jar:
CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/l
ib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/
home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libe
xec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/li
b/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-
dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/hom
e/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hi
ve_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/
lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-con
nectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1
.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:
/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libe
xec/hive_client/lib/derby.jar:/home/y/libex
ec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbas
e-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.ja
r:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hiv
e_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_
contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/
libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_clie
nt/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.
0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/l
ibexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/
lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.
6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/
y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/l
ib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libe
xec/hive_client/lib/libfb303.jar:/home/y/libexec/hi
ve_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15
.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_cli
ent/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtem
plate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y
/libexec/hive_client/lib/zookeeper-3.2.2.jar:
[2]
cat /home/y/libexec/hive_client/conf/hive-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<!-- Hive Configuration can either be stored in this file or in the
hadoop configuration files -->
<!-- that are implied by Hadoop setup variables.
-->
<!-- Aside from Hadoop setup variables - this file is provided as a
convenience so that Hive -->
<!-- users do not have to edit hadoop configuration files (that may be
managed as a centralized -->
<!-- resource).
-->
<!-- Hive Execution Parameters -->
<property>
<name>hive.metastore.uris</name>
<value>thrift://localhost:9080</value>
<description>host and port for the thrift metastore
server</description>
</property>
<property>
<name>hive.metastore.local</name>
<value>false</value>
<description>controls whether to connect to remove metastore server or
open a new metastore server in Hive Client JVM</description>
</property>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/user/pradeepk/hive/warehouse</value>
<description>location of default database for the
warehouse</description>
</property>
</configuration>
________________________________
From: Ning Zhang [mailto:nzhang@facebook.com]
Sent: Friday, June 18, 2010 10:50 AM
To: hive-user@hadoop.apache.org
Subject: Re: alter table add partition error
Pradeep,
I ran the commands you provided and it succeeded with the expected
behavior.
One possibility is that there are multiple versions of libthrift.jar in
your CLASSPATH (hadoop & hive). Can you check in the Hadoop & Hive
CLASSPATH so that no other libthrift.jar is there? What is in
hive_config_without_thrift?
Thanks,
Ning
On Jun 18, 2010, at 10:19 AM, Pradeep Kamath wrote:
I think there are two separate issues here - I want to open a jira for
the first one since I am now able to reproduce it even with text format
with builtin Serdes. Essentially this is a bug in the thrift code (not
sure if it is in the client or server) since the same alter table
statement works fine when the hive client does not use thrift. Here are
the details:
cat create_dummy.sql
CREATE external TABLE if not exists dummy (
partition_name string
,partition_id int
)
PARTITIONED BY ( datestamp string, srcid string, action string, testid
string )
row format delimited
stored as textfile
location '/user/pradeepk/dummy';
hive -f create_dummy.sql
10/06/18 10:13:36 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006181013_184583537.txt
OK
Time taken: 0.627 seconds
hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid
= '100',action='view',testid='10') location
'/user/pradeepk/dummy/20100602/100/view/10';"
10/06/18 10:14:11 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_700722546.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException:
get_partition failed: unknown result
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
hive --config hive_conf_without_thrift -e "ALTER TABLE dummy add
partition(datestamp = '20100602', srcid =
'100',action='view',testid='10') location
'/user/pradeepk/dummy/20100602/100/view/10';"
10/06/18 10:14:31 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_598649843.txt
OK
Time taken: 5.849 seconds
Is there some thrift setting I am missing or is this a bug? - If it is
the latter, I can open a jira with the above details.
Thanks,
Pradeep
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Thursday, June 17, 2010 1:25 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Here are the create table and alter table statements:
CREATE external TABLE if not exists mytable (
bc string
,src_spaceid string
,srcpvid string
,dstpvid string
,dst_spaceid string
,page_params map<string, string>
,clickinfo map<string, string>
,viewinfo array<map<string, string>>
)
PARTITIONED BY ( datestamp string, srcid string, action string, testid
string )
row format serde 'com.yahoo.mySerde'
stored as inputformat 'org.apache.hadoop.mapred.SequenceFileInputFormat'
outputformat 'org.apache.hadoop.mapred.SequenceFileOutputFormat'
location '/user/pradeepk/mytable';
hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add
partition(datestamp = '20091101', srcid =
'19174',action='click',testid='NOTESTID') location
'/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"
I get the following error:
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException:
get_partition failed: unknown result
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
If I don't use thrift and use a hive-site.xml to directly talk to the
db, the alter table seems to succeed:
hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "ALTER
TABLE mytable add partition(datestamp = '20091101', srcid =
'19174',action='click',testid='NOTESTID') location
'/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"
However I get errors when I try to run a query:
[pradeepk@chargesize:~/dev]hive --auxpath ult-serde.jar --config
hive_conf_without_thrift -e "select src_spaceid from
ult_search_austria_ult where datestamp='20091101';"
10/06/17 13:22:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006171322_1913647383.txt
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
java.lang.IllegalArgumentException: Can not create a Path from an empty
string
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)
at org.apache.hadoop.fs.Path.<init>(Path.java:90)
at org.apache.hadoop.fs.Path.<init>(Path.java:50)
at
org.apache.hadoop.mapred.JobClient.copyRemoteFiles(JobClient.java:523)
at
org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient
.java:603)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)
at
org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
at
org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:684)
at
org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
Any help is much appreciated.
Pradeep
________________________________
From: Ashish Thusoo [mailto:athusoo@facebook.com]
Sent: Thursday, June 17, 2010 11:15 AM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
hmm... Can you send the exact command and also the create table command
for this table.
Ashish
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Thursday, June 17, 2010 9:09 AM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Sorry - that was a cut-paste error - I don't have the action part - so I
am specifying key-value pairs. Since what I am trying to do seems like a
basic operation, I am wondering if it's something to do with my Serde -
unfortunately the error I see gives me no clue of what could be wrong -
any help would be greatly appreciated!
Thanks,
Pradeep
________________________________
From: yq he [mailto:hhh.heyq@gmail.com]
Sent: Wednesday, June 16, 2010 5:54 PM
To: hive-user@hadoop.apache.org
Subject: Re: alter table add partition error
Hi Pradeep,
partition definition need to be key-value pairs. partition key `action`
seems missed the value.
Thanks
Yongqiang
On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath <pr...@yahoo-inc.com>
wrote:
Hi,
I am trying to create an external table against already existing
data in sequencefile format. However I have written a custom Serde to
interpret the data. I am able to create the table fine but get the
exception shown in the session output below when I try to add partition
- any help would be greatly appreciated.
Thanks,
Pradeep
== session output ===
[pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER
TABLE mytable add partition(datestamp = '20091101', srcid = '10',action)
location '/user/pradeepk/mytable/20091101/10';"
10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException:
get_partition failed: unknown result
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
[pradeepk@chargesize:~/dev/howl]
== session output ===
/tmp/pradeepk/hive.log has:
2010-06-16 17:09:00,841 ERROR exec.DDLTask
(SessionState.java:printError(269)) - FAILED: Error in metadata:
org.apache.thrift.TApplicationException: get_partition failed: unknown
result
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.thrift.TApplicationException: get_partition failed: unknown
result
at
org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
at
org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
at
org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: org.apache.thrift.TApplicationException: get_partition
failed: unknown result
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get
_partition(ThriftHiveMetastore.java:931)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_part
ition(ThriftHiveMetastore.java:899)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMe
taStoreClient.java:500)
at
org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
... 15 more
The thrift server messages are:
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table :
db=default tbl=mytable
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store
with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize
called
10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition :
db=default tbl=mytable
RE: alter table add partition error
Posted by Pradeep Kamath <pr...@yahoo-inc.com>.
I have posted a patch for this - not sure how to write unit tests for it
- so patch doesn't have tests.
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Wednesday, June 23, 2010 9:04 AM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
In the second step of
http://wiki.apache.org/hadoop/Hive/HowToContribute#Generating_Code
"Use the approved version of thrift. This is currently
thrift-instant-r790732, which you can obtain from
http://instant.thrift-rpc.org"
Looks like that site is down since it is not responding to requests - is
there an alternate url I can try? Also, is the approved version still
thrift-instant-r790732?
________________________________
From: Paul Yang [mailto:pyang@facebook.com]
Sent: Tuesday, June 22, 2010 6:06 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Hey Pradeep,
Yep, that's on the right path :) Since you made a change to
hive_metastore.thrift, you should run the thrift compiler by calling
'ant thriftif' within trunk/metastore. But before you can do that, the
thrift compiler has to be in your path. JVS has a nice write up on how
to do this:
http://wiki.apache.org/hadoop/Hive/HowToContribute#Generating_Code
Also, I've set up a JIRA for this issue at:
https://issues.apache.org/jira/browse/HIVE-1428
Can you post your patch there? It makes it a lot easier to track issues.
Thanks!
-Paul
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Tuesday, June 22, 2010 10:22 AM
To: hive-user@hadoop.apache.org
Subject: Re: alter table add partition error
Hi Paul,
I have attached a draft of a patch (incomplete). The problem I am
facing is that to make this change I seem to need to change the thrift
idl (hive_metastore.thrift). Looking at the code structure it seems like
the generate java files are checked in rather than generated through the
build (is this so that the build does not depend on the thrift
compiler?). I tried downloading and building thrift but hit issues in
the make install - wanted to check with you if the patch looks correct
so far or if I am on the wrong path. Also if the direction is right, I
need some guidance on generating the java files from the modified idl.
Thanks,
Pradeep
Paul Yang wrote:
Well, the section of code you are referring to is automatically
generated by the thrift compiler so we can't modify it. As the inability
to return null is a limitation of Thrift, we can work around that issue
by throwing an exception instead.
Check out this thread:
http://publists.facebook.com/pipermail/thrift/2007-September/000085.html
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Monday, June 21, 2010 5:46 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Hi Paul,
Yes I don't mind working on a patch though I have no knowledge of how
thrift code works. I was also confused by your suggestion for a fix. I
thought the fix would be in
ThriftHiveMetaStore.Client.recv_get_partition() - to first check
if(result.o1 != null) throw result.o1, then to just return
result.success and not throw the "unknown result" exception. I tried
something along these lines and the alter table worked. Am I missing
something?
Thanks,
Pradeep
________________________________
From: Paul Yang [mailto:pyang@facebook.com]
Sent: Monday, June 21, 2010 2:35 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Ah okay. So what's going on is that getPartition() is supposed to return
null if the specified partition doesn't exist. But because Thrift can't
handle null return values, we are seeing this exception. This isn't a
problem with a local metastore because Thrift isn't used.
One solution to this problem would be modify ObjectStore.getPartition()
to throw a NoSuchObjectException instead of returning null for
non-existent partitions. Then Hive.getPartition() can catch the
exception and return null to retain the original behavior.
Pradeep, are you interested in working on a patch for this problem?
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Monday, June 21, 2010 2:11 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
I tried debugging in code a little more. Here is what I found:
The code in ThriftHiveMetaStore eventually makes a call -
get_partition() passing the partition key values for the partition I am
trying to add using alter table. I assume this is to check that the
partition doesn't already exist.
I added a debug line in the following code:
public Partition recv_get_partition() throws MetaException, TException
{
TMessage msg = iprot_.readMessageBegin();
if (msg.type == TMessageType.EXCEPTION) {
TApplicationException x = TApplicationException.read(iprot_);
iprot_.readMessageEnd();
throw x;
}
get_partition_result result = new get_partition_result();
result.read(iprot_);
System.err.println("XXX: result:" + result);
iprot_.readMessageEnd();
if (result.isSetSuccess()) {
return result.success;
}
if (result.o1 != null) {
throw result.o1;
}
throw new
TApplicationException(TApplicationException.MISSING_RESULT,
"get_partition failed: unknown result");
}
I also put a debug statements in the read() method:
public void read(TProtocol iprot) throws TException {
TField field;
iprot.readStructBegin();
System.err.println("XXX: Reading TProtocol object:");
while (true)
{
field = iprot.readFieldBegin();
System.err.println("XXX: field just read:" + field);
if (field.type == TType.STOP) {
break;
I got
XXX: Reading TProtocol object:
XXX: field just read:<TField name:'' type:0 field-id:0>
XXX: result:get_partition_result(success:null, o1:null)
The field read in the thrift response message is of type "STOP" and with
id of type SUCCESS. This seems right since there are no existing
partitions. But the way the rest of the code handles this, results in
the exception.
Any pointers?
TIA,
Pradeep
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Friday, June 18, 2010 2:51 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Looks like the standalone script works fine against the existing
partition:
./ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name
default dummy datestamp=20100602/srcid=100/action=view/testid=10
Partition(parameters={'transient_lastDdlTime': '1276881277'},
tableName='dummy', createTime=1276881277, lastAccessTime=0,
values=['20100602', '100', 'view', '10'], dbName='default',
sd=StorageDescriptor(outputFormat='org.apache.hadoop.hive.ql.io.HiveIgno
reKeyTextOutputFormat', sortCols=[],
inputFormat='org.apache.hadoop.mapred.TextInputFormat',
cols=[FieldSchema(comment=None, type='string', name='partition_name'),
FieldSchema(comment=None, type='int', name='partition_id')],
compressed=False, bucketCols=[], numBuckets=-1, parameters={},
serdeInfo=SerDeInfo(serializationLib='org.apache.hadoop.hive.serde2.lazy
.LazySimpleSerDe', name=None, parameters={'serialization.format': '1'}),
location='hdfs://wilbur21.labs.corp.sp1.yahoo.com/user/pradeepk/dummy/20
100602/100/view/10'))
[pradeepk@chargesize:~/dev/howl/src/metastore/src/gen-py/hive_metastore]
However when I tried to add another partition with the hive cli using
thrift:
hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid
= '100',action='click',testid='10') location
'/user/pradeepk/dummy/20100602/100/click/10';"
10/06/18 14:49:13 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006181449_1158492515.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException:
get_partition failed: unknown result
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
tail -30 /tmp/pradeepk/hive.log
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
2010-06-18 14:49:14,124 ERROR exec.DDLTask
(SessionState.java:printError(277)) - FAILED: Error in metadata:
org.apache.thrift.TApplicationException: get_partition failed: unknown
result
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.thrift.TApplicationException: get_partition failed: unknown
result
at
org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
at
org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:255)
at
org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:169)
at
org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:633)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:506)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:384)
at
org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:267)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: org.apache.thrift.TApplicationException: get_partition
failed: unknown result
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get
_partition(ThriftHiveMetastore.java:931)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_part
ition(ThriftHiveMetastore.java:899)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMe
taStoreClient.java:500)
at
org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
... 15 more
2010-06-18 14:49:14,124 ERROR ql.Driver
(SessionState.java:printError(277)) - FAILED: Execution Error, return
code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
________________________________
From: Paul Yang [mailto:pyang@facebook.com]
Sent: Friday, June 18, 2010 2:19 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Looks like the thrift python libraries aren't in your path. run:
export PYTHONPATH=<path-to-trunk>/build/dist/lib/py/
before trying the remote command
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Friday, June 18, 2010 1:38 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Sorry, don't know python - so couldn't make sense out of the following
error when I run the command you suggested:
[src/metastore/src/gen-py/hive_metastore]./ThriftHiveMetastore-remote -h
localhost:9080 get_partition_by_name default dummy
datestamp=20100602/srcid=100/action=view/testid=10
Traceback (most recent call last):
File "./ThriftHiveMetastore-remote", line 11, in ?
from thrift.transport import TTransport
ImportError: No module named thrift.transport
Thanks,
Pradeep
________________________________
From: Paul Yang [mailto:pyang@facebook.com]
Sent: Friday, June 18, 2010 12:10 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Hi Pradeep,
In trunk/metastore/src/gen-py/hive_metastore/, there is a script called
ThriftHiveMetastore-remote that can be used to test out the thrift
server independent of the CLI. As a quick test to narrow down the
problem, after the partition is created, can you try running
ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name
default dummy datestamp=20100602/srcid=100/action=view/testid=10
Cheers,
Paul
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Friday, June 18, 2010 11:26 AM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Hi Ning,
I have pasted below [1] the values for HADOOP_CLASSPATH and CLASSPATH
from an "env" command printed just before the following line in
execHiveCmd.sh
exec $HADOOP jar ${HIVE_LIB}/hive-cli-*.jar $CLASS $HIVE_OPTS "$@"
There is only one occurrence of libthrift.jar whose cksum matches the
one from lib/libthrift.jar from a hive checkout build.
The hive_config_without_thrift is a conf directory containing a
hive-site.xml with jdo settings to connect to the database directly. By
default I have hive set up to use a hive-site.xml [2] which connects to
a thrift server running on localhost.
Any pointers appreciated!
Thanks,
Pradeep
[1]
HADOOP_CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_c
lient/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.
1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home
/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_cl
ient/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/c
ommons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.j
ar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/lib
exec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_
client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucl
eus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus
-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1
.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home
/y/libexec/hive_client/lib/derby.jar:/home/y/libex
ec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbas
e-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.ja
r:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hiv
e_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_
contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/
libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_clie
nt/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.
0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/l
ibexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/
lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.
6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/
y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/l
ib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libe
xec/hive_client/lib/libfb303.jar:/home/y/libexec/hi
ve_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15
.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_cli
ent/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtem
plate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y
/libexec/hive_client/lib/zookeeper-3.2.2.jar:
CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/l
ib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/
home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libe
xec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/li
b/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-
dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/hom
e/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hi
ve_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/
lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-con
nectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1
.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:
/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libe
xec/hive_client/lib/derby.jar:/home/y/libex
ec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbas
e-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.ja
r:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hiv
e_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_
contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/
libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_clie
nt/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.
0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/l
ibexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/
lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.
6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/
y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/l
ib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libe
xec/hive_client/lib/libfb303.jar:/home/y/libexec/hi
ve_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15
.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_cli
ent/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtem
plate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y
/libexec/hive_client/lib/zookeeper-3.2.2.jar:
[2]
cat /home/y/libexec/hive_client/conf/hive-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<!-- Hive Configuration can either be stored in this file or in the
hadoop configuration files -->
<!-- that are implied by Hadoop setup variables.
-->
<!-- Aside from Hadoop setup variables - this file is provided as a
convenience so that Hive -->
<!-- users do not have to edit hadoop configuration files (that may be
managed as a centralized -->
<!-- resource).
-->
<!-- Hive Execution Parameters -->
<property>
<name>hive.metastore.uris</name>
<value>thrift://localhost:9080</value>
<description>host and port for the thrift metastore
server</description>
</property>
<property>
<name>hive.metastore.local</name>
<value>false</value>
<description>controls whether to connect to remove metastore server or
open a new metastore server in Hive Client JVM</description>
</property>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/user/pradeepk/hive/warehouse</value>
<description>location of default database for the
warehouse</description>
</property>
</configuration>
________________________________
From: Ning Zhang [mailto:nzhang@facebook.com]
Sent: Friday, June 18, 2010 10:50 AM
To: hive-user@hadoop.apache.org
Subject: Re: alter table add partition error
Pradeep,
I ran the commands you provided and it succeeded with the expected
behavior.
One possibility is that there are multiple versions of libthrift.jar in
your CLASSPATH (hadoop & hive). Can you check in the Hadoop & Hive
CLASSPATH so that no other libthrift.jar is there? What is in
hive_config_without_thrift?
Thanks,
Ning
On Jun 18, 2010, at 10:19 AM, Pradeep Kamath wrote:
I think there are two separate issues here - I want to open a jira for
the first one since I am now able to reproduce it even with text format
with builtin Serdes. Essentially this is a bug in the thrift code (not
sure if it is in the client or server) since the same alter table
statement works fine when the hive client does not use thrift. Here are
the details:
cat create_dummy.sql
CREATE external TABLE if not exists dummy (
partition_name string
,partition_id int
)
PARTITIONED BY ( datestamp string, srcid string, action string, testid
string )
row format delimited
stored as textfile
location '/user/pradeepk/dummy';
hive -f create_dummy.sql
10/06/18 10:13:36 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006181013_184583537.txt
OK
Time taken: 0.627 seconds
hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid
= '100',action='view',testid='10') location
'/user/pradeepk/dummy/20100602/100/view/10';"
10/06/18 10:14:11 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_700722546.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException:
get_partition failed: unknown result
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
hive --config hive_conf_without_thrift -e "ALTER TABLE dummy add
partition(datestamp = '20100602', srcid =
'100',action='view',testid='10') location
'/user/pradeepk/dummy/20100602/100/view/10';"
10/06/18 10:14:31 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_598649843.txt
OK
Time taken: 5.849 seconds
Is there some thrift setting I am missing or is this a bug? - If it is
the latter, I can open a jira with the above details.
Thanks,
Pradeep
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Thursday, June 17, 2010 1:25 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Here are the create table and alter table statements:
CREATE external TABLE if not exists mytable (
bc string
,src_spaceid string
,srcpvid string
,dstpvid string
,dst_spaceid string
,page_params map<string, string>
,clickinfo map<string, string>
,viewinfo array<map<string, string>>
)
PARTITIONED BY ( datestamp string, srcid string, action string, testid
string )
row format serde 'com.yahoo.mySerde'
stored as inputformat 'org.apache.hadoop.mapred.SequenceFileInputFormat'
outputformat 'org.apache.hadoop.mapred.SequenceFileOutputFormat'
location '/user/pradeepk/mytable';
hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add
partition(datestamp = '20091101', srcid =
'19174',action='click',testid='NOTESTID') location
'/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"
I get the following error:
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException:
get_partition failed: unknown result
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
If I don't use thrift and use a hive-site.xml to directly talk to the
db, the alter table seems to succeed:
hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "ALTER
TABLE mytable add partition(datestamp = '20091101', srcid =
'19174',action='click',testid='NOTESTID') location
'/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"
However I get errors when I try to run a query:
[pradeepk@chargesize:~/dev]hive --auxpath ult-serde.jar --config
hive_conf_without_thrift -e "select src_spaceid from
ult_search_austria_ult where datestamp='20091101';"
10/06/17 13:22:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006171322_1913647383.txt
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
java.lang.IllegalArgumentException: Can not create a Path from an empty
string
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)
at org.apache.hadoop.fs.Path.<init>(Path.java:90)
at org.apache.hadoop.fs.Path.<init>(Path.java:50)
at
org.apache.hadoop.mapred.JobClient.copyRemoteFiles(JobClient.java:523)
at
org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient
.java:603)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)
at
org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
at
org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:684)
at
org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
Any help is much appreciated.
Pradeep
________________________________
From: Ashish Thusoo [mailto:athusoo@facebook.com]
Sent: Thursday, June 17, 2010 11:15 AM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
hmm... Can you send the exact command and also the create table command
for this table.
Ashish
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Thursday, June 17, 2010 9:09 AM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Sorry - that was a cut-paste error - I don't have the action part - so I
am specifying key-value pairs. Since what I am trying to do seems like a
basic operation, I am wondering if it's something to do with my Serde -
unfortunately the error I see gives me no clue of what could be wrong -
any help would be greatly appreciated!
Thanks,
Pradeep
________________________________
From: yq he [mailto:hhh.heyq@gmail.com]
Sent: Wednesday, June 16, 2010 5:54 PM
To: hive-user@hadoop.apache.org
Subject: Re: alter table add partition error
Hi Pradeep,
partition definition need to be key-value pairs. partition key `action`
seems missed the value.
Thanks
Yongqiang
On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath <pr...@yahoo-inc.com>
wrote:
Hi,
I am trying to create an external table against already existing
data in sequencefile format. However I have written a custom Serde to
interpret the data. I am able to create the table fine but get the
exception shown in the session output below when I try to add partition
- any help would be greatly appreciated.
Thanks,
Pradeep
== session output ===
[pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER
TABLE mytable add partition(datestamp = '20091101', srcid = '10',action)
location '/user/pradeepk/mytable/20091101/10';"
10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException:
get_partition failed: unknown result
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
[pradeepk@chargesize:~/dev/howl]
== session output ===
/tmp/pradeepk/hive.log has:
2010-06-16 17:09:00,841 ERROR exec.DDLTask
(SessionState.java:printError(269)) - FAILED: Error in metadata:
org.apache.thrift.TApplicationException: get_partition failed: unknown
result
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.thrift.TApplicationException: get_partition failed: unknown
result
at
org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
at
org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
at
org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: org.apache.thrift.TApplicationException: get_partition
failed: unknown result
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get
_partition(ThriftHiveMetastore.java:931)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_part
ition(ThriftHiveMetastore.java:899)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMe
taStoreClient.java:500)
at
org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
... 15 more
The thrift server messages are:
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table :
db=default tbl=mytable
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store
with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize
called
10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition :
db=default tbl=mytable
RE: alter table add partition error
Posted by Pradeep Kamath <pr...@yahoo-inc.com>.
In the second step of
http://wiki.apache.org/hadoop/Hive/HowToContribute#Generating_Code
"Use the approved version of thrift. This is currently
thrift-instant-r790732, which you can obtain from
http://instant.thrift-rpc.org"
Looks like that site is down since it is not responding to requests - is
there an alternate url I can try? Also, is the approved version still
thrift-instant-r790732?
________________________________
From: Paul Yang [mailto:pyang@facebook.com]
Sent: Tuesday, June 22, 2010 6:06 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Hey Pradeep,
Yep, that's on the right path :) Since you made a change to
hive_metastore.thrift, you should run the thrift compiler by calling
'ant thriftif' within trunk/metastore. But before you can do that, the
thrift compiler has to be in your path. JVS has a nice write up on how
to do this:
http://wiki.apache.org/hadoop/Hive/HowToContribute#Generating_Code
Also, I've set up a JIRA for this issue at:
https://issues.apache.org/jira/browse/HIVE-1428
Can you post your patch there? It makes it a lot easier to track issues.
Thanks!
-Paul
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Tuesday, June 22, 2010 10:22 AM
To: hive-user@hadoop.apache.org
Subject: Re: alter table add partition error
Hi Paul,
I have attached a draft of a patch (incomplete). The problem I am
facing is that to make this change I seem to need to change the thrift
idl (hive_metastore.thrift). Looking at the code structure it seems like
the generate java files are checked in rather than generated through the
build (is this so that the build does not depend on the thrift
compiler?). I tried downloading and building thrift but hit issues in
the make install - wanted to check with you if the patch looks correct
so far or if I am on the wrong path. Also if the direction is right, I
need some guidance on generating the java files from the modified idl.
Thanks,
Pradeep
Paul Yang wrote:
Well, the section of code you are referring to is automatically
generated by the thrift compiler so we can't modify it. As the inability
to return null is a limitation of Thrift, we can work around that issue
by throwing an exception instead.
Check out this thread:
http://publists.facebook.com/pipermail/thrift/2007-September/000085.html
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Monday, June 21, 2010 5:46 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Hi Paul,
Yes I don't mind working on a patch though I have no knowledge of how
thrift code works. I was also confused by your suggestion for a fix. I
thought the fix would be in
ThriftHiveMetaStore.Client.recv_get_partition() - to first check
if(result.o1 != null) throw result.o1, then to just return
result.success and not throw the "unknown result" exception. I tried
something along these lines and the alter table worked. Am I missing
something?
Thanks,
Pradeep
________________________________
From: Paul Yang [mailto:pyang@facebook.com]
Sent: Monday, June 21, 2010 2:35 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Ah okay. So what's going on is that getPartition() is supposed to return
null if the specified partition doesn't exist. But because Thrift can't
handle null return values, we are seeing this exception. This isn't a
problem with a local metastore because Thrift isn't used.
One solution to this problem would be modify ObjectStore.getPartition()
to throw a NoSuchObjectException instead of returning null for
non-existent partitions. Then Hive.getPartition() can catch the
exception and return null to retain the original behavior.
Pradeep, are you interested in working on a patch for this problem?
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Monday, June 21, 2010 2:11 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
I tried debugging in code a little more. Here is what I found:
The code in ThriftHiveMetaStore eventually makes a call -
get_partition() passing the partition key values for the partition I am
trying to add using alter table. I assume this is to check that the
partition doesn't already exist.
I added a debug line in the following code:
public Partition recv_get_partition() throws MetaException, TException
{
TMessage msg = iprot_.readMessageBegin();
if (msg.type == TMessageType.EXCEPTION) {
TApplicationException x = TApplicationException.read(iprot_);
iprot_.readMessageEnd();
throw x;
}
get_partition_result result = new get_partition_result();
result.read(iprot_);
System.err.println("XXX: result:" + result);
iprot_.readMessageEnd();
if (result.isSetSuccess()) {
return result.success;
}
if (result.o1 != null) {
throw result.o1;
}
throw new
TApplicationException(TApplicationException.MISSING_RESULT,
"get_partition failed: unknown result");
}
I also put a debug statements in the read() method:
public void read(TProtocol iprot) throws TException {
TField field;
iprot.readStructBegin();
System.err.println("XXX: Reading TProtocol object:");
while (true)
{
field = iprot.readFieldBegin();
System.err.println("XXX: field just read:" + field);
if (field.type == TType.STOP) {
break;
I got
XXX: Reading TProtocol object:
XXX: field just read:<TField name:'' type:0 field-id:0>
XXX: result:get_partition_result(success:null, o1:null)
The field read in the thrift response message is of type "STOP" and with
id of type SUCCESS. This seems right since there are no existing
partitions. But the way the rest of the code handles this, results in
the exception.
Any pointers?
TIA,
Pradeep
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Friday, June 18, 2010 2:51 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Looks like the standalone script works fine against the existing
partition:
./ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name
default dummy datestamp=20100602/srcid=100/action=view/testid=10
Partition(parameters={'transient_lastDdlTime': '1276881277'},
tableName='dummy', createTime=1276881277, lastAccessTime=0,
values=['20100602', '100', 'view', '10'], dbName='default',
sd=StorageDescriptor(outputFormat='org.apache.hadoop.hive.ql.io.HiveIgno
reKeyTextOutputFormat', sortCols=[],
inputFormat='org.apache.hadoop.mapred.TextInputFormat',
cols=[FieldSchema(comment=None, type='string', name='partition_name'),
FieldSchema(comment=None, type='int', name='partition_id')],
compressed=False, bucketCols=[], numBuckets=-1, parameters={},
serdeInfo=SerDeInfo(serializationLib='org.apache.hadoop.hive.serde2.lazy
.LazySimpleSerDe', name=None, parameters={'serialization.format': '1'}),
location='hdfs://wilbur21.labs.corp.sp1.yahoo.com/user/pradeepk/dummy/20
100602/100/view/10'))
[pradeepk@chargesize:~/dev/howl/src/metastore/src/gen-py/hive_metastore]
However when I tried to add another partition with the hive cli using
thrift:
hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid
= '100',action='click',testid='10') location
'/user/pradeepk/dummy/20100602/100/click/10';"
10/06/18 14:49:13 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006181449_1158492515.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException:
get_partition failed: unknown result
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
tail -30 /tmp/pradeepk/hive.log
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
2010-06-18 14:49:14,124 ERROR exec.DDLTask
(SessionState.java:printError(277)) - FAILED: Error in metadata:
org.apache.thrift.TApplicationException: get_partition failed: unknown
result
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.thrift.TApplicationException: get_partition failed: unknown
result
at
org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
at
org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:255)
at
org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:169)
at
org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:633)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:506)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:384)
at
org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:267)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: org.apache.thrift.TApplicationException: get_partition
failed: unknown result
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get
_partition(ThriftHiveMetastore.java:931)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_part
ition(ThriftHiveMetastore.java:899)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMe
taStoreClient.java:500)
at
org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
... 15 more
2010-06-18 14:49:14,124 ERROR ql.Driver
(SessionState.java:printError(277)) - FAILED: Execution Error, return
code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
________________________________
From: Paul Yang [mailto:pyang@facebook.com]
Sent: Friday, June 18, 2010 2:19 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Looks like the thrift python libraries aren't in your path. run:
export PYTHONPATH=<path-to-trunk>/build/dist/lib/py/
before trying the remote command
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Friday, June 18, 2010 1:38 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Sorry, don't know python - so couldn't make sense out of the following
error when I run the command you suggested:
[src/metastore/src/gen-py/hive_metastore]./ThriftHiveMetastore-remote -h
localhost:9080 get_partition_by_name default dummy
datestamp=20100602/srcid=100/action=view/testid=10
Traceback (most recent call last):
File "./ThriftHiveMetastore-remote", line 11, in ?
from thrift.transport import TTransport
ImportError: No module named thrift.transport
Thanks,
Pradeep
________________________________
From: Paul Yang [mailto:pyang@facebook.com]
Sent: Friday, June 18, 2010 12:10 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Hi Pradeep,
In trunk/metastore/src/gen-py/hive_metastore/, there is a script called
ThriftHiveMetastore-remote that can be used to test out the thrift
server independent of the CLI. As a quick test to narrow down the
problem, after the partition is created, can you try running
ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name
default dummy datestamp=20100602/srcid=100/action=view/testid=10
Cheers,
Paul
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Friday, June 18, 2010 11:26 AM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Hi Ning,
I have pasted below [1] the values for HADOOP_CLASSPATH and CLASSPATH
from an "env" command printed just before the following line in
execHiveCmd.sh
exec $HADOOP jar ${HIVE_LIB}/hive-cli-*.jar $CLASS $HIVE_OPTS "$@"
There is only one occurrence of libthrift.jar whose cksum matches the
one from lib/libthrift.jar from a hive checkout build.
The hive_config_without_thrift is a conf directory containing a
hive-site.xml with jdo settings to connect to the database directly. By
default I have hive set up to use a hive-site.xml [2] which connects to
a thrift server running on localhost.
Any pointers appreciated!
Thanks,
Pradeep
[1]
HADOOP_CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_c
lient/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.
1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home
/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_cl
ient/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/c
ommons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.j
ar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/lib
exec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_
client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucl
eus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus
-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1
.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home
/y/libexec/hive_client/lib/derby.jar:/home/y/libex
ec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbas
e-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.ja
r:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hiv
e_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_
contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/
libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_clie
nt/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.
0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/l
ibexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/
lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.
6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/
y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/l
ib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libe
xec/hive_client/lib/libfb303.jar:/home/y/libexec/hi
ve_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15
.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_cli
ent/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtem
plate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y
/libexec/hive_client/lib/zookeeper-3.2.2.jar:
CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/l
ib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/
home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libe
xec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/li
b/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-
dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/hom
e/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hi
ve_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/
lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-con
nectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1
.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:
/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libe
xec/hive_client/lib/derby.jar:/home/y/libex
ec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbas
e-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.ja
r:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hiv
e_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_
contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/
libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_clie
nt/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.
0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/l
ibexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/
lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.
6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/
y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/l
ib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libe
xec/hive_client/lib/libfb303.jar:/home/y/libexec/hi
ve_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15
.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_cli
ent/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtem
plate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y
/libexec/hive_client/lib/zookeeper-3.2.2.jar:
[2]
cat /home/y/libexec/hive_client/conf/hive-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<!-- Hive Configuration can either be stored in this file or in the
hadoop configuration files -->
<!-- that are implied by Hadoop setup variables.
-->
<!-- Aside from Hadoop setup variables - this file is provided as a
convenience so that Hive -->
<!-- users do not have to edit hadoop configuration files (that may be
managed as a centralized -->
<!-- resource).
-->
<!-- Hive Execution Parameters -->
<property>
<name>hive.metastore.uris</name>
<value>thrift://localhost:9080</value>
<description>host and port for the thrift metastore
server</description>
</property>
<property>
<name>hive.metastore.local</name>
<value>false</value>
<description>controls whether to connect to remove metastore server or
open a new metastore server in Hive Client JVM</description>
</property>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/user/pradeepk/hive/warehouse</value>
<description>location of default database for the
warehouse</description>
</property>
</configuration>
________________________________
From: Ning Zhang [mailto:nzhang@facebook.com]
Sent: Friday, June 18, 2010 10:50 AM
To: hive-user@hadoop.apache.org
Subject: Re: alter table add partition error
Pradeep,
I ran the commands you provided and it succeeded with the expected
behavior.
One possibility is that there are multiple versions of libthrift.jar in
your CLASSPATH (hadoop & hive). Can you check in the Hadoop & Hive
CLASSPATH so that no other libthrift.jar is there? What is in
hive_config_without_thrift?
Thanks,
Ning
On Jun 18, 2010, at 10:19 AM, Pradeep Kamath wrote:
I think there are two separate issues here - I want to open a jira for
the first one since I am now able to reproduce it even with text format
with builtin Serdes. Essentially this is a bug in the thrift code (not
sure if it is in the client or server) since the same alter table
statement works fine when the hive client does not use thrift. Here are
the details:
cat create_dummy.sql
CREATE external TABLE if not exists dummy (
partition_name string
,partition_id int
)
PARTITIONED BY ( datestamp string, srcid string, action string, testid
string )
row format delimited
stored as textfile
location '/user/pradeepk/dummy';
hive -f create_dummy.sql
10/06/18 10:13:36 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006181013_184583537.txt
OK
Time taken: 0.627 seconds
hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid
= '100',action='view',testid='10') location
'/user/pradeepk/dummy/20100602/100/view/10';"
10/06/18 10:14:11 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_700722546.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException:
get_partition failed: unknown result
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
hive --config hive_conf_without_thrift -e "ALTER TABLE dummy add
partition(datestamp = '20100602', srcid =
'100',action='view',testid='10') location
'/user/pradeepk/dummy/20100602/100/view/10';"
10/06/18 10:14:31 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_598649843.txt
OK
Time taken: 5.849 seconds
Is there some thrift setting I am missing or is this a bug? - If it is
the latter, I can open a jira with the above details.
Thanks,
Pradeep
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Thursday, June 17, 2010 1:25 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Here are the create table and alter table statements:
CREATE external TABLE if not exists mytable (
bc string
,src_spaceid string
,srcpvid string
,dstpvid string
,dst_spaceid string
,page_params map<string, string>
,clickinfo map<string, string>
,viewinfo array<map<string, string>>
)
PARTITIONED BY ( datestamp string, srcid string, action string, testid
string )
row format serde 'com.yahoo.mySerde'
stored as inputformat 'org.apache.hadoop.mapred.SequenceFileInputFormat'
outputformat 'org.apache.hadoop.mapred.SequenceFileOutputFormat'
location '/user/pradeepk/mytable';
hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add
partition(datestamp = '20091101', srcid =
'19174',action='click',testid='NOTESTID') location
'/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"
I get the following error:
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException:
get_partition failed: unknown result
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
If I don't use thrift and use a hive-site.xml to directly talk to the
db, the alter table seems to succeed:
hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "ALTER
TABLE mytable add partition(datestamp = '20091101', srcid =
'19174',action='click',testid='NOTESTID') location
'/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"
However I get errors when I try to run a query:
[pradeepk@chargesize:~/dev]hive --auxpath ult-serde.jar --config
hive_conf_without_thrift -e "select src_spaceid from
ult_search_austria_ult where datestamp='20091101';"
10/06/17 13:22:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006171322_1913647383.txt
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
java.lang.IllegalArgumentException: Can not create a Path from an empty
string
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)
at org.apache.hadoop.fs.Path.<init>(Path.java:90)
at org.apache.hadoop.fs.Path.<init>(Path.java:50)
at
org.apache.hadoop.mapred.JobClient.copyRemoteFiles(JobClient.java:523)
at
org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient
.java:603)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)
at
org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
at
org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:684)
at
org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
Any help is much appreciated.
Pradeep
________________________________
From: Ashish Thusoo [mailto:athusoo@facebook.com]
Sent: Thursday, June 17, 2010 11:15 AM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
hmm... Can you send the exact command and also the create table command
for this table.
Ashish
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Thursday, June 17, 2010 9:09 AM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Sorry - that was a cut-paste error - I don't have the action part - so I
am specifying key-value pairs. Since what I am trying to do seems like a
basic operation, I am wondering if it's something to do with my Serde -
unfortunately the error I see gives me no clue of what could be wrong -
any help would be greatly appreciated!
Thanks,
Pradeep
________________________________
From: yq he [mailto:hhh.heyq@gmail.com]
Sent: Wednesday, June 16, 2010 5:54 PM
To: hive-user@hadoop.apache.org
Subject: Re: alter table add partition error
Hi Pradeep,
partition definition need to be key-value pairs. partition key `action`
seems missed the value.
Thanks
Yongqiang
On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath <pr...@yahoo-inc.com>
wrote:
Hi,
I am trying to create an external table against already existing
data in sequencefile format. However I have written a custom Serde to
interpret the data. I am able to create the table fine but get the
exception shown in the session output below when I try to add partition
- any help would be greatly appreciated.
Thanks,
Pradeep
== session output ===
[pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER
TABLE mytable add partition(datestamp = '20091101', srcid = '10',action)
location '/user/pradeepk/mytable/20091101/10';"
10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException:
get_partition failed: unknown result
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
[pradeepk@chargesize:~/dev/howl]
== session output ===
/tmp/pradeepk/hive.log has:
2010-06-16 17:09:00,841 ERROR exec.DDLTask
(SessionState.java:printError(269)) - FAILED: Error in metadata:
org.apache.thrift.TApplicationException: get_partition failed: unknown
result
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.thrift.TApplicationException: get_partition failed: unknown
result
at
org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
at
org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
at
org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: org.apache.thrift.TApplicationException: get_partition
failed: unknown result
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get
_partition(ThriftHiveMetastore.java:931)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_part
ition(ThriftHiveMetastore.java:899)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMe
taStoreClient.java:500)
at
org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
... 15 more
The thrift server messages are:
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table :
db=default tbl=mytable
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store
with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize
called
10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition :
db=default tbl=mytable
RE: alter table add partition error
Posted by Paul Yang <py...@facebook.com>.
Hey Pradeep,
Yep, that's on the right path :) Since you made a change to hive_metastore.thrift, you should run the thrift compiler by calling 'ant thriftif' within trunk/metastore. But before you can do that, the thrift compiler has to be in your path. JVS has a nice write up on how to do this:
http://wiki.apache.org/hadoop/Hive/HowToContribute#Generating_Code
Also, I've set up a JIRA for this issue at:
https://issues.apache.org/jira/browse/HIVE-1428
Can you post your patch there? It makes it a lot easier to track issues. Thanks!
-Paul
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Tuesday, June 22, 2010 10:22 AM
To: hive-user@hadoop.apache.org
Subject: Re: alter table add partition error
Hi Paul,
I have attached a draft of a patch (incomplete). The problem I am facing is that to make this change I seem to need to change the thrift idl (hive_metastore.thrift). Looking at the code structure it seems like the generate java files are checked in rather than generated through the build (is this so that the build does not depend on the thrift compiler?). I tried downloading and building thrift but hit issues in the make install - wanted to check with you if the patch looks correct so far or if I am on the wrong path. Also if the direction is right, I need some guidance on generating the java files from the modified idl.
Thanks,
Pradeep
Paul Yang wrote:
Well, the section of code you are referring to is automatically generated by the thrift compiler so we can't modify it. As the inability to return null is a limitation of Thrift, we can work around that issue by throwing an exception instead.
Check out this thread:
http://publists.facebook.com/pipermail/thrift/2007-September/000085.html
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Monday, June 21, 2010 5:46 PM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: alter table add partition error
Hi Paul,
Yes I don't mind working on a patch though I have no knowledge of how thrift code works. I was also confused by your suggestion for a fix. I thought the fix would be in ThriftHiveMetaStore.Client.recv_get_partition() - to first check if(result.o1 != null) throw result.o1, then to just return result.success and not throw the "unknown result" exception. I tried something along these lines and the alter table worked. Am I missing something?
Thanks,
Pradeep
________________________________
From: Paul Yang [mailto:pyang@facebook.com]
Sent: Monday, June 21, 2010 2:35 PM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: alter table add partition error
Ah okay. So what's going on is that getPartition() is supposed to return null if the specified partition doesn't exist. But because Thrift can't handle null return values, we are seeing this exception. This isn't a problem with a local metastore because Thrift isn't used.
One solution to this problem would be modify ObjectStore.getPartition() to throw a NoSuchObjectException instead of returning null for non-existent partitions. Then Hive.getPartition() can catch the exception and return null to retain the original behavior.
Pradeep, are you interested in working on a patch for this problem?
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Monday, June 21, 2010 2:11 PM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: alter table add partition error
I tried debugging in code a little more. Here is what I found:
The code in ThriftHiveMetaStore eventually makes a call - get_partition() passing the partition key values for the partition I am trying to add using alter table. I assume this is to check that the partition doesn't already exist.
I added a debug line in the following code:
public Partition recv_get_partition() throws MetaException, TException
{
TMessage msg = iprot_.readMessageBegin();
if (msg.type == TMessageType.EXCEPTION) {
TApplicationException x = TApplicationException.read(iprot_);
iprot_.readMessageEnd();
throw x;
}
get_partition_result result = new get_partition_result();
result.read(iprot_);
System.err.println("XXX: result:" + result);
iprot_.readMessageEnd();
if (result.isSetSuccess()) {
return result.success;
}
if (result.o1 != null) {
throw result.o1;
}
throw new TApplicationException(TApplicationException.MISSING_RESULT, "get_partition failed: unknown result");
}
I also put a debug statements in the read() method:
public void read(TProtocol iprot) throws TException {
TField field;
iprot.readStructBegin();
System.err.println("XXX: Reading TProtocol object:");
while (true)
{
field = iprot.readFieldBegin();
System.err.println("XXX: field just read:" + field);
if (field.type == TType.STOP) {
break;
I got
XXX: Reading TProtocol object:
XXX: field just read:<TField name:'' type:0 field-id:0>
XXX: result:get_partition_result(success:null, o1:null)
The field read in the thrift response message is of type "STOP" and with id of type SUCCESS. This seems right since there are no existing partitions. But the way the rest of the code handles this, results in the exception.
Any pointers?
TIA,
Pradeep
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Friday, June 18, 2010 2:51 PM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: alter table add partition error
Looks like the standalone script works fine against the existing partition:
./ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name default dummy datestamp=20100602/srcid=100/action=view/testid=10
Partition(parameters={'transient_lastDdlTime': '1276881277'}, tableName='dummy', createTime=1276881277, lastAccessTime=0, values=['20100602', '100', 'view', '10'], dbName='default', sd=StorageDescriptor(outputFormat='org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat', sortCols=[], inputFormat='org.apache.hadoop.mapred.TextInputFormat', cols=[FieldSchema(comment=None, type='string', name='partition_name'), FieldSchema(comment=None, type='int', name='partition_id')], compressed=False, bucketCols=[], numBuckets=-1, parameters={}, serdeInfo=SerDeInfo(serializationLib='org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe', name=None, parameters={'serialization.format': '1'}), location='hdfs://wilbur21.labs.corp.sp1.yahoo.com/user/pradeepk/dummy/20100602/100/view/10'))
[pradeepk@chargesize:~/dev/howl/src/metastore/src/gen-py/hive_metastore]
However when I tried to add another partition with the hive cli using thrift:
hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid = '100',action='click',testid='10') location '/user/pradeepk/dummy/20100602/100/click/10';"
10/06/18 14:49:13 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181449_1158492515.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
tail -30 /tmp/pradeepk/hive.log
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
2010-06-18 14:49:14,124 ERROR exec.DDLTask (SessionState.java:printError(277)) - FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.TApplicationException: get_partition failed: unknown result
at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
at org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:255)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:169)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:633)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:506)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:384)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:267)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: org.apache.thrift.TApplicationException: get_partition failed: unknown result
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition(ThriftHiveMetastore.java:931)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition(ThriftHiveMetastore.java:899)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:500)
at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
... 15 more
2010-06-18 14:49:14,124 ERROR ql.Driver (SessionState.java:printError(277)) - FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
________________________________
From: Paul Yang [mailto:pyang@facebook.com]
Sent: Friday, June 18, 2010 2:19 PM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: alter table add partition error
Looks like the thrift python libraries aren't in your path. run:
export PYTHONPATH=<path-to-trunk>/build/dist/lib/py/
before trying the remote command
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Friday, June 18, 2010 1:38 PM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: alter table add partition error
Sorry, don't know python - so couldn't make sense out of the following error when I run the command you suggested:
[src/metastore/src/gen-py/hive_metastore]./ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name default dummy datestamp=20100602/srcid=100/action=view/testid=10
Traceback (most recent call last):
File "./ThriftHiveMetastore-remote", line 11, in ?
from thrift.transport import TTransport
ImportError: No module named thrift.transport
Thanks,
Pradeep
________________________________
From: Paul Yang [mailto:pyang@facebook.com]
Sent: Friday, June 18, 2010 12:10 PM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: alter table add partition error
Hi Pradeep,
In trunk/metastore/src/gen-py/hive_metastore/, there is a script called ThriftHiveMetastore-remote that can be used to test out the thrift server independent of the CLI. As a quick test to narrow down the problem, after the partition is created, can you try running
ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name default dummy datestamp=20100602/srcid=100/action=view/testid=10
Cheers,
Paul
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Friday, June 18, 2010 11:26 AM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: alter table add partition error
Hi Ning,
I have pasted below [1] the values for HADOOP_CLASSPATH and CLASSPATH from an "env" command printed just before the following line in execHiveCmd.sh
exec $HADOOP jar ${HIVE_LIB}/hive-cli-*.jar $CLASS $HIVE_OPTS "$@"
There is only one occurrence of libthrift.jar whose cksum matches the one from lib/libthrift.jar from a hive checkout build.
The hive_config_without_thrift is a conf directory containing a hive-site.xml with jdo settings to connect to the database directly. By default I have hive set up to use a hive-site.xml [2] which connects to a thrift server running on localhost.
Any pointers appreciated!
Thanks,
Pradeep
[1]
HADOOP_CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libexec/hive_client/lib/derby.jar:/home/y/libex ec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.jar:/home/y/libexec/hi ve_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:
CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libexec/hive_client/lib/derby.jar:/home/y/libex ec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.jar:/home/y/libexec/hi ve_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:
[2]
cat /home/y/libexec/hive_client/conf/hive-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<!-- Hive Configuration can either be stored in this file or in the hadoop configuration files -->
<!-- that are implied by Hadoop setup variables. -->
<!-- Aside from Hadoop setup variables - this file is provided as a convenience so that Hive -->
<!-- users do not have to edit hadoop configuration files (that may be managed as a centralized -->
<!-- resource). -->
<!-- Hive Execution Parameters -->
<property>
<name>hive.metastore.uris</name>
<value>thrift://localhost:9080</value>
<description>host and port for the thrift metastore server</description>
</property>
<property>
<name>hive.metastore.local</name>
<value>false</value>
<description>controls whether to connect to remove metastore server or open a new metastore server in Hive Client JVM</description>
</property>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/user/pradeepk/hive/warehouse</value>
<description>location of default database for the warehouse</description>
</property>
</configuration>
________________________________
From: Ning Zhang [mailto:nzhang@facebook.com]
Sent: Friday, June 18, 2010 10:50 AM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: alter table add partition error
Pradeep,
I ran the commands you provided and it succeeded with the expected behavior.
One possibility is that there are multiple versions of libthrift.jar in your CLASSPATH (hadoop & hive). Can you check in the Hadoop & Hive CLASSPATH so that no other libthrift.jar is there? What is in hive_config_without_thrift?
Thanks,
Ning
On Jun 18, 2010, at 10:19 AM, Pradeep Kamath wrote:
I think there are two separate issues here - I want to open a jira for the first one since I am now able to reproduce it even with text format with builtin Serdes. Essentially this is a bug in the thrift code (not sure if it is in the client or server) since the same alter table statement works fine when the hive client does not use thrift. Here are the details:
cat create_dummy.sql
CREATE external TABLE if not exists dummy (
partition_name string
,partition_id int
)
PARTITIONED BY ( datestamp string, srcid string, action string, testid string )
row format delimited
stored as textfile
location '/user/pradeepk/dummy';
hive -f create_dummy.sql
10/06/18 10:13:36 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181013_184583537.txt
OK
Time taken: 0.627 seconds
hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid = '100',action='view',testid='10') location '/user/pradeepk/dummy/20100602/100/view/10';"
10/06/18 10:14:11 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_700722546.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
hive --config hive_conf_without_thrift -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid = '100',action='view',testid='10') location '/user/pradeepk/dummy/20100602/100/view/10';"
10/06/18 10:14:31 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_598649843.txt
OK
Time taken: 5.849 seconds
Is there some thrift setting I am missing or is this a bug? - If it is the latter, I can open a jira with the above details.
Thanks,
Pradeep
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Thursday, June 17, 2010 1:25 PM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: alter table add partition error
Here are the create table and alter table statements:
CREATE external TABLE if not exists mytable (
bc string
,src_spaceid string
,srcpvid string
,dstpvid string
,dst_spaceid string
,page_params map<string, string>
,clickinfo map<string, string>
,viewinfo array<map<string, string>>
)
PARTITIONED BY ( datestamp string, srcid string, action string, testid string )
row format serde 'com.yahoo.mySerde'
stored as inputformat 'org.apache.hadoop.mapred.SequenceFileInputFormat' outputformat 'org.apache.hadoop.mapred.SequenceFileOutputFormat'
location '/user/pradeepk/mytable';
hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '19174',action='click',testid='NOTESTID') location '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"
I get the following error:
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
If I don't use thrift and use a hive-site.xml to directly talk to the db, the alter table seems to succeed:
hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '19174',action='click',testid='NOTESTID') location '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"
However I get errors when I try to run a query:
[pradeepk@chargesize:~/dev]hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "select src_spaceid from ult_search_austria_ult where datestamp='20091101';"
10/06/17 13:22:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006171322_1913647383.txt
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
java.lang.IllegalArgumentException: Can not create a Path from an empty string
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)
at org.apache.hadoop.fs.Path.<init>(Path.java:90)
at org.apache.hadoop.fs.Path.<init>(Path.java:50)
at org.apache.hadoop.mapred.JobClient.copyRemoteFiles(JobClient.java:523)
at org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient.java:603)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:684)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
Any help is much appreciated.
Pradeep
________________________________
From: Ashish Thusoo [mailto:athusoo@facebook.com]
Sent: Thursday, June 17, 2010 11:15 AM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: alter table add partition error
hmm... Can you send the exact command and also the create table command for this table.
Ashish
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Thursday, June 17, 2010 9:09 AM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: alter table add partition error
Sorry - that was a cut-paste error - I don't have the action part - so I am specifying key-value pairs. Since what I am trying to do seems like a basic operation, I am wondering if it's something to do with my Serde - unfortunately the error I see gives me no clue of what could be wrong - any help would be greatly appreciated!
Thanks,
Pradeep
________________________________
From: yq he [mailto:hhh.heyq@gmail.com]
Sent: Wednesday, June 16, 2010 5:54 PM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: alter table add partition error
Hi Pradeep,
partition definition need to be key-value pairs. partition key `action` seems missed the value.
Thanks
Yongqiang
On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath <pr...@yahoo-inc.com>> wrote:
Hi,
I am trying to create an external table against already existing data in sequencefile format. However I have written a custom Serde to interpret the data. I am able to create the table fine but get the exception shown in the session output below when I try to add partition - any help would be greatly appreciated.
Thanks,
Pradeep
== session output ===
[pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '10',action) location '/user/pradeepk/mytable/20091101/10';"
10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
[pradeepk@chargesize:~/dev/howl]
== session output ===
/tmp/pradeepk/hive.log has:
2010-06-16 17:09:00,841 ERROR exec.DDLTask (SessionState.java:printError(269)) - FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.TApplicationException: get_partition failed: unknown result
at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
at org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: org.apache.thrift.TApplicationException: get_partition failed: unknown result
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition(ThriftHiveMetastore.java:931)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition(ThriftHiveMetastore.java:899)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:500)
at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
... 15 more
The thrift server messages are:
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table : db=default tbl=mytable
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize called
10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition : db=default tbl=mytable
Re: alter table add partition error
Posted by Pradeep Kamath <pr...@yahoo-inc.com>.
Hi Paul,
I have attached a draft of a patch (incomplete). The problem I am
facing is that to make this change I seem to need to change the thrift
idl (hive_metastore.thrift). Looking at the code structure it seems like
the generate java files are checked in rather than generated through the
build (is this so that the build does not depend on the thrift
compiler?). I tried downloading and building thrift but hit issues in
the make install - wanted to check with you if the patch looks correct
so far or if I am on the wrong path. Also if the direction is right, I
need some guidance on generating the java files from the modified idl.
Thanks,
Pradeep
Paul Yang wrote:
>
> Well, the section of code you are referring to is automatically
> generated by the thrift compiler so we can't modify it. As the
> inability to return null is a limitation of Thrift, we can work around
> that issue by throwing an exception instead.
>
>
>
> Check out this thread:
>
> http://publists.facebook.com/pipermail/thrift/2007-September/000085.html
>
>
>
>
>
> *From:* Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
> *Sent:* Monday, June 21, 2010 5:46 PM
> *To:* hive-user@hadoop.apache.org
> *Subject:* RE: alter table add partition error
>
>
>
> Hi Paul,
>
> Yes I don't mind working on a patch though I have no knowledge of
> how thrift code works. I was also confused by your suggestion for a
> fix. I thought the fix would be in
> ThriftHiveMetaStore.Client.recv_get_partition() -- to first check
> if(result.o1 != null) throw result.o1, then to just return
> result.success and not throw the "unknown result" exception. I tried
> something along these lines and the alter table worked. Am I missing
> something?
>
>
>
> Thanks,
>
> Pradeep
>
>
>
> ------------------------------------------------------------------------
>
> *From:* Paul Yang [mailto:pyang@facebook.com]
> *Sent:* Monday, June 21, 2010 2:35 PM
> *To:* hive-user@hadoop.apache.org
> *Subject:* RE: alter table add partition error
>
>
>
> Ah okay. So what's going on is that getPartition() is supposed to
> return null if the specified partition doesn't exist. But because
> Thrift can't handle null return values, we are seeing this exception.
> This isn't a problem with a local metastore because Thrift isn't used.
>
>
>
> One solution to this problem would be modify
> ObjectStore.getPartition() to throw a NoSuchObjectException instead of
> returning null for non-existent partitions. Then Hive.getPartition()
> can catch the exception and return null to retain the original behavior.
>
>
>
> Pradeep, are you interested in working on a patch for this problem?
>
>
>
> *From:* Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
> *Sent:* Monday, June 21, 2010 2:11 PM
> *To:* hive-user@hadoop.apache.org
> *Subject:* RE: alter table add partition error
>
>
>
> I tried debugging in code a little more. Here is what I found:
>
> The code in ThriftHiveMetaStore eventually makes a call --
> get_partition() passing the partition key values for the partition I
> am trying to add using alter table. I assume this is to check that the
> partition doesn't already exist.
>
> I added a debug line in the following code:
>
> public Partition recv_get_partition() throws MetaException, TException
>
> {
>
> TMessage msg = iprot_.readMessageBegin();
>
> if (msg.type == TMessageType.EXCEPTION) {
>
> TApplicationException x = TApplicationException.read(iprot_);
>
> iprot_.readMessageEnd();
>
> throw x;
>
> }
>
> get_partition_result result = new get_partition_result();
>
>
>
> result.read(iprot_);
>
> *System.err.println("XXX: result:" + result);*
>
> iprot_.readMessageEnd();
>
> if (result.isSetSuccess()) {
>
> return result.success;
>
> }
>
> if (result.o1 != null) {
>
> throw result.o1;
>
> }
>
> * throw new
> TApplicationException(TApplicationException.MISSING_RESULT,
> "get_partition failed: unknown result");*
>
> }
>
>
>
> I also put a debug statements in the read() method:
>
> public void read(TProtocol iprot) throws TException {
>
> TField field;
>
> iprot.readStructBegin();
>
> * System.err.println("XXX: Reading TProtocol object:");*
>
> while (true)
>
> {
>
> field = iprot.readFieldBegin();
>
> * System.err.println("XXX: field just read:" + field);*
>
> if (field.type == TType.STOP) {
>
> break;
>
>
>
> *I got *
>
> *XXX: Reading TProtocol object:*
>
> *XXX: field just read:<TField name:'' type:0 field-id:0>*
>
> *XXX: result:get_partition_result(success:null, o1:null)*
>
>
>
> The field read in the thrift response message is of type "STOP" and
> with id of type SUCCESS. This seems right since there are no existing
> partitions. But the way the rest of the code handles this, results in
> the exception.
>
>
>
> Any pointers?
>
>
>
> TIA,
>
> Pradeep
>
> ------------------------------------------------------------------------
>
> *From:* Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
> *Sent:* Friday, June 18, 2010 2:51 PM
> *To:* hive-user@hadoop.apache.org
> *Subject:* RE: alter table add partition error
>
>
>
> *Looks like the standalone script works fine against the existing
> partition:*
>
> *./ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name
> default dummy datestamp=20100602/srcid=100/action=view/testid=10*
>
> Partition(parameters={'transient_lastDdlTime': '1276881277'},
> tableName='dummy', createTime=1276881277, lastAccessTime=0,
> values=['20100602', '100', 'view', '10'], dbName='default',
> sd=StorageDescriptor(outputFormat='org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat',
> sortCols=[], inputFormat='org.apache.hadoop.mapred.TextInputFormat',
> cols=[FieldSchema(comment=None, type='string', name='partition_name'),
> FieldSchema(comment=None, type='int', name='partition_id')],
> compressed=False, bucketCols=[], numBuckets=-1, parameters={},
> serdeInfo=SerDeInfo(serializationLib='org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe',
> name=None, parameters={'serialization.format': '1'}),
> location='hdfs://wilbur21.labs.corp.sp1.yahoo.com/user/pradeepk/dummy/20100602/100/view/10'))
>
> [pradeepk@chargesize:~/dev/howl/src/metastore/src/gen-py/hive_metastore]
>
>
>
> *However when I tried to add another partition with the hive cli using
> thrift:*
>
> *hive -e "ALTER TABLE dummy add partition(datestamp = '20100602',
> srcid = '100',action='click',testid='10') location
> '/user/pradeepk/dummy/20100602/100/click/10';"*
>
> 10/06/18 14:49:13 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
> found in the classpath. Usage of hadoop-site.xml is deprecated.
> Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to
> override properties of core-default.xml, mapred-default.xml and
> hdfs-default.xml respectively
>
> Hive history
> file=/tmp/pradeepk/hive_job_log_pradeepk_201006181449_1158492515.txt
>
> *FAILED: Error in metadata: org.apache.thrift.TApplicationException:
> get_partition failed: unknown result*
>
> *FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask*
>
>
>
> *tail -30 /tmp/pradeepk/hive.log*
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>
> at java.lang.reflect.Method.invoke(Method.java:597)
>
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>
>
>
> 2010-06-18 14:49:14,124 ERROR exec.DDLTask
> (SessionState.java:printError(277)) - FAILED: Error in metadata:
> org.apache.thrift.TApplicationException: get_partition failed: unknown
> result
>
> org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.thrift.TApplicationException: get_partition failed: unknown
> result
>
> at
> org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
>
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:255)
>
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:169)
>
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
>
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
>
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:633)
>
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:506)
>
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:384)
>
> at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
>
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
>
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:267)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>
> at java.lang.reflect.Method.invoke(Method.java:597)
>
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>
> Caused by: org.apache.thrift.TApplicationException: get_partition
> failed: unknown result
>
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition(ThriftHiveMetastore.java:931)
>
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition(ThriftHiveMetastore.java:899)
>
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:500)
>
> at
> org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
>
> ... 15 more
>
>
>
> 2010-06-18 14:49:14,124 ERROR ql.Driver
> (SessionState.java:printError(277)) - FAILED: Execution Error, return
> code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
>
>
>
> ------------------------------------------------------------------------
>
> *From:* Paul Yang [mailto:pyang@facebook.com]
> *Sent:* Friday, June 18, 2010 2:19 PM
> *To:* hive-user@hadoop.apache.org
> *Subject:* RE: alter table add partition error
>
>
>
> Looks like the thrift python libraries aren't in your path. run:
>
>
>
> export PYTHONPATH=<path-to-trunk>/build/dist/lib/py/
>
>
>
> before trying the remote command
>
>
>
> *From:* Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
> *Sent:* Friday, June 18, 2010 1:38 PM
> *To:* hive-user@hadoop.apache.org
> *Subject:* RE: alter table add partition error
>
>
>
> Sorry, don't know python -- so couldn't make sense out of the
> following error when I run the command you suggested:
>
> [src/metastore/src/gen-py/hive_metastore]./ThriftHiveMetastore-remote
> -h localhost:9080 get_partition_by_name default dummy
> datestamp=20100602/srcid=100/action=view/testid=10
>
> Traceback (most recent call last):
>
> File "./ThriftHiveMetastore-remote", line 11, in ?
>
> from thrift.transport import TTransport
>
> ImportError: No module named thrift.transport
>
>
>
> Thanks,
>
> Pradeep
>
>
>
> ------------------------------------------------------------------------
>
> *From:* Paul Yang [mailto:pyang@facebook.com]
> *Sent:* Friday, June 18, 2010 12:10 PM
> *To:* hive-user@hadoop.apache.org
> *Subject:* RE: alter table add partition error
>
>
>
> Hi Pradeep,
>
>
>
> In trunk/metastore/src/gen-py/hive_metastore/, there is a script
> called ThriftHiveMetastore-remote that can be used to test out the
> thrift server independent of the CLI. As a quick test to narrow down
> the problem, after the partition is created, can you try running
>
>
>
> ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name
> default dummy datestamp=20100602/srcid=100/action=view/testid=10
>
>
>
> Cheers,
>
> Paul
>
>
>
> *From:* Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
> *Sent:* Friday, June 18, 2010 11:26 AM
> *To:* hive-user@hadoop.apache.org
> *Subject:* RE: alter table add partition error
>
>
>
> Hi Ning,
>
> I have pasted below [1] the values for HADOOP_CLASSPATH and CLASSPATH
> from an "env" command printed just before the following line in
> execHiveCmd.sh
>
> exec $HADOOP jar ${HIVE_LIB}/hive-cli-*.jar $CLASS $HIVE_OPTS "$@"
>
>
>
> There is only one occurrence of libthrift.jar whose cksum matches the
> one from lib/libthrift.jar from a hive checkout build.
>
>
>
> The hive_config_without_thrift is a conf directory containing a
> hive-site.xml with jdo settings to connect to the database directly.
> By default I have hive set up to use a hive-site.xml [2] which
> connects to a thrift server running on localhost.
>
>
>
> Any pointers appreciated!
>
>
>
> Thanks,
>
> Pradeep
>
>
>
> [1]
>
> *HADOOP_CLASSPATH=/*home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libexec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.jar*:/home/y/libexec/hive_client/lib/libthrift.jar*:/home/y/libexec/hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:
>
>
>
> *CLASSPATH=/*home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libexec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.jar*:/home/y/libexec/hive_client/lib/libthrift.jar*:/home/y/libexec/hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:
>
>
>
> [2]
>
> *cat /home/y/libexec/hive_client/conf/hive-site.xml *
>
> <?xml version="1.0"?>
>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>
>
>
> <configuration>
>
>
>
> <!-- Hive Configuration can either be stored in this file or in the
> hadoop configuration files -->
>
> <!-- that are implied by Hadoop setup
> variables. -->
>
> <!-- Aside from Hadoop setup variables - this file is provided as a
> convenience so that Hive -->
>
> <!-- users do not have to edit hadoop configuration files (that may be
> managed as a centralized -->
>
> <!--
> resource).
> -->
>
>
>
> <!-- Hive Execution Parameters -->
>
> <property>
>
> * <name>hive.metastore.uris</name>*
>
> * <value>thrift://localhost:9080</value>*
>
> <description>host and port for the thrift metastore server</description>
>
> </property>
>
>
>
> <property>
>
> * <name>hive.metastore.local</name>*
>
> * <value>false</value>*
>
> <description>controls whether to connect to remove metastore server
> or open a new metastore server in Hive Client JVM</description>
>
> </property>
>
>
>
> <property>
>
> <name>hive.metastore.warehouse.dir</name>
>
> <value>/user/pradeepk/hive/warehouse</value>
>
> <description>location of default database for the
> warehouse</description>
>
> </property>
>
>
>
>
>
> </configuration>
>
> ------------------------------------------------------------------------
>
> *From:* Ning Zhang [mailto:nzhang@facebook.com]
> *Sent:* Friday, June 18, 2010 10:50 AM
> *To:* hive-user@hadoop.apache.org
> *Subject:* Re: alter table add partition error
>
>
>
> Pradeep,
>
>
>
> I ran the commands you provided and it succeeded with the expected
> behavior.
>
>
>
> One possibility is that there are multiple versions of libthrift.jar
> in your CLASSPATH (hadoop & hive). Can you check in the Hadoop & Hive
> CLASSPATH so that no other libthrift.jar is there? What is in
> hive_config_without_thrift?
>
>
>
> Thanks,
>
> Ning
>
>
>
> On Jun 18, 2010, at 10:19 AM, Pradeep Kamath wrote:
>
>
>
> I think there are two separate issues here -- I want to open a jira
> for the first one since I am now able to reproduce it even with text
> format with builtin Serdes. Essentially this is a bug in the thrift
> code (not sure if it is in the client or server) since the same alter
> table statement works fine when the hive client does not use thrift.
> Here are the details:
>
>
>
> *cat create_dummy.sql*
>
> CREATE external TABLE if not exists dummy (
>
>
>
> partition_name string
>
> ,partition_id int
>
> )
>
> PARTITIONED BY ( datestamp string, srcid string, action string, testid
> string )
>
> row format delimited
>
> stored as textfile
>
> location '/user/pradeepk/dummy';
>
>
>
> *hive -f create_dummy.sql*
>
> 10/06/18 10:13:36 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
> found in the classpath. Usage of hadoop-site.xml is deprecated.
> Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to
> override properties of core-default.xml, mapred-default.xml and
> hdfs-default.xml respectively
>
> Hive history
> file=/tmp/pradeepk/hive_job_log_pradeepk_201006181013_184583537.txt
>
> OK
>
> Time taken: 0.627 seconds
>
> * *
>
> *hive -e "ALTER TABLE dummy add partition(datestamp = '20100602',
> srcid = '100',action='view',testid='10') location
> '/user/pradeepk/dummy/20100602/100/view/10';"*
>
> 10/06/18 10:14:11 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
> found in the classpath. Usage of hadoop-site.xml is deprecated.
> Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to
> override properties of core-default.xml, mapred-default.xml and
> hdfs-default.xml respectively
>
> Hive history
> file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_700722546.txt
>
> *FAILED: Error in metadata: org.apache.thrift.TApplicationException:
> get_partition failed: unknown result*
>
> *FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask*
>
>
>
> *hive --config hive_conf_without_thrift -e "ALTER TABLE dummy add
> partition(datestamp = '20100602', srcid =
> '100',action='view',testid='10') location
> '/user/pradeepk/dummy/20100602/100/view/10';"*
>
> 10/06/18 10:14:31 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
> found in the classpath. Usage of hadoop-site.xml is deprecated.
> Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to
> override properties of core-default.xml, mapred-default.xml and
> hdfs-default.xml respectively
>
> Hive history
> file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_598649843.txt
>
> *OK*
>
> Time taken: 5.849 seconds
>
>
>
> Is there some thrift setting I am missing or is this a bug? -- If it
> is the latter, I can open a jira with the above details.
>
>
>
> Thanks,
>
> Pradeep
>
>
>
>
>
> ------------------------------------------------------------------------
>
> *From:* Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
> *Sent:* Thursday, June 17, 2010 1:25 PM
> *To:* hive-user@hadoop.apache.org <ma...@hadoop.apache.org>
> *Subject:* RE: alter table add partition error
>
>
>
> Here are the create table and alter table statements:
>
> CREATE external TABLE if not exists mytable (
>
>
>
> bc string
>
> ,src_spaceid string
>
> ,srcpvid string
>
> ,dstpvid string
>
> ,dst_spaceid string
>
> ,page_params map<string, string>
>
> ,clickinfo map<string, string>
>
> ,viewinfo array<map<string, string>>
>
>
>
> )
>
> PARTITIONED BY ( datestamp string, srcid string, action string, testid
> string )
>
> row format serde 'com.yahoo.mySerde'
>
> stored as inputformat
> 'org.apache.hadoop.mapred.SequenceFileInputFormat' outputformat
> 'org.apache.hadoop.mapred.SequenceFileOutputFormat'
>
> location '/user/pradeepk/mytable';
>
>
>
> hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add
> partition(datestamp = '20091101', srcid =
> '19174',action='click',testid='NOTESTID') location
> '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"
>
>
>
> I get the following error:
>
> Hive history
> file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
>
> *FAILED: Error in metadata: org.apache.thrift.TApplicationException:
> get_partition failed: unknown result*
>
> *FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask*
>
> If I don't use thrift and use a hive-site.xml to directly talk to the
> db, the alter table seems to succeed:
>
> hive --auxpath ult-serde.jar *--config hive_conf_without_thrift* -e
> "ALTER TABLE mytable add partition(datestamp = '20091101', srcid =
> '19174',action='click',testid='NOTESTID') location
> '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"
>
>
>
> However I get errors when I try to run a query:
>
> [pradeepk@chargesize:~/dev]*hive --auxpath ult-serde.jar --config
> hive_conf_without_thrift -e "select src_spaceid from
> ult_search_austria_ult where datestamp='20091101';"*
>
> 10/06/17 13:22:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
> found in the classpath. Usage of hadoop-site.xml is deprecated.
> Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to
> override properties of core-default.xml, mapred-default.xml and
> hdfs-default.xml respectively
>
> Hive history
> file=/tmp/pradeepk/hive_job_log_pradeepk_201006171322_1913647383.txt
>
> Total MapReduce jobs = 1
>
> Launching Job 1 out of 1
>
> Number of reduce tasks is set to 0 since there's no reduce operator
>
> java.lang.IllegalArgumentException: Can not create a Path from an
> empty string
>
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)
>
> at org.apache.hadoop.fs.Path.<init>(Path.java:90)
>
> at org.apache.hadoop.fs.Path.<init>(Path.java:50)
>
> at
> org.apache.hadoop.mapred.JobClient.copyRemoteFiles(JobClient.java:523)
>
> at
> org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient.java:603)
>
> at
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)
>
> at
> org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
>
> at
> org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:684)
>
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
>
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
>
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
>
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
>
>
>
> Any help is much appreciated.
>
>
>
> Pradeep
>
>
>
> ------------------------------------------------------------------------
>
> *From:* Ashish Thusoo [mailto:athusoo@facebook.com]
> *Sent:* Thursday, June 17, 2010 11:15 AM
> *To:* hive-user@hadoop.apache.org <ma...@hadoop.apache.org>
> *Subject:* RE: alter table add partition error
>
>
>
> hmm... Can you send the exact command and also the create table
> command for this table.
>
>
>
> Ashish
>
>
>
> ------------------------------------------------------------------------
>
> *From:* Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
> *Sent:* Thursday, June 17, 2010 9:09 AM
> *To:* hive-user@hadoop.apache.org <ma...@hadoop.apache.org>
> *Subject:* RE: alter table add partition error
>
> Sorry -- that was a cut-paste error -- I don't have the action part --
> so I am specifying key-value pairs. Since what I am trying to do seems
> like a basic operation, I am wondering if it's something to do with my
> Serde -- unfortunately the error I see gives me no clue of what could
> be wrong -- any help would be greatly appreciated!
>
>
>
> Thanks,
>
> Pradeep
>
>
>
> ------------------------------------------------------------------------
>
> *From:* yq he [mailto:hhh.heyq@gmail.com]
> *Sent:* Wednesday, June 16, 2010 5:54 PM
> *To:* hive-user@hadoop.apache.org <ma...@hadoop.apache.org>
> *Subject:* Re: alter table add partition error
>
>
>
> Hi Pradeep,
>
>
>
> partition definition need to be key-value pairs. partition key
> `action` seems missed the value.
>
>
>
> Thanks
>
> Yongqiang
>
> On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath
> <pradeepk@yahoo-inc.com <ma...@yahoo-inc.com>> wrote:
>
> Hi,
>
> I am trying to create an external table against already existing
> data in sequencefile format. However I have written a custom Serde to
> interpret the data. I am able to create the table fine but get the
> exception shown in the session output below when I try to add
> partition -- any help would be greatly appreciated.
>
>
>
> Thanks,
>
> Pradeep
>
>
>
> == session output ===
>
>
>
> [pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER
> TABLE mytable add partition(datestamp = '20091101', srcid =
> '10',action) location '/user/pradeepk/mytable/20091101/10';"
>
> 10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
> found in the classpath. Usage of hadoop-site.xml is deprecated.
> Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to
> override properties of core-default.xml, mapred-default.xml and
> hdfs-default.xml respectively
>
> Hive history
> file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
>
> *FAILED: Error in metadata: org.apache.thrift.TApplicationException:
> get_partition failed: unknown result*
>
> *FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask*
>
> [pradeepk@chargesize:~/dev/howl]
>
>
>
> == session output ===
>
>
>
> /tmp/pradeepk/hive.log has:
>
> 2010-06-16 17:09:00,841 ERROR exec.DDLTask
> (SessionState.java:printError(269)) - FAILED: Error in metadata:
> org.apache.thrift.TApplicationException: get_partition failed: unknown
> result
>
> org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.thrift.TApplicationException: get_partition failed: unknown
> result
>
> at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
>
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)
>
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)
>
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
>
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
>
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
>
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
>
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
>
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
>
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
>
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>
> at java.lang.reflect.Method.invoke(Method.java:597)
>
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>
> *Caused by: org.apache.thrift.TApplicationException: get_partition
> failed: unknown result*
>
> * at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition(ThriftHiveMetastore.java:931)*
>
> * at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition(ThriftHiveMetastore.java:899)*
>
> * at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:500)*
>
> * at
> org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)*
>
> * ... 15 more*
>
>
>
> The thrift server messages are:
>
> 10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table :
> db=default tbl=mytable
>
> 10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store
> with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
>
> 10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize
> called
>
> 10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore
>
> 10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition :
> db=default tbl=mytable
>
>
>
>
>
>
>
>
>
RE: alter table add partition error
Posted by Paul Yang <py...@facebook.com>.
Well, the section of code you are referring to is automatically generated by the thrift compiler so we can't modify it. As the inability to return null is a limitation of Thrift, we can work around that issue by throwing an exception instead.
Check out this thread:
http://publists.facebook.com/pipermail/thrift/2007-September/000085.html
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Monday, June 21, 2010 5:46 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Hi Paul,
Yes I don't mind working on a patch though I have no knowledge of how thrift code works. I was also confused by your suggestion for a fix. I thought the fix would be in ThriftHiveMetaStore.Client.recv_get_partition() - to first check if(result.o1 != null) throw result.o1, then to just return result.success and not throw the "unknown result" exception. I tried something along these lines and the alter table worked. Am I missing something?
Thanks,
Pradeep
________________________________
From: Paul Yang [mailto:pyang@facebook.com]
Sent: Monday, June 21, 2010 2:35 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Ah okay. So what's going on is that getPartition() is supposed to return null if the specified partition doesn't exist. But because Thrift can't handle null return values, we are seeing this exception. This isn't a problem with a local metastore because Thrift isn't used.
One solution to this problem would be modify ObjectStore.getPartition() to throw a NoSuchObjectException instead of returning null for non-existent partitions. Then Hive.getPartition() can catch the exception and return null to retain the original behavior.
Pradeep, are you interested in working on a patch for this problem?
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Monday, June 21, 2010 2:11 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
I tried debugging in code a little more. Here is what I found:
The code in ThriftHiveMetaStore eventually makes a call - get_partition() passing the partition key values for the partition I am trying to add using alter table. I assume this is to check that the partition doesn't already exist.
I added a debug line in the following code:
public Partition recv_get_partition() throws MetaException, TException
{
TMessage msg = iprot_.readMessageBegin();
if (msg.type == TMessageType.EXCEPTION) {
TApplicationException x = TApplicationException.read(iprot_);
iprot_.readMessageEnd();
throw x;
}
get_partition_result result = new get_partition_result();
result.read(iprot_);
System.err.println("XXX: result:" + result);
iprot_.readMessageEnd();
if (result.isSetSuccess()) {
return result.success;
}
if (result.o1 != null) {
throw result.o1;
}
throw new TApplicationException(TApplicationException.MISSING_RESULT, "get_partition failed: unknown result");
}
I also put a debug statements in the read() method:
public void read(TProtocol iprot) throws TException {
TField field;
iprot.readStructBegin();
System.err.println("XXX: Reading TProtocol object:");
while (true)
{
field = iprot.readFieldBegin();
System.err.println("XXX: field just read:" + field);
if (field.type == TType.STOP) {
break;
I got
XXX: Reading TProtocol object:
XXX: field just read:<TField name:'' type:0 field-id:0>
XXX: result:get_partition_result(success:null, o1:null)
The field read in the thrift response message is of type "STOP" and with id of type SUCCESS. This seems right since there are no existing partitions. But the way the rest of the code handles this, results in the exception.
Any pointers?
TIA,
Pradeep
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Friday, June 18, 2010 2:51 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Looks like the standalone script works fine against the existing partition:
./ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name default dummy datestamp=20100602/srcid=100/action=view/testid=10
Partition(parameters={'transient_lastDdlTime': '1276881277'}, tableName='dummy', createTime=1276881277, lastAccessTime=0, values=['20100602', '100', 'view', '10'], dbName='default', sd=StorageDescriptor(outputFormat='org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat', sortCols=[], inputFormat='org.apache.hadoop.mapred.TextInputFormat', cols=[FieldSchema(comment=None, type='string', name='partition_name'), FieldSchema(comment=None, type='int', name='partition_id')], compressed=False, bucketCols=[], numBuckets=-1, parameters={}, serdeInfo=SerDeInfo(serializationLib='org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe', name=None, parameters={'serialization.format': '1'}), location='hdfs://wilbur21.labs.corp.sp1.yahoo.com/user/pradeepk/dummy/20100602/100/view/10'))
[pradeepk@chargesize:~/dev/howl/src/metastore/src/gen-py/hive_metastore]
However when I tried to add another partition with the hive cli using thrift:
hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid = '100',action='click',testid='10') location '/user/pradeepk/dummy/20100602/100/click/10';"
10/06/18 14:49:13 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181449_1158492515.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
tail -30 /tmp/pradeepk/hive.log
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
2010-06-18 14:49:14,124 ERROR exec.DDLTask (SessionState.java:printError(277)) - FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.TApplicationException: get_partition failed: unknown result
at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
at org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:255)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:169)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:633)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:506)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:384)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:267)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: org.apache.thrift.TApplicationException: get_partition failed: unknown result
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition(ThriftHiveMetastore.java:931)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition(ThriftHiveMetastore.java:899)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:500)
at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
... 15 more
2010-06-18 14:49:14,124 ERROR ql.Driver (SessionState.java:printError(277)) - FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
________________________________
From: Paul Yang [mailto:pyang@facebook.com]
Sent: Friday, June 18, 2010 2:19 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Looks like the thrift python libraries aren't in your path. run:
export PYTHONPATH=<path-to-trunk>/build/dist/lib/py/
before trying the remote command
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Friday, June 18, 2010 1:38 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Sorry, don't know python - so couldn't make sense out of the following error when I run the command you suggested:
[src/metastore/src/gen-py/hive_metastore]./ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name default dummy datestamp=20100602/srcid=100/action=view/testid=10
Traceback (most recent call last):
File "./ThriftHiveMetastore-remote", line 11, in ?
from thrift.transport import TTransport
ImportError: No module named thrift.transport
Thanks,
Pradeep
________________________________
From: Paul Yang [mailto:pyang@facebook.com]
Sent: Friday, June 18, 2010 12:10 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Hi Pradeep,
In trunk/metastore/src/gen-py/hive_metastore/, there is a script called ThriftHiveMetastore-remote that can be used to test out the thrift server independent of the CLI. As a quick test to narrow down the problem, after the partition is created, can you try running
ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name default dummy datestamp=20100602/srcid=100/action=view/testid=10
Cheers,
Paul
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Friday, June 18, 2010 11:26 AM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Hi Ning,
I have pasted below [1] the values for HADOOP_CLASSPATH and CLASSPATH from an "env" command printed just before the following line in execHiveCmd.sh
exec $HADOOP jar ${HIVE_LIB}/hive-cli-*.jar $CLASS $HIVE_OPTS "$@"
There is only one occurrence of libthrift.jar whose cksum matches the one from lib/libthrift.jar from a hive checkout build.
The hive_config_without_thrift is a conf directory containing a hive-site.xml with jdo settings to connect to the database directly. By default I have hive set up to use a hive-site.xml [2] which connects to a thrift server running on localhost.
Any pointers appreciated!
Thanks,
Pradeep
[1]
HADOOP_CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libexec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.jar:/home/y/libexec/hive_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:
CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libexec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.jar:/home/y/libexec/hive_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:
[2]
cat /home/y/libexec/hive_client/conf/hive-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<!-- Hive Configuration can either be stored in this file or in the hadoop configuration files -->
<!-- that are implied by Hadoop setup variables. -->
<!-- Aside from Hadoop setup variables - this file is provided as a convenience so that Hive -->
<!-- users do not have to edit hadoop configuration files (that may be managed as a centralized -->
<!-- resource). -->
<!-- Hive Execution Parameters -->
<property>
<name>hive.metastore.uris</name>
<value>thrift://localhost:9080</value>
<description>host and port for the thrift metastore server</description>
</property>
<property>
<name>hive.metastore.local</name>
<value>false</value>
<description>controls whether to connect to remove metastore server or open a new metastore server in Hive Client JVM</description>
</property>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/user/pradeepk/hive/warehouse</value>
<description>location of default database for the warehouse</description>
</property>
</configuration>
________________________________
From: Ning Zhang [mailto:nzhang@facebook.com]
Sent: Friday, June 18, 2010 10:50 AM
To: hive-user@hadoop.apache.org
Subject: Re: alter table add partition error
Pradeep,
I ran the commands you provided and it succeeded with the expected behavior.
One possibility is that there are multiple versions of libthrift.jar in your CLASSPATH (hadoop & hive). Can you check in the Hadoop & Hive CLASSPATH so that no other libthrift.jar is there? What is in hive_config_without_thrift?
Thanks,
Ning
On Jun 18, 2010, at 10:19 AM, Pradeep Kamath wrote:
I think there are two separate issues here - I want to open a jira for the first one since I am now able to reproduce it even with text format with builtin Serdes. Essentially this is a bug in the thrift code (not sure if it is in the client or server) since the same alter table statement works fine when the hive client does not use thrift. Here are the details:
cat create_dummy.sql
CREATE external TABLE if not exists dummy (
partition_name string
,partition_id int
)
PARTITIONED BY ( datestamp string, srcid string, action string, testid string )
row format delimited
stored as textfile
location '/user/pradeepk/dummy';
hive -f create_dummy.sql
10/06/18 10:13:36 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181013_184583537.txt
OK
Time taken: 0.627 seconds
hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid = '100',action='view',testid='10') location '/user/pradeepk/dummy/20100602/100/view/10';"
10/06/18 10:14:11 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_700722546.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
hive --config hive_conf_without_thrift -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid = '100',action='view',testid='10') location '/user/pradeepk/dummy/20100602/100/view/10';"
10/06/18 10:14:31 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_598649843.txt
OK
Time taken: 5.849 seconds
Is there some thrift setting I am missing or is this a bug? - If it is the latter, I can open a jira with the above details.
Thanks,
Pradeep
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Thursday, June 17, 2010 1:25 PM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: alter table add partition error
Here are the create table and alter table statements:
CREATE external TABLE if not exists mytable (
bc string
,src_spaceid string
,srcpvid string
,dstpvid string
,dst_spaceid string
,page_params map<string, string>
,clickinfo map<string, string>
,viewinfo array<map<string, string>>
)
PARTITIONED BY ( datestamp string, srcid string, action string, testid string )
row format serde 'com.yahoo.mySerde'
stored as inputformat 'org.apache.hadoop.mapred.SequenceFileInputFormat' outputformat 'org.apache.hadoop.mapred.SequenceFileOutputFormat'
location '/user/pradeepk/mytable';
hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '19174',action='click',testid='NOTESTID') location '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"
I get the following error:
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
If I don't use thrift and use a hive-site.xml to directly talk to the db, the alter table seems to succeed:
hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '19174',action='click',testid='NOTESTID') location '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"
However I get errors when I try to run a query:
[pradeepk@chargesize:~/dev]hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "select src_spaceid from ult_search_austria_ult where datestamp='20091101';"
10/06/17 13:22:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006171322_1913647383.txt
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
java.lang.IllegalArgumentException: Can not create a Path from an empty string
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)
at org.apache.hadoop.fs.Path.<init>(Path.java:90)
at org.apache.hadoop.fs.Path.<init>(Path.java:50)
at org.apache.hadoop.mapred.JobClient.copyRemoteFiles(JobClient.java:523)
at org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient.java:603)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:684)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
Any help is much appreciated.
Pradeep
________________________________
From: Ashish Thusoo [mailto:athusoo@facebook.com]
Sent: Thursday, June 17, 2010 11:15 AM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: alter table add partition error
hmm... Can you send the exact command and also the create table command for this table.
Ashish
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Thursday, June 17, 2010 9:09 AM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: alter table add partition error
Sorry - that was a cut-paste error - I don't have the action part - so I am specifying key-value pairs. Since what I am trying to do seems like a basic operation, I am wondering if it's something to do with my Serde - unfortunately the error I see gives me no clue of what could be wrong - any help would be greatly appreciated!
Thanks,
Pradeep
________________________________
From: yq he [mailto:hhh.heyq@gmail.com]
Sent: Wednesday, June 16, 2010 5:54 PM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: alter table add partition error
Hi Pradeep,
partition definition need to be key-value pairs. partition key `action` seems missed the value.
Thanks
Yongqiang
On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath <pr...@yahoo-inc.com>> wrote:
Hi,
I am trying to create an external table against already existing data in sequencefile format. However I have written a custom Serde to interpret the data. I am able to create the table fine but get the exception shown in the session output below when I try to add partition - any help would be greatly appreciated.
Thanks,
Pradeep
== session output ===
[pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '10',action) location '/user/pradeepk/mytable/20091101/10';"
10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
[pradeepk@chargesize:~/dev/howl]
== session output ===
/tmp/pradeepk/hive.log has:
2010-06-16 17:09:00,841 ERROR exec.DDLTask (SessionState.java:printError(269)) - FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.TApplicationException: get_partition failed: unknown result
at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
at org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: org.apache.thrift.TApplicationException: get_partition failed: unknown result
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition(ThriftHiveMetastore.java:931)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition(ThriftHiveMetastore.java:899)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:500)
at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
... 15 more
The thrift server messages are:
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table : db=default tbl=mytable
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize called
10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition : db=default tbl=mytable
RE: alter table add partition error
Posted by Pradeep Kamath <pr...@yahoo-inc.com>.
Hi Paul,
Yes I don't mind working on a patch though I have no knowledge of how
thrift code works. I was also confused by your suggestion for a fix. I
thought the fix would be in
ThriftHiveMetaStore.Client.recv_get_partition() - to first check
if(result.o1 != null) throw result.o1, then to just return
result.success and not throw the "unknown result" exception. I tried
something along these lines and the alter table worked. Am I missing
something?
Thanks,
Pradeep
________________________________
From: Paul Yang [mailto:pyang@facebook.com]
Sent: Monday, June 21, 2010 2:35 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Ah okay. So what's going on is that getPartition() is supposed to return
null if the specified partition doesn't exist. But because Thrift can't
handle null return values, we are seeing this exception. This isn't a
problem with a local metastore because Thrift isn't used.
One solution to this problem would be modify ObjectStore.getPartition()
to throw a NoSuchObjectException instead of returning null for
non-existent partitions. Then Hive.getPartition() can catch the
exception and return null to retain the original behavior.
Pradeep, are you interested in working on a patch for this problem?
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Monday, June 21, 2010 2:11 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
I tried debugging in code a little more. Here is what I found:
The code in ThriftHiveMetaStore eventually makes a call -
get_partition() passing the partition key values for the partition I am
trying to add using alter table. I assume this is to check that the
partition doesn't already exist.
I added a debug line in the following code:
public Partition recv_get_partition() throws MetaException, TException
{
TMessage msg = iprot_.readMessageBegin();
if (msg.type == TMessageType.EXCEPTION) {
TApplicationException x = TApplicationException.read(iprot_);
iprot_.readMessageEnd();
throw x;
}
get_partition_result result = new get_partition_result();
result.read(iprot_);
System.err.println("XXX: result:" + result);
iprot_.readMessageEnd();
if (result.isSetSuccess()) {
return result.success;
}
if (result.o1 != null) {
throw result.o1;
}
throw new
TApplicationException(TApplicationException.MISSING_RESULT,
"get_partition failed: unknown result");
}
I also put a debug statements in the read() method:
public void read(TProtocol iprot) throws TException {
TField field;
iprot.readStructBegin();
System.err.println("XXX: Reading TProtocol object:");
while (true)
{
field = iprot.readFieldBegin();
System.err.println("XXX: field just read:" + field);
if (field.type == TType.STOP) {
break;
I got
XXX: Reading TProtocol object:
XXX: field just read:<TField name:'' type:0 field-id:0>
XXX: result:get_partition_result(success:null, o1:null)
The field read in the thrift response message is of type "STOP" and with
id of type SUCCESS. This seems right since there are no existing
partitions. But the way the rest of the code handles this, results in
the exception.
Any pointers?
TIA,
Pradeep
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Friday, June 18, 2010 2:51 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Looks like the standalone script works fine against the existing
partition:
./ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name
default dummy datestamp=20100602/srcid=100/action=view/testid=10
Partition(parameters={'transient_lastDdlTime': '1276881277'},
tableName='dummy', createTime=1276881277, lastAccessTime=0,
values=['20100602', '100', 'view', '10'], dbName='default',
sd=StorageDescriptor(outputFormat='org.apache.hadoop.hive.ql.io.HiveIgno
reKeyTextOutputFormat', sortCols=[],
inputFormat='org.apache.hadoop.mapred.TextInputFormat',
cols=[FieldSchema(comment=None, type='string', name='partition_name'),
FieldSchema(comment=None, type='int', name='partition_id')],
compressed=False, bucketCols=[], numBuckets=-1, parameters={},
serdeInfo=SerDeInfo(serializationLib='org.apache.hadoop.hive.serde2.lazy
.LazySimpleSerDe', name=None, parameters={'serialization.format': '1'}),
location='hdfs://wilbur21.labs.corp.sp1.yahoo.com/user/pradeepk/dummy/20
100602/100/view/10'))
[pradeepk@chargesize:~/dev/howl/src/metastore/src/gen-py/hive_metastore]
However when I tried to add another partition with the hive cli using
thrift:
hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid
= '100',action='click',testid='10') location
'/user/pradeepk/dummy/20100602/100/click/10';"
10/06/18 14:49:13 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006181449_1158492515.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException:
get_partition failed: unknown result
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
tail -30 /tmp/pradeepk/hive.log
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
2010-06-18 14:49:14,124 ERROR exec.DDLTask
(SessionState.java:printError(277)) - FAILED: Error in metadata:
org.apache.thrift.TApplicationException: get_partition failed: unknown
result
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.thrift.TApplicationException: get_partition failed: unknown
result
at
org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
at
org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:255)
at
org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:169)
at
org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:633)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:506)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:384)
at
org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:267)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: org.apache.thrift.TApplicationException: get_partition
failed: unknown result
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get
_partition(ThriftHiveMetastore.java:931)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_part
ition(ThriftHiveMetastore.java:899)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMe
taStoreClient.java:500)
at
org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
... 15 more
2010-06-18 14:49:14,124 ERROR ql.Driver
(SessionState.java:printError(277)) - FAILED: Execution Error, return
code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
________________________________
From: Paul Yang [mailto:pyang@facebook.com]
Sent: Friday, June 18, 2010 2:19 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Looks like the thrift python libraries aren't in your path. run:
export PYTHONPATH=<path-to-trunk>/build/dist/lib/py/
before trying the remote command
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Friday, June 18, 2010 1:38 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Sorry, don't know python - so couldn't make sense out of the following
error when I run the command you suggested:
[src/metastore/src/gen-py/hive_metastore]./ThriftHiveMetastore-remote -h
localhost:9080 get_partition_by_name default dummy
datestamp=20100602/srcid=100/action=view/testid=10
Traceback (most recent call last):
File "./ThriftHiveMetastore-remote", line 11, in ?
from thrift.transport import TTransport
ImportError: No module named thrift.transport
Thanks,
Pradeep
________________________________
From: Paul Yang [mailto:pyang@facebook.com]
Sent: Friday, June 18, 2010 12:10 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Hi Pradeep,
In trunk/metastore/src/gen-py/hive_metastore/, there is a script called
ThriftHiveMetastore-remote that can be used to test out the thrift
server independent of the CLI. As a quick test to narrow down the
problem, after the partition is created, can you try running
ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name
default dummy datestamp=20100602/srcid=100/action=view/testid=10
Cheers,
Paul
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Friday, June 18, 2010 11:26 AM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Hi Ning,
I have pasted below [1] the values for HADOOP_CLASSPATH and CLASSPATH
from an "env" command printed just before the following line in
execHiveCmd.sh
exec $HADOOP jar ${HIVE_LIB}/hive-cli-*.jar $CLASS $HIVE_OPTS "$@"
There is only one occurrence of libthrift.jar whose cksum matches the
one from lib/libthrift.jar from a hive checkout build.
The hive_config_without_thrift is a conf directory containing a
hive-site.xml with jdo settings to connect to the database directly. By
default I have hive set up to use a hive-site.xml [2] which connects to
a thrift server running on localhost.
Any pointers appreciated!
Thanks,
Pradeep
[1]
HADOOP_CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_c
lient/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.
1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home
/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_cl
ient/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/c
ommons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.j
ar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/lib
exec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_
client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucl
eus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus
-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1
.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home
/y/libexec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hba
se-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/hom
e/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive
_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-comm
on-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/li
bexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/li
b/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.
jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/
hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib
/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0
.jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexe
c/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/
lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/li
bexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/li
bfb303.jar:/home/y/libexec/hive_client/lib/libthrift.jar:/home/y/libexec
/hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6
.jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libe
xec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client
/lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.ja
r:
CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/l
ib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/
home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libe
xec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/li
b/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-
dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/hom
e/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hi
ve_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/
lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-con
nectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1
.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:
/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libe
xec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hbase-0.20
.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/lib
exec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client
/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.
0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/h
ive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_
hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/ho
me/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_cl
ient/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-s
erde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/h
ome/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_
client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jli
ne-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/h
ive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.
jar:/home/y/libexec/hive_client/lib/libthrift.jar:/home/y/libexec/hive_c
lient/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/h
ome/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hiv
e_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/ve
locity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:
[2]
cat /home/y/libexec/hive_client/conf/hive-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<!-- Hive Configuration can either be stored in this file or in the
hadoop configuration files -->
<!-- that are implied by Hadoop setup variables.
-->
<!-- Aside from Hadoop setup variables - this file is provided as a
convenience so that Hive -->
<!-- users do not have to edit hadoop configuration files (that may be
managed as a centralized -->
<!-- resource).
-->
<!-- Hive Execution Parameters -->
<property>
<name>hive.metastore.uris</name>
<value>thrift://localhost:9080</value>
<description>host and port for the thrift metastore
server</description>
</property>
<property>
<name>hive.metastore.local</name>
<value>false</value>
<description>controls whether to connect to remove metastore server or
open a new metastore server in Hive Client JVM</description>
</property>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/user/pradeepk/hive/warehouse</value>
<description>location of default database for the
warehouse</description>
</property>
</configuration>
________________________________
From: Ning Zhang [mailto:nzhang@facebook.com]
Sent: Friday, June 18, 2010 10:50 AM
To: hive-user@hadoop.apache.org
Subject: Re: alter table add partition error
Pradeep,
I ran the commands you provided and it succeeded with the expected
behavior.
One possibility is that there are multiple versions of libthrift.jar in
your CLASSPATH (hadoop & hive). Can you check in the Hadoop & Hive
CLASSPATH so that no other libthrift.jar is there? What is in
hive_config_without_thrift?
Thanks,
Ning
On Jun 18, 2010, at 10:19 AM, Pradeep Kamath wrote:
I think there are two separate issues here - I want to open a jira for
the first one since I am now able to reproduce it even with text format
with builtin Serdes. Essentially this is a bug in the thrift code (not
sure if it is in the client or server) since the same alter table
statement works fine when the hive client does not use thrift. Here are
the details:
cat create_dummy.sql
CREATE external TABLE if not exists dummy (
partition_name string
,partition_id int
)
PARTITIONED BY ( datestamp string, srcid string, action string, testid
string )
row format delimited
stored as textfile
location '/user/pradeepk/dummy';
hive -f create_dummy.sql
10/06/18 10:13:36 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006181013_184583537.txt
OK
Time taken: 0.627 seconds
hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid
= '100',action='view',testid='10') location
'/user/pradeepk/dummy/20100602/100/view/10';"
10/06/18 10:14:11 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_700722546.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException:
get_partition failed: unknown result
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
hive --config hive_conf_without_thrift -e "ALTER TABLE dummy add
partition(datestamp = '20100602', srcid =
'100',action='view',testid='10') location
'/user/pradeepk/dummy/20100602/100/view/10';"
10/06/18 10:14:31 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_598649843.txt
OK
Time taken: 5.849 seconds
Is there some thrift setting I am missing or is this a bug? - If it is
the latter, I can open a jira with the above details.
Thanks,
Pradeep
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Thursday, June 17, 2010 1:25 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Here are the create table and alter table statements:
CREATE external TABLE if not exists mytable (
bc string
,src_spaceid string
,srcpvid string
,dstpvid string
,dst_spaceid string
,page_params map<string, string>
,clickinfo map<string, string>
,viewinfo array<map<string, string>>
)
PARTITIONED BY ( datestamp string, srcid string, action string, testid
string )
row format serde 'com.yahoo.mySerde'
stored as inputformat 'org.apache.hadoop.mapred.SequenceFileInputFormat'
outputformat 'org.apache.hadoop.mapred.SequenceFileOutputFormat'
location '/user/pradeepk/mytable';
hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add
partition(datestamp = '20091101', srcid =
'19174',action='click',testid='NOTESTID') location
'/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"
I get the following error:
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException:
get_partition failed: unknown result
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
If I don't use thrift and use a hive-site.xml to directly talk to the
db, the alter table seems to succeed:
hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "ALTER
TABLE mytable add partition(datestamp = '20091101', srcid =
'19174',action='click',testid='NOTESTID') location
'/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"
However I get errors when I try to run a query:
[pradeepk@chargesize:~/dev]hive --auxpath ult-serde.jar --config
hive_conf_without_thrift -e "select src_spaceid from
ult_search_austria_ult where datestamp='20091101';"
10/06/17 13:22:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006171322_1913647383.txt
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
java.lang.IllegalArgumentException: Can not create a Path from an empty
string
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)
at org.apache.hadoop.fs.Path.<init>(Path.java:90)
at org.apache.hadoop.fs.Path.<init>(Path.java:50)
at
org.apache.hadoop.mapred.JobClient.copyRemoteFiles(JobClient.java:523)
at
org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient
.java:603)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)
at
org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
at
org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:684)
at
org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
Any help is much appreciated.
Pradeep
________________________________
From: Ashish Thusoo [mailto:athusoo@facebook.com]
Sent: Thursday, June 17, 2010 11:15 AM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
hmm... Can you send the exact command and also the create table command
for this table.
Ashish
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Thursday, June 17, 2010 9:09 AM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Sorry - that was a cut-paste error - I don't have the action part - so I
am specifying key-value pairs. Since what I am trying to do seems like a
basic operation, I am wondering if it's something to do with my Serde -
unfortunately the error I see gives me no clue of what could be wrong -
any help would be greatly appreciated!
Thanks,
Pradeep
________________________________
From: yq he [mailto:hhh.heyq@gmail.com]
Sent: Wednesday, June 16, 2010 5:54 PM
To: hive-user@hadoop.apache.org
Subject: Re: alter table add partition error
Hi Pradeep,
partition definition need to be key-value pairs. partition key `action`
seems missed the value.
Thanks
Yongqiang
On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath <pr...@yahoo-inc.com>
wrote:
Hi,
I am trying to create an external table against already existing
data in sequencefile format. However I have written a custom Serde to
interpret the data. I am able to create the table fine but get the
exception shown in the session output below when I try to add partition
- any help would be greatly appreciated.
Thanks,
Pradeep
== session output ===
[pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER
TABLE mytable add partition(datestamp = '20091101', srcid = '10',action)
location '/user/pradeepk/mytable/20091101/10';"
10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException:
get_partition failed: unknown result
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
[pradeepk@chargesize:~/dev/howl]
== session output ===
/tmp/pradeepk/hive.log has:
2010-06-16 17:09:00,841 ERROR exec.DDLTask
(SessionState.java:printError(269)) - FAILED: Error in metadata:
org.apache.thrift.TApplicationException: get_partition failed: unknown
result
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.thrift.TApplicationException: get_partition failed: unknown
result
at
org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
at
org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
at
org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: org.apache.thrift.TApplicationException: get_partition
failed: unknown result
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get
_partition(ThriftHiveMetastore.java:931)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_part
ition(ThriftHiveMetastore.java:899)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMe
taStoreClient.java:500)
at
org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
... 15 more
The thrift server messages are:
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table :
db=default tbl=mytable
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store
with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize
called
10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition :
db=default tbl=mytable
RE: alter table add partition error
Posted by Paul Yang <py...@facebook.com>.
Ah okay. So what's going on is that getPartition() is supposed to return null if the specified partition doesn't exist. But because Thrift can't handle null return values, we are seeing this exception. This isn't a problem with a local metastore because Thrift isn't used.
One solution to this problem would be modify ObjectStore.getPartition() to throw a NoSuchObjectException instead of returning null for non-existent partitions. Then Hive.getPartition() can catch the exception and return null to retain the original behavior.
Pradeep, are you interested in working on a patch for this problem?
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Monday, June 21, 2010 2:11 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
I tried debugging in code a little more. Here is what I found:
The code in ThriftHiveMetaStore eventually makes a call - get_partition() passing the partition key values for the partition I am trying to add using alter table. I assume this is to check that the partition doesn't already exist.
I added a debug line in the following code:
public Partition recv_get_partition() throws MetaException, TException
{
TMessage msg = iprot_.readMessageBegin();
if (msg.type == TMessageType.EXCEPTION) {
TApplicationException x = TApplicationException.read(iprot_);
iprot_.readMessageEnd();
throw x;
}
get_partition_result result = new get_partition_result();
result.read(iprot_);
System.err.println("XXX: result:" + result);
iprot_.readMessageEnd();
if (result.isSetSuccess()) {
return result.success;
}
if (result.o1 != null) {
throw result.o1;
}
throw new TApplicationException(TApplicationException.MISSING_RESULT, "get_partition failed: unknown result");
}
I also put a debug statements in the read() method:
public void read(TProtocol iprot) throws TException {
TField field;
iprot.readStructBegin();
System.err.println("XXX: Reading TProtocol object:");
while (true)
{
field = iprot.readFieldBegin();
System.err.println("XXX: field just read:" + field);
if (field.type == TType.STOP) {
break;
I got
XXX: Reading TProtocol object:
XXX: field just read:<TField name:'' type:0 field-id:0>
XXX: result:get_partition_result(success:null, o1:null)
The field read in the thrift response message is of type "STOP" and with id of type SUCCESS. This seems right since there are no existing partitions. But the way the rest of the code handles this, results in the exception.
Any pointers?
TIA,
Pradeep
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Friday, June 18, 2010 2:51 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Looks like the standalone script works fine against the existing partition:
./ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name default dummy datestamp=20100602/srcid=100/action=view/testid=10
Partition(parameters={'transient_lastDdlTime': '1276881277'}, tableName='dummy', createTime=1276881277, lastAccessTime=0, values=['20100602', '100', 'view', '10'], dbName='default', sd=StorageDescriptor(outputFormat='org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat', sortCols=[], inputFormat='org.apache.hadoop.mapred.TextInputFormat', cols=[FieldSchema(comment=None, type='string', name='partition_name'), FieldSchema(comment=None, type='int', name='partition_id')], compressed=False, bucketCols=[], numBuckets=-1, parameters={}, serdeInfo=SerDeInfo(serializationLib='org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe', name=None, parameters={'serialization.format': '1'}), location='hdfs://wilbur21.labs.corp.sp1.yahoo.com/user/pradeepk/dummy/20100602/100/view/10'))
[pradeepk@chargesize:~/dev/howl/src/metastore/src/gen-py/hive_metastore]
However when I tried to add another partition with the hive cli using thrift:
hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid = '100',action='click',testid='10') location '/user/pradeepk/dummy/20100602/100/click/10';"
10/06/18 14:49:13 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181449_1158492515.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
tail -30 /tmp/pradeepk/hive.log
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
2010-06-18 14:49:14,124 ERROR exec.DDLTask (SessionState.java:printError(277)) - FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.TApplicationException: get_partition failed: unknown result
at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
at org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:255)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:169)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:633)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:506)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:384)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:267)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: org.apache.thrift.TApplicationException: get_partition failed: unknown result
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition(ThriftHiveMetastore.java:931)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition(ThriftHiveMetastore.java:899)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:500)
at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
... 15 more
2010-06-18 14:49:14,124 ERROR ql.Driver (SessionState.java:printError(277)) - FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
________________________________
From: Paul Yang [mailto:pyang@facebook.com]
Sent: Friday, June 18, 2010 2:19 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Looks like the thrift python libraries aren't in your path. run:
export PYTHONPATH=<path-to-trunk>/build/dist/lib/py/
before trying the remote command
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Friday, June 18, 2010 1:38 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Sorry, don't know python - so couldn't make sense out of the following error when I run the command you suggested:
[src/metastore/src/gen-py/hive_metastore]./ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name default dummy datestamp=20100602/srcid=100/action=view/testid=10
Traceback (most recent call last):
File "./ThriftHiveMetastore-remote", line 11, in ?
from thrift.transport import TTransport
ImportError: No module named thrift.transport
Thanks,
Pradeep
________________________________
From: Paul Yang [mailto:pyang@facebook.com]
Sent: Friday, June 18, 2010 12:10 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Hi Pradeep,
In trunk/metastore/src/gen-py/hive_metastore/, there is a script called ThriftHiveMetastore-remote that can be used to test out the thrift server independent of the CLI. As a quick test to narrow down the problem, after the partition is created, can you try running
ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name default dummy datestamp=20100602/srcid=100/action=view/testid=10
Cheers,
Paul
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Friday, June 18, 2010 11:26 AM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Hi Ning,
I have pasted below [1] the values for HADOOP_CLASSPATH and CLASSPATH from an "env" command printed just before the following line in execHiveCmd.sh
exec $HADOOP jar ${HIVE_LIB}/hive-cli-*.jar $CLASS $HIVE_OPTS "$@"
There is only one occurrence of libthrift.jar whose cksum matches the one from lib/libthrift.jar from a hive checkout build.
The hive_config_without_thrift is a conf directory containing a hive-site.xml with jdo settings to connect to the database directly. By default I have hive set up to use a hive-site.xml [2] which connects to a thrift server running on localhost.
Any pointers appreciated!
Thanks,
Pradeep
[1]
HADOOP_CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libexec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.jar:/home/y/libexec/hive_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:
CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libexec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.jar:/home/y/libexec/hive_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:
[2]
cat /home/y/libexec/hive_client/conf/hive-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<!-- Hive Configuration can either be stored in this file or in the hadoop configuration files -->
<!-- that are implied by Hadoop setup variables. -->
<!-- Aside from Hadoop setup variables - this file is provided as a convenience so that Hive -->
<!-- users do not have to edit hadoop configuration files (that may be managed as a centralized -->
<!-- resource). -->
<!-- Hive Execution Parameters -->
<property>
<name>hive.metastore.uris</name>
<value>thrift://localhost:9080</value>
<description>host and port for the thrift metastore server</description>
</property>
<property>
<name>hive.metastore.local</name>
<value>false</value>
<description>controls whether to connect to remove metastore server or open a new metastore server in Hive Client JVM</description>
</property>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/user/pradeepk/hive/warehouse</value>
<description>location of default database for the warehouse</description>
</property>
</configuration>
________________________________
From: Ning Zhang [mailto:nzhang@facebook.com]
Sent: Friday, June 18, 2010 10:50 AM
To: hive-user@hadoop.apache.org
Subject: Re: alter table add partition error
Pradeep,
I ran the commands you provided and it succeeded with the expected behavior.
One possibility is that there are multiple versions of libthrift.jar in your CLASSPATH (hadoop & hive). Can you check in the Hadoop & Hive CLASSPATH so that no other libthrift.jar is there? What is in hive_config_without_thrift?
Thanks,
Ning
On Jun 18, 2010, at 10:19 AM, Pradeep Kamath wrote:
I think there are two separate issues here - I want to open a jira for the first one since I am now able to reproduce it even with text format with builtin Serdes. Essentially this is a bug in the thrift code (not sure if it is in the client or server) since the same alter table statement works fine when the hive client does not use thrift. Here are the details:
cat create_dummy.sql
CREATE external TABLE if not exists dummy (
partition_name string
,partition_id int
)
PARTITIONED BY ( datestamp string, srcid string, action string, testid string )
row format delimited
stored as textfile
location '/user/pradeepk/dummy';
hive -f create_dummy.sql
10/06/18 10:13:36 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181013_184583537.txt
OK
Time taken: 0.627 seconds
hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid = '100',action='view',testid='10') location '/user/pradeepk/dummy/20100602/100/view/10';"
10/06/18 10:14:11 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_700722546.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
hive --config hive_conf_without_thrift -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid = '100',action='view',testid='10') location '/user/pradeepk/dummy/20100602/100/view/10';"
10/06/18 10:14:31 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_598649843.txt
OK
Time taken: 5.849 seconds
Is there some thrift setting I am missing or is this a bug? - If it is the latter, I can open a jira with the above details.
Thanks,
Pradeep
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Thursday, June 17, 2010 1:25 PM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: alter table add partition error
Here are the create table and alter table statements:
CREATE external TABLE if not exists mytable (
bc string
,src_spaceid string
,srcpvid string
,dstpvid string
,dst_spaceid string
,page_params map<string, string>
,clickinfo map<string, string>
,viewinfo array<map<string, string>>
)
PARTITIONED BY ( datestamp string, srcid string, action string, testid string )
row format serde 'com.yahoo.mySerde'
stored as inputformat 'org.apache.hadoop.mapred.SequenceFileInputFormat' outputformat 'org.apache.hadoop.mapred.SequenceFileOutputFormat'
location '/user/pradeepk/mytable';
hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '19174',action='click',testid='NOTESTID') location '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"
I get the following error:
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
If I don't use thrift and use a hive-site.xml to directly talk to the db, the alter table seems to succeed:
hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '19174',action='click',testid='NOTESTID') location '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"
However I get errors when I try to run a query:
[pradeepk@chargesize:~/dev]hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "select src_spaceid from ult_search_austria_ult where datestamp='20091101';"
10/06/17 13:22:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006171322_1913647383.txt
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
java.lang.IllegalArgumentException: Can not create a Path from an empty string
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)
at org.apache.hadoop.fs.Path.<init>(Path.java:90)
at org.apache.hadoop.fs.Path.<init>(Path.java:50)
at org.apache.hadoop.mapred.JobClient.copyRemoteFiles(JobClient.java:523)
at org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient.java:603)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:684)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
Any help is much appreciated.
Pradeep
________________________________
From: Ashish Thusoo [mailto:athusoo@facebook.com]
Sent: Thursday, June 17, 2010 11:15 AM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: alter table add partition error
hmm... Can you send the exact command and also the create table command for this table.
Ashish
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Thursday, June 17, 2010 9:09 AM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: alter table add partition error
Sorry - that was a cut-paste error - I don't have the action part - so I am specifying key-value pairs. Since what I am trying to do seems like a basic operation, I am wondering if it's something to do with my Serde - unfortunately the error I see gives me no clue of what could be wrong - any help would be greatly appreciated!
Thanks,
Pradeep
________________________________
From: yq he [mailto:hhh.heyq@gmail.com]
Sent: Wednesday, June 16, 2010 5:54 PM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: alter table add partition error
Hi Pradeep,
partition definition need to be key-value pairs. partition key `action` seems missed the value.
Thanks
Yongqiang
On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath <pr...@yahoo-inc.com>> wrote:
Hi,
I am trying to create an external table against already existing data in sequencefile format. However I have written a custom Serde to interpret the data. I am able to create the table fine but get the exception shown in the session output below when I try to add partition - any help would be greatly appreciated.
Thanks,
Pradeep
== session output ===
[pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '10',action) location '/user/pradeepk/mytable/20091101/10';"
10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
[pradeepk@chargesize:~/dev/howl]
== session output ===
/tmp/pradeepk/hive.log has:
2010-06-16 17:09:00,841 ERROR exec.DDLTask (SessionState.java:printError(269)) - FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.TApplicationException: get_partition failed: unknown result
at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
at org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: org.apache.thrift.TApplicationException: get_partition failed: unknown result
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition(ThriftHiveMetastore.java:931)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition(ThriftHiveMetastore.java:899)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:500)
at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
... 15 more
The thrift server messages are:
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table : db=default tbl=mytable
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize called
10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition : db=default tbl=mytable
RE: alter table add partition error
Posted by Pradeep Kamath <pr...@yahoo-inc.com>.
I tried debugging in code a little more. Here is what I found:
The code in ThriftHiveMetaStore eventually makes a call -
get_partition() passing the partition key values for the partition I am
trying to add using alter table. I assume this is to check that the
partition doesn't already exist.
I added a debug line in the following code:
public Partition recv_get_partition() throws MetaException, TException
{
TMessage msg = iprot_.readMessageBegin();
if (msg.type == TMessageType.EXCEPTION) {
TApplicationException x = TApplicationException.read(iprot_);
iprot_.readMessageEnd();
throw x;
}
get_partition_result result = new get_partition_result();
result.read(iprot_);
System.err.println("XXX: result:" + result);
iprot_.readMessageEnd();
if (result.isSetSuccess()) {
return result.success;
}
if (result.o1 != null) {
throw result.o1;
}
throw new
TApplicationException(TApplicationException.MISSING_RESULT,
"get_partition failed: unknown result");
}
I also put a debug statements in the read() method:
public void read(TProtocol iprot) throws TException {
TField field;
iprot.readStructBegin();
System.err.println("XXX: Reading TProtocol object:");
while (true)
{
field = iprot.readFieldBegin();
System.err.println("XXX: field just read:" + field);
if (field.type == TType.STOP) {
break;
I got
XXX: Reading TProtocol object:
XXX: field just read:<TField name:'' type:0 field-id:0>
XXX: result:get_partition_result(success:null, o1:null)
The field read in the thrift response message is of type "STOP" and with
id of type SUCCESS. This seems right since there are no existing
partitions. But the way the rest of the code handles this, results in
the exception.
Any pointers?
TIA,
Pradeep
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Friday, June 18, 2010 2:51 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Looks like the standalone script works fine against the existing
partition:
./ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name
default dummy datestamp=20100602/srcid=100/action=view/testid=10
Partition(parameters={'transient_lastDdlTime': '1276881277'},
tableName='dummy', createTime=1276881277, lastAccessTime=0,
values=['20100602', '100', 'view', '10'], dbName='default',
sd=StorageDescriptor(outputFormat='org.apache.hadoop.hive.ql.io.HiveIgno
reKeyTextOutputFormat', sortCols=[],
inputFormat='org.apache.hadoop.mapred.TextInputFormat',
cols=[FieldSchema(comment=None, type='string', name='partition_name'),
FieldSchema(comment=None, type='int', name='partition_id')],
compressed=False, bucketCols=[], numBuckets=-1, parameters={},
serdeInfo=SerDeInfo(serializationLib='org.apache.hadoop.hive.serde2.lazy
.LazySimpleSerDe', name=None, parameters={'serialization.format': '1'}),
location='hdfs://wilbur21.labs.corp.sp1.yahoo.com/user/pradeepk/dummy/20
100602/100/view/10'))
[pradeepk@chargesize:~/dev/howl/src/metastore/src/gen-py/hive_metastore]
However when I tried to add another partition with the hive cli using
thrift:
hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid
= '100',action='click',testid='10') location
'/user/pradeepk/dummy/20100602/100/click/10';"
10/06/18 14:49:13 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006181449_1158492515.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException:
get_partition failed: unknown result
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
tail -30 /tmp/pradeepk/hive.log
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
2010-06-18 14:49:14,124 ERROR exec.DDLTask
(SessionState.java:printError(277)) - FAILED: Error in metadata:
org.apache.thrift.TApplicationException: get_partition failed: unknown
result
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.thrift.TApplicationException: get_partition failed: unknown
result
at
org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
at
org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:255)
at
org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:169)
at
org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:633)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:506)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:384)
at
org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:267)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: org.apache.thrift.TApplicationException: get_partition
failed: unknown result
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get
_partition(ThriftHiveMetastore.java:931)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_part
ition(ThriftHiveMetastore.java:899)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMe
taStoreClient.java:500)
at
org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
... 15 more
2010-06-18 14:49:14,124 ERROR ql.Driver
(SessionState.java:printError(277)) - FAILED: Execution Error, return
code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
________________________________
From: Paul Yang [mailto:pyang@facebook.com]
Sent: Friday, June 18, 2010 2:19 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Looks like the thrift python libraries aren't in your path. run:
export PYTHONPATH=<path-to-trunk>/build/dist/lib/py/
before trying the remote command
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Friday, June 18, 2010 1:38 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Sorry, don't know python - so couldn't make sense out of the following
error when I run the command you suggested:
[src/metastore/src/gen-py/hive_metastore]./ThriftHiveMetastore-remote -h
localhost:9080 get_partition_by_name default dummy
datestamp=20100602/srcid=100/action=view/testid=10
Traceback (most recent call last):
File "./ThriftHiveMetastore-remote", line 11, in ?
from thrift.transport import TTransport
ImportError: No module named thrift.transport
Thanks,
Pradeep
________________________________
From: Paul Yang [mailto:pyang@facebook.com]
Sent: Friday, June 18, 2010 12:10 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Hi Pradeep,
In trunk/metastore/src/gen-py/hive_metastore/, there is a script called
ThriftHiveMetastore-remote that can be used to test out the thrift
server independent of the CLI. As a quick test to narrow down the
problem, after the partition is created, can you try running
ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name
default dummy datestamp=20100602/srcid=100/action=view/testid=10
Cheers,
Paul
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Friday, June 18, 2010 11:26 AM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Hi Ning,
I have pasted below [1] the values for HADOOP_CLASSPATH and CLASSPATH
from an "env" command printed just before the following line in
execHiveCmd.sh
exec $HADOOP jar ${HIVE_LIB}/hive-cli-*.jar $CLASS $HIVE_OPTS "$@"
There is only one occurrence of libthrift.jar whose cksum matches the
one from lib/libthrift.jar from a hive checkout build.
The hive_config_without_thrift is a conf directory containing a
hive-site.xml with jdo settings to connect to the database directly. By
default I have hive set up to use a hive-site.xml [2] which connects to
a thrift server running on localhost.
Any pointers appreciated!
Thanks,
Pradeep
[1]
HADOOP_CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_c
lient/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.
1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home
/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_cl
ient/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/c
ommons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.j
ar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/lib
exec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_
client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucl
eus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus
-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1
.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home
/y/libexec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hba
se-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/hom
e/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive
_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-comm
on-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/li
bexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/li
b/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.
jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/
hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib
/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0
.jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexe
c/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/
lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/li
bexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/li
bfb303.jar:/home/y/libexec/hive_client/lib/libthrift.jar:/home/y/libexec
/hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6
.jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libe
xec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client
/lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.ja
r:
CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/l
ib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/
home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libe
xec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/li
b/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-
dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/hom
e/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hi
ve_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/
lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-con
nectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1
.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:
/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libe
xec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hbase-0.20
.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/lib
exec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client
/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.
0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/h
ive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_
hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/ho
me/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_cl
ient/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-s
erde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/h
ome/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_
client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jli
ne-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/h
ive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.
jar:/home/y/libexec/hive_client/lib/libthrift.jar:/home/y/libexec/hive_c
lient/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/h
ome/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hiv
e_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/ve
locity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:
[2]
cat /home/y/libexec/hive_client/conf/hive-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<!-- Hive Configuration can either be stored in this file or in the
hadoop configuration files -->
<!-- that are implied by Hadoop setup variables.
-->
<!-- Aside from Hadoop setup variables - this file is provided as a
convenience so that Hive -->
<!-- users do not have to edit hadoop configuration files (that may be
managed as a centralized -->
<!-- resource).
-->
<!-- Hive Execution Parameters -->
<property>
<name>hive.metastore.uris</name>
<value>thrift://localhost:9080</value>
<description>host and port for the thrift metastore
server</description>
</property>
<property>
<name>hive.metastore.local</name>
<value>false</value>
<description>controls whether to connect to remove metastore server or
open a new metastore server in Hive Client JVM</description>
</property>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/user/pradeepk/hive/warehouse</value>
<description>location of default database for the
warehouse</description>
</property>
</configuration>
________________________________
From: Ning Zhang [mailto:nzhang@facebook.com]
Sent: Friday, June 18, 2010 10:50 AM
To: hive-user@hadoop.apache.org
Subject: Re: alter table add partition error
Pradeep,
I ran the commands you provided and it succeeded with the expected
behavior.
One possibility is that there are multiple versions of libthrift.jar in
your CLASSPATH (hadoop & hive). Can you check in the Hadoop & Hive
CLASSPATH so that no other libthrift.jar is there? What is in
hive_config_without_thrift?
Thanks,
Ning
On Jun 18, 2010, at 10:19 AM, Pradeep Kamath wrote:
I think there are two separate issues here - I want to open a jira for
the first one since I am now able to reproduce it even with text format
with builtin Serdes. Essentially this is a bug in the thrift code (not
sure if it is in the client or server) since the same alter table
statement works fine when the hive client does not use thrift. Here are
the details:
cat create_dummy.sql
CREATE external TABLE if not exists dummy (
partition_name string
,partition_id int
)
PARTITIONED BY ( datestamp string, srcid string, action string, testid
string )
row format delimited
stored as textfile
location '/user/pradeepk/dummy';
hive -f create_dummy.sql
10/06/18 10:13:36 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006181013_184583537.txt
OK
Time taken: 0.627 seconds
hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid
= '100',action='view',testid='10') location
'/user/pradeepk/dummy/20100602/100/view/10';"
10/06/18 10:14:11 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_700722546.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException:
get_partition failed: unknown result
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
hive --config hive_conf_without_thrift -e "ALTER TABLE dummy add
partition(datestamp = '20100602', srcid =
'100',action='view',testid='10') location
'/user/pradeepk/dummy/20100602/100/view/10';"
10/06/18 10:14:31 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_598649843.txt
OK
Time taken: 5.849 seconds
Is there some thrift setting I am missing or is this a bug? - If it is
the latter, I can open a jira with the above details.
Thanks,
Pradeep
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Thursday, June 17, 2010 1:25 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Here are the create table and alter table statements:
CREATE external TABLE if not exists mytable (
bc string
,src_spaceid string
,srcpvid string
,dstpvid string
,dst_spaceid string
,page_params map<string, string>
,clickinfo map<string, string>
,viewinfo array<map<string, string>>
)
PARTITIONED BY ( datestamp string, srcid string, action string, testid
string )
row format serde 'com.yahoo.mySerde'
stored as inputformat 'org.apache.hadoop.mapred.SequenceFileInputFormat'
outputformat 'org.apache.hadoop.mapred.SequenceFileOutputFormat'
location '/user/pradeepk/mytable';
hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add
partition(datestamp = '20091101', srcid =
'19174',action='click',testid='NOTESTID') location
'/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"
I get the following error:
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException:
get_partition failed: unknown result
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
If I don't use thrift and use a hive-site.xml to directly talk to the
db, the alter table seems to succeed:
hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "ALTER
TABLE mytable add partition(datestamp = '20091101', srcid =
'19174',action='click',testid='NOTESTID') location
'/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"
However I get errors when I try to run a query:
[pradeepk@chargesize:~/dev]hive --auxpath ult-serde.jar --config
hive_conf_without_thrift -e "select src_spaceid from
ult_search_austria_ult where datestamp='20091101';"
10/06/17 13:22:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006171322_1913647383.txt
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
java.lang.IllegalArgumentException: Can not create a Path from an empty
string
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)
at org.apache.hadoop.fs.Path.<init>(Path.java:90)
at org.apache.hadoop.fs.Path.<init>(Path.java:50)
at
org.apache.hadoop.mapred.JobClient.copyRemoteFiles(JobClient.java:523)
at
org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient
.java:603)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)
at
org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
at
org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:684)
at
org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
Any help is much appreciated.
Pradeep
________________________________
From: Ashish Thusoo [mailto:athusoo@facebook.com]
Sent: Thursday, June 17, 2010 11:15 AM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
hmm... Can you send the exact command and also the create table command
for this table.
Ashish
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Thursday, June 17, 2010 9:09 AM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Sorry - that was a cut-paste error - I don't have the action part - so I
am specifying key-value pairs. Since what I am trying to do seems like a
basic operation, I am wondering if it's something to do with my Serde -
unfortunately the error I see gives me no clue of what could be wrong -
any help would be greatly appreciated!
Thanks,
Pradeep
________________________________
From: yq he [mailto:hhh.heyq@gmail.com]
Sent: Wednesday, June 16, 2010 5:54 PM
To: hive-user@hadoop.apache.org
Subject: Re: alter table add partition error
Hi Pradeep,
partition definition need to be key-value pairs. partition key `action`
seems missed the value.
Thanks
Yongqiang
On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath <pr...@yahoo-inc.com>
wrote:
Hi,
I am trying to create an external table against already existing
data in sequencefile format. However I have written a custom Serde to
interpret the data. I am able to create the table fine but get the
exception shown in the session output below when I try to add partition
- any help would be greatly appreciated.
Thanks,
Pradeep
== session output ===
[pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER
TABLE mytable add partition(datestamp = '20091101', srcid = '10',action)
location '/user/pradeepk/mytable/20091101/10';"
10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException:
get_partition failed: unknown result
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
[pradeepk@chargesize:~/dev/howl]
== session output ===
/tmp/pradeepk/hive.log has:
2010-06-16 17:09:00,841 ERROR exec.DDLTask
(SessionState.java:printError(269)) - FAILED: Error in metadata:
org.apache.thrift.TApplicationException: get_partition failed: unknown
result
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.thrift.TApplicationException: get_partition failed: unknown
result
at
org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
at
org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
at
org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: org.apache.thrift.TApplicationException: get_partition
failed: unknown result
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get
_partition(ThriftHiveMetastore.java:931)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_part
ition(ThriftHiveMetastore.java:899)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMe
taStoreClient.java:500)
at
org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
... 15 more
The thrift server messages are:
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table :
db=default tbl=mytable
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store
with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize
called
10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition :
db=default tbl=mytable
RE: alter table add partition error
Posted by Pradeep Kamath <pr...@yahoo-inc.com>.
Looks like the standalone script works fine against the existing
partition:
./ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name
default dummy datestamp=20100602/srcid=100/action=view/testid=10
Partition(parameters={'transient_lastDdlTime': '1276881277'},
tableName='dummy', createTime=1276881277, lastAccessTime=0,
values=['20100602', '100', 'view', '10'], dbName='default',
sd=StorageDescriptor(outputFormat='org.apache.hadoop.hive.ql.io.HiveIgno
reKeyTextOutputFormat', sortCols=[],
inputFormat='org.apache.hadoop.mapred.TextInputFormat',
cols=[FieldSchema(comment=None, type='string', name='partition_name'),
FieldSchema(comment=None, type='int', name='partition_id')],
compressed=False, bucketCols=[], numBuckets=-1, parameters={},
serdeInfo=SerDeInfo(serializationLib='org.apache.hadoop.hive.serde2.lazy
.LazySimpleSerDe', name=None, parameters={'serialization.format': '1'}),
location='hdfs://wilbur21.labs.corp.sp1.yahoo.com/user/pradeepk/dummy/20
100602/100/view/10'))
[pradeepk@chargesize:~/dev/howl/src/metastore/src/gen-py/hive_metastore]
However when I tried to add another partition with the hive cli using
thrift:
hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid
= '100',action='click',testid='10') location
'/user/pradeepk/dummy/20100602/100/click/10';"
10/06/18 14:49:13 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006181449_1158492515.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException:
get_partition failed: unknown result
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
tail -30 /tmp/pradeepk/hive.log
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
2010-06-18 14:49:14,124 ERROR exec.DDLTask
(SessionState.java:printError(277)) - FAILED: Error in metadata:
org.apache.thrift.TApplicationException: get_partition failed: unknown
result
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.thrift.TApplicationException: get_partition failed: unknown
result
at
org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
at
org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:255)
at
org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:169)
at
org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:633)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:506)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:384)
at
org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:267)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: org.apache.thrift.TApplicationException: get_partition
failed: unknown result
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get
_partition(ThriftHiveMetastore.java:931)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_part
ition(ThriftHiveMetastore.java:899)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMe
taStoreClient.java:500)
at
org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
... 15 more
2010-06-18 14:49:14,124 ERROR ql.Driver
(SessionState.java:printError(277)) - FAILED: Execution Error, return
code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
________________________________
From: Paul Yang [mailto:pyang@facebook.com]
Sent: Friday, June 18, 2010 2:19 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Looks like the thrift python libraries aren't in your path. run:
export PYTHONPATH=<path-to-trunk>/build/dist/lib/py/
before trying the remote command
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Friday, June 18, 2010 1:38 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Sorry, don't know python - so couldn't make sense out of the following
error when I run the command you suggested:
[src/metastore/src/gen-py/hive_metastore]./ThriftHiveMetastore-remote -h
localhost:9080 get_partition_by_name default dummy
datestamp=20100602/srcid=100/action=view/testid=10
Traceback (most recent call last):
File "./ThriftHiveMetastore-remote", line 11, in ?
from thrift.transport import TTransport
ImportError: No module named thrift.transport
Thanks,
Pradeep
________________________________
From: Paul Yang [mailto:pyang@facebook.com]
Sent: Friday, June 18, 2010 12:10 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Hi Pradeep,
In trunk/metastore/src/gen-py/hive_metastore/, there is a script called
ThriftHiveMetastore-remote that can be used to test out the thrift
server independent of the CLI. As a quick test to narrow down the
problem, after the partition is created, can you try running
ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name
default dummy datestamp=20100602/srcid=100/action=view/testid=10
Cheers,
Paul
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Friday, June 18, 2010 11:26 AM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Hi Ning,
I have pasted below [1] the values for HADOOP_CLASSPATH and CLASSPATH
from an "env" command printed just before the following line in
execHiveCmd.sh
exec $HADOOP jar ${HIVE_LIB}/hive-cli-*.jar $CLASS $HIVE_OPTS "$@"
There is only one occurrence of libthrift.jar whose cksum matches the
one from lib/libthrift.jar from a hive checkout build.
The hive_config_without_thrift is a conf directory containing a
hive-site.xml with jdo settings to connect to the database directly. By
default I have hive set up to use a hive-site.xml [2] which connects to
a thrift server running on localhost.
Any pointers appreciated!
Thanks,
Pradeep
[1]
HADOOP_CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_c
lient/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.
1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home
/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_cl
ient/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/c
ommons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.j
ar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/lib
exec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_
client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucl
eus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus
-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1
.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home
/y/libexec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hba
se-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/hom
e/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive
_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-comm
on-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/li
bexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/li
b/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.
jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/
hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib
/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0
.jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexe
c/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/
lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/li
bexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/li
bfb303.jar:/home/y/libexec/hive_client/lib/libthrift.jar:/home/y/libexec
/hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6
.jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libe
xec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client
/lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.ja
r:
CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/l
ib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/
home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libe
xec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/li
b/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-
dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/hom
e/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hi
ve_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/
lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-con
nectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1
.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:
/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libe
xec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hbase-0.20
.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/lib
exec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client
/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.
0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/h
ive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_
hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/ho
me/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_cl
ient/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-s
erde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/h
ome/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_
client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jli
ne-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/h
ive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.
jar:/home/y/libexec/hive_client/lib/libthrift.jar:/home/y/libexec/hive_c
lient/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/h
ome/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hiv
e_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/ve
locity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:
[2]
cat /home/y/libexec/hive_client/conf/hive-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<!-- Hive Configuration can either be stored in this file or in the
hadoop configuration files -->
<!-- that are implied by Hadoop setup variables.
-->
<!-- Aside from Hadoop setup variables - this file is provided as a
convenience so that Hive -->
<!-- users do not have to edit hadoop configuration files (that may be
managed as a centralized -->
<!-- resource).
-->
<!-- Hive Execution Parameters -->
<property>
<name>hive.metastore.uris</name>
<value>thrift://localhost:9080</value>
<description>host and port for the thrift metastore
server</description>
</property>
<property>
<name>hive.metastore.local</name>
<value>false</value>
<description>controls whether to connect to remove metastore server or
open a new metastore server in Hive Client JVM</description>
</property>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/user/pradeepk/hive/warehouse</value>
<description>location of default database for the
warehouse</description>
</property>
</configuration>
________________________________
From: Ning Zhang [mailto:nzhang@facebook.com]
Sent: Friday, June 18, 2010 10:50 AM
To: hive-user@hadoop.apache.org
Subject: Re: alter table add partition error
Pradeep,
I ran the commands you provided and it succeeded with the expected
behavior.
One possibility is that there are multiple versions of libthrift.jar in
your CLASSPATH (hadoop & hive). Can you check in the Hadoop & Hive
CLASSPATH so that no other libthrift.jar is there? What is in
hive_config_without_thrift?
Thanks,
Ning
On Jun 18, 2010, at 10:19 AM, Pradeep Kamath wrote:
I think there are two separate issues here - I want to open a jira for
the first one since I am now able to reproduce it even with text format
with builtin Serdes. Essentially this is a bug in the thrift code (not
sure if it is in the client or server) since the same alter table
statement works fine when the hive client does not use thrift. Here are
the details:
cat create_dummy.sql
CREATE external TABLE if not exists dummy (
partition_name string
,partition_id int
)
PARTITIONED BY ( datestamp string, srcid string, action string, testid
string )
row format delimited
stored as textfile
location '/user/pradeepk/dummy';
hive -f create_dummy.sql
10/06/18 10:13:36 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006181013_184583537.txt
OK
Time taken: 0.627 seconds
hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid
= '100',action='view',testid='10') location
'/user/pradeepk/dummy/20100602/100/view/10';"
10/06/18 10:14:11 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_700722546.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException:
get_partition failed: unknown result
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
hive --config hive_conf_without_thrift -e "ALTER TABLE dummy add
partition(datestamp = '20100602', srcid =
'100',action='view',testid='10') location
'/user/pradeepk/dummy/20100602/100/view/10';"
10/06/18 10:14:31 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_598649843.txt
OK
Time taken: 5.849 seconds
Is there some thrift setting I am missing or is this a bug? - If it is
the latter, I can open a jira with the above details.
Thanks,
Pradeep
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Thursday, June 17, 2010 1:25 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Here are the create table and alter table statements:
CREATE external TABLE if not exists mytable (
bc string
,src_spaceid string
,srcpvid string
,dstpvid string
,dst_spaceid string
,page_params map<string, string>
,clickinfo map<string, string>
,viewinfo array<map<string, string>>
)
PARTITIONED BY ( datestamp string, srcid string, action string, testid
string )
row format serde 'com.yahoo.mySerde'
stored as inputformat 'org.apache.hadoop.mapred.SequenceFileInputFormat'
outputformat 'org.apache.hadoop.mapred.SequenceFileOutputFormat'
location '/user/pradeepk/mytable';
hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add
partition(datestamp = '20091101', srcid =
'19174',action='click',testid='NOTESTID') location
'/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"
I get the following error:
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException:
get_partition failed: unknown result
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
If I don't use thrift and use a hive-site.xml to directly talk to the
db, the alter table seems to succeed:
hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "ALTER
TABLE mytable add partition(datestamp = '20091101', srcid =
'19174',action='click',testid='NOTESTID') location
'/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"
However I get errors when I try to run a query:
[pradeepk@chargesize:~/dev]hive --auxpath ult-serde.jar --config
hive_conf_without_thrift -e "select src_spaceid from
ult_search_austria_ult where datestamp='20091101';"
10/06/17 13:22:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006171322_1913647383.txt
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
java.lang.IllegalArgumentException: Can not create a Path from an empty
string
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)
at org.apache.hadoop.fs.Path.<init>(Path.java:90)
at org.apache.hadoop.fs.Path.<init>(Path.java:50)
at
org.apache.hadoop.mapred.JobClient.copyRemoteFiles(JobClient.java:523)
at
org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient
.java:603)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)
at
org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
at
org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:684)
at
org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
Any help is much appreciated.
Pradeep
________________________________
From: Ashish Thusoo [mailto:athusoo@facebook.com]
Sent: Thursday, June 17, 2010 11:15 AM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
hmm... Can you send the exact command and also the create table command
for this table.
Ashish
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Thursday, June 17, 2010 9:09 AM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Sorry - that was a cut-paste error - I don't have the action part - so I
am specifying key-value pairs. Since what I am trying to do seems like a
basic operation, I am wondering if it's something to do with my Serde -
unfortunately the error I see gives me no clue of what could be wrong -
any help would be greatly appreciated!
Thanks,
Pradeep
________________________________
From: yq he [mailto:hhh.heyq@gmail.com]
Sent: Wednesday, June 16, 2010 5:54 PM
To: hive-user@hadoop.apache.org
Subject: Re: alter table add partition error
Hi Pradeep,
partition definition need to be key-value pairs. partition key `action`
seems missed the value.
Thanks
Yongqiang
On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath <pr...@yahoo-inc.com>
wrote:
Hi,
I am trying to create an external table against already existing
data in sequencefile format. However I have written a custom Serde to
interpret the data. I am able to create the table fine but get the
exception shown in the session output below when I try to add partition
- any help would be greatly appreciated.
Thanks,
Pradeep
== session output ===
[pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER
TABLE mytable add partition(datestamp = '20091101', srcid = '10',action)
location '/user/pradeepk/mytable/20091101/10';"
10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException:
get_partition failed: unknown result
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
[pradeepk@chargesize:~/dev/howl]
== session output ===
/tmp/pradeepk/hive.log has:
2010-06-16 17:09:00,841 ERROR exec.DDLTask
(SessionState.java:printError(269)) - FAILED: Error in metadata:
org.apache.thrift.TApplicationException: get_partition failed: unknown
result
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.thrift.TApplicationException: get_partition failed: unknown
result
at
org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
at
org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
at
org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: org.apache.thrift.TApplicationException: get_partition
failed: unknown result
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get
_partition(ThriftHiveMetastore.java:931)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_part
ition(ThriftHiveMetastore.java:899)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMe
taStoreClient.java:500)
at
org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
... 15 more
The thrift server messages are:
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table :
db=default tbl=mytable
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store
with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize
called
10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition :
db=default tbl=mytable
RE: alter table add partition error
Posted by Paul Yang <py...@facebook.com>.
Looks like the thrift python libraries aren't in your path. run:
export PYTHONPATH=<path-to-trunk>/build/dist/lib/py/
before trying the remote command
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Friday, June 18, 2010 1:38 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Sorry, don't know python - so couldn't make sense out of the following error when I run the command you suggested:
[src/metastore/src/gen-py/hive_metastore]./ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name default dummy datestamp=20100602/srcid=100/action=view/testid=10
Traceback (most recent call last):
File "./ThriftHiveMetastore-remote", line 11, in ?
from thrift.transport import TTransport
ImportError: No module named thrift.transport
Thanks,
Pradeep
________________________________
From: Paul Yang [mailto:pyang@facebook.com]
Sent: Friday, June 18, 2010 12:10 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Hi Pradeep,
In trunk/metastore/src/gen-py/hive_metastore/, there is a script called ThriftHiveMetastore-remote that can be used to test out the thrift server independent of the CLI. As a quick test to narrow down the problem, after the partition is created, can you try running
ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name default dummy datestamp=20100602/srcid=100/action=view/testid=10
Cheers,
Paul
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Friday, June 18, 2010 11:26 AM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Hi Ning,
I have pasted below [1] the values for HADOOP_CLASSPATH and CLASSPATH from an "env" command printed just before the following line in execHiveCmd.sh
exec $HADOOP jar ${HIVE_LIB}/hive-cli-*.jar $CLASS $HIVE_OPTS "$@"
There is only one occurrence of libthrift.jar whose cksum matches the one from lib/libthrift.jar from a hive checkout build.
The hive_config_without_thrift is a conf directory containing a hive-site.xml with jdo settings to connect to the database directly. By default I have hive set up to use a hive-site.xml [2] which connects to a thrift server running on localhost.
Any pointers appreciated!
Thanks,
Pradeep
[1]
HADOOP_CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libexec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.jar:/home/y/libexec/hive_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:
CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libexec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.jar:/home/y/libexec/hive_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:
[2]
cat /home/y/libexec/hive_client/conf/hive-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<!-- Hive Configuration can either be stored in this file or in the hadoop configuration files -->
<!-- that are implied by Hadoop setup variables. -->
<!-- Aside from Hadoop setup variables - this file is provided as a convenience so that Hive -->
<!-- users do not have to edit hadoop configuration files (that may be managed as a centralized -->
<!-- resource). -->
<!-- Hive Execution Parameters -->
<property>
<name>hive.metastore.uris</name>
<value>thrift://localhost:9080</value>
<description>host and port for the thrift metastore server</description>
</property>
<property>
<name>hive.metastore.local</name>
<value>false</value>
<description>controls whether to connect to remove metastore server or open a new metastore server in Hive Client JVM</description>
</property>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/user/pradeepk/hive/warehouse</value>
<description>location of default database for the warehouse</description>
</property>
</configuration>
________________________________
From: Ning Zhang [mailto:nzhang@facebook.com]
Sent: Friday, June 18, 2010 10:50 AM
To: hive-user@hadoop.apache.org
Subject: Re: alter table add partition error
Pradeep,
I ran the commands you provided and it succeeded with the expected behavior.
One possibility is that there are multiple versions of libthrift.jar in your CLASSPATH (hadoop & hive). Can you check in the Hadoop & Hive CLASSPATH so that no other libthrift.jar is there? What is in hive_config_without_thrift?
Thanks,
Ning
On Jun 18, 2010, at 10:19 AM, Pradeep Kamath wrote:
I think there are two separate issues here - I want to open a jira for the first one since I am now able to reproduce it even with text format with builtin Serdes. Essentially this is a bug in the thrift code (not sure if it is in the client or server) since the same alter table statement works fine when the hive client does not use thrift. Here are the details:
cat create_dummy.sql
CREATE external TABLE if not exists dummy (
partition_name string
,partition_id int
)
PARTITIONED BY ( datestamp string, srcid string, action string, testid string )
row format delimited
stored as textfile
location '/user/pradeepk/dummy';
hive -f create_dummy.sql
10/06/18 10:13:36 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181013_184583537.txt
OK
Time taken: 0.627 seconds
hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid = '100',action='view',testid='10') location '/user/pradeepk/dummy/20100602/100/view/10';"
10/06/18 10:14:11 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_700722546.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
hive --config hive_conf_without_thrift -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid = '100',action='view',testid='10') location '/user/pradeepk/dummy/20100602/100/view/10';"
10/06/18 10:14:31 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_598649843.txt
OK
Time taken: 5.849 seconds
Is there some thrift setting I am missing or is this a bug? - If it is the latter, I can open a jira with the above details.
Thanks,
Pradeep
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Thursday, June 17, 2010 1:25 PM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: alter table add partition error
Here are the create table and alter table statements:
CREATE external TABLE if not exists mytable (
bc string
,src_spaceid string
,srcpvid string
,dstpvid string
,dst_spaceid string
,page_params map<string, string>
,clickinfo map<string, string>
,viewinfo array<map<string, string>>
)
PARTITIONED BY ( datestamp string, srcid string, action string, testid string )
row format serde 'com.yahoo.mySerde'
stored as inputformat 'org.apache.hadoop.mapred.SequenceFileInputFormat' outputformat 'org.apache.hadoop.mapred.SequenceFileOutputFormat'
location '/user/pradeepk/mytable';
hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '19174',action='click',testid='NOTESTID') location '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"
I get the following error:
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
If I don't use thrift and use a hive-site.xml to directly talk to the db, the alter table seems to succeed:
hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '19174',action='click',testid='NOTESTID') location '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"
However I get errors when I try to run a query:
[pradeepk@chargesize:~/dev]hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "select src_spaceid from ult_search_austria_ult where datestamp='20091101';"
10/06/17 13:22:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006171322_1913647383.txt
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
java.lang.IllegalArgumentException: Can not create a Path from an empty string
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)
at org.apache.hadoop.fs.Path.<init>(Path.java:90)
at org.apache.hadoop.fs.Path.<init>(Path.java:50)
at org.apache.hadoop.mapred.JobClient.copyRemoteFiles(JobClient.java:523)
at org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient.java:603)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:684)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
Any help is much appreciated.
Pradeep
________________________________
From: Ashish Thusoo [mailto:athusoo@facebook.com]
Sent: Thursday, June 17, 2010 11:15 AM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: alter table add partition error
hmm... Can you send the exact command and also the create table command for this table.
Ashish
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Thursday, June 17, 2010 9:09 AM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: alter table add partition error
Sorry - that was a cut-paste error - I don't have the action part - so I am specifying key-value pairs. Since what I am trying to do seems like a basic operation, I am wondering if it's something to do with my Serde - unfortunately the error I see gives me no clue of what could be wrong - any help would be greatly appreciated!
Thanks,
Pradeep
________________________________
From: yq he [mailto:hhh.heyq@gmail.com]
Sent: Wednesday, June 16, 2010 5:54 PM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: alter table add partition error
Hi Pradeep,
partition definition need to be key-value pairs. partition key `action` seems missed the value.
Thanks
Yongqiang
On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath <pr...@yahoo-inc.com>> wrote:
Hi,
I am trying to create an external table against already existing data in sequencefile format. However I have written a custom Serde to interpret the data. I am able to create the table fine but get the exception shown in the session output below when I try to add partition - any help would be greatly appreciated.
Thanks,
Pradeep
== session output ===
[pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '10',action) location '/user/pradeepk/mytable/20091101/10';"
10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
[pradeepk@chargesize:~/dev/howl]
== session output ===
/tmp/pradeepk/hive.log has:
2010-06-16 17:09:00,841 ERROR exec.DDLTask (SessionState.java:printError(269)) - FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.TApplicationException: get_partition failed: unknown result
at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
at org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: org.apache.thrift.TApplicationException: get_partition failed: unknown result
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition(ThriftHiveMetastore.java:931)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition(ThriftHiveMetastore.java:899)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:500)
at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
... 15 more
The thrift server messages are:
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table : db=default tbl=mytable
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize called
10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition : db=default tbl=mytable
RE: alter table add partition error
Posted by Pradeep Kamath <pr...@yahoo-inc.com>.
Sorry, don't know python - so couldn't make sense out of the following
error when I run the command you suggested:
[src/metastore/src/gen-py/hive_metastore]./ThriftHiveMetastore-remote -h
localhost:9080 get_partition_by_name default dummy
datestamp=20100602/srcid=100/action=view/testid=10
Traceback (most recent call last):
File "./ThriftHiveMetastore-remote", line 11, in ?
from thrift.transport import TTransport
ImportError: No module named thrift.transport
Thanks,
Pradeep
________________________________
From: Paul Yang [mailto:pyang@facebook.com]
Sent: Friday, June 18, 2010 12:10 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Hi Pradeep,
In trunk/metastore/src/gen-py/hive_metastore/, there is a script called
ThriftHiveMetastore-remote that can be used to test out the thrift
server independent of the CLI. As a quick test to narrow down the
problem, after the partition is created, can you try running
ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name
default dummy datestamp=20100602/srcid=100/action=view/testid=10
Cheers,
Paul
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Friday, June 18, 2010 11:26 AM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Hi Ning,
I have pasted below [1] the values for HADOOP_CLASSPATH and CLASSPATH
from an "env" command printed just before the following line in
execHiveCmd.sh
exec $HADOOP jar ${HIVE_LIB}/hive-cli-*.jar $CLASS $HIVE_OPTS "$@"
There is only one occurrence of libthrift.jar whose cksum matches the
one from lib/libthrift.jar from a hive checkout build.
The hive_config_without_thrift is a conf directory containing a
hive-site.xml with jdo settings to connect to the database directly. By
default I have hive set up to use a hive-site.xml [2] which connects to
a thrift server running on localhost.
Any pointers appreciated!
Thanks,
Pradeep
[1]
HADOOP_CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_c
lient/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.
1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home
/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_cl
ient/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/c
ommons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.j
ar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/lib
exec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_
client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucl
eus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus
-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1
.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home
/y/libexec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hba
se-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/hom
e/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive
_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-comm
on-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/li
bexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/li
b/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.
jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/
hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib
/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0
.jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexe
c/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/
lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/li
bexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/li
bfb303.jar:/home/y/libexec/hive_client/lib/libthrift.jar:/home/y/libexec
/hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6
.jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libe
xec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client
/lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.ja
r:
CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/l
ib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/
home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libe
xec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/li
b/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-
dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/hom
e/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hi
ve_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/
lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-con
nectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1
.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:
/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libe
xec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hbase-0.20
.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/lib
exec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client
/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.
0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/h
ive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_
hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/ho
me/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_cl
ient/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-s
erde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/h
ome/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_
client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jli
ne-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/h
ive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.
jar:/home/y/libexec/hive_client/lib/libthrift.jar:/home/y/libexec/hive_c
lient/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/h
ome/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hiv
e_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/ve
locity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:
[2]
cat /home/y/libexec/hive_client/conf/hive-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<!-- Hive Configuration can either be stored in this file or in the
hadoop configuration files -->
<!-- that are implied by Hadoop setup variables.
-->
<!-- Aside from Hadoop setup variables - this file is provided as a
convenience so that Hive -->
<!-- users do not have to edit hadoop configuration files (that may be
managed as a centralized -->
<!-- resource).
-->
<!-- Hive Execution Parameters -->
<property>
<name>hive.metastore.uris</name>
<value>thrift://localhost:9080</value>
<description>host and port for the thrift metastore
server</description>
</property>
<property>
<name>hive.metastore.local</name>
<value>false</value>
<description>controls whether to connect to remove metastore server or
open a new metastore server in Hive Client JVM</description>
</property>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/user/pradeepk/hive/warehouse</value>
<description>location of default database for the
warehouse</description>
</property>
</configuration>
________________________________
From: Ning Zhang [mailto:nzhang@facebook.com]
Sent: Friday, June 18, 2010 10:50 AM
To: hive-user@hadoop.apache.org
Subject: Re: alter table add partition error
Pradeep,
I ran the commands you provided and it succeeded with the expected
behavior.
One possibility is that there are multiple versions of libthrift.jar in
your CLASSPATH (hadoop & hive). Can you check in the Hadoop & Hive
CLASSPATH so that no other libthrift.jar is there? What is in
hive_config_without_thrift?
Thanks,
Ning
On Jun 18, 2010, at 10:19 AM, Pradeep Kamath wrote:
I think there are two separate issues here - I want to open a jira for
the first one since I am now able to reproduce it even with text format
with builtin Serdes. Essentially this is a bug in the thrift code (not
sure if it is in the client or server) since the same alter table
statement works fine when the hive client does not use thrift. Here are
the details:
cat create_dummy.sql
CREATE external TABLE if not exists dummy (
partition_name string
,partition_id int
)
PARTITIONED BY ( datestamp string, srcid string, action string, testid
string )
row format delimited
stored as textfile
location '/user/pradeepk/dummy';
hive -f create_dummy.sql
10/06/18 10:13:36 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006181013_184583537.txt
OK
Time taken: 0.627 seconds
hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid
= '100',action='view',testid='10') location
'/user/pradeepk/dummy/20100602/100/view/10';"
10/06/18 10:14:11 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_700722546.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException:
get_partition failed: unknown result
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
hive --config hive_conf_without_thrift -e "ALTER TABLE dummy add
partition(datestamp = '20100602', srcid =
'100',action='view',testid='10') location
'/user/pradeepk/dummy/20100602/100/view/10';"
10/06/18 10:14:31 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_598649843.txt
OK
Time taken: 5.849 seconds
Is there some thrift setting I am missing or is this a bug? - If it is
the latter, I can open a jira with the above details.
Thanks,
Pradeep
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Thursday, June 17, 2010 1:25 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Here are the create table and alter table statements:
CREATE external TABLE if not exists mytable (
bc string
,src_spaceid string
,srcpvid string
,dstpvid string
,dst_spaceid string
,page_params map<string, string>
,clickinfo map<string, string>
,viewinfo array<map<string, string>>
)
PARTITIONED BY ( datestamp string, srcid string, action string, testid
string )
row format serde 'com.yahoo.mySerde'
stored as inputformat 'org.apache.hadoop.mapred.SequenceFileInputFormat'
outputformat 'org.apache.hadoop.mapred.SequenceFileOutputFormat'
location '/user/pradeepk/mytable';
hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add
partition(datestamp = '20091101', srcid =
'19174',action='click',testid='NOTESTID') location
'/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"
I get the following error:
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException:
get_partition failed: unknown result
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
If I don't use thrift and use a hive-site.xml to directly talk to the
db, the alter table seems to succeed:
hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "ALTER
TABLE mytable add partition(datestamp = '20091101', srcid =
'19174',action='click',testid='NOTESTID') location
'/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"
However I get errors when I try to run a query:
[pradeepk@chargesize:~/dev]hive --auxpath ult-serde.jar --config
hive_conf_without_thrift -e "select src_spaceid from
ult_search_austria_ult where datestamp='20091101';"
10/06/17 13:22:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006171322_1913647383.txt
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
java.lang.IllegalArgumentException: Can not create a Path from an empty
string
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)
at org.apache.hadoop.fs.Path.<init>(Path.java:90)
at org.apache.hadoop.fs.Path.<init>(Path.java:50)
at
org.apache.hadoop.mapred.JobClient.copyRemoteFiles(JobClient.java:523)
at
org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient
.java:603)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)
at
org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
at
org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:684)
at
org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
Any help is much appreciated.
Pradeep
________________________________
From: Ashish Thusoo [mailto:athusoo@facebook.com]
Sent: Thursday, June 17, 2010 11:15 AM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
hmm... Can you send the exact command and also the create table command
for this table.
Ashish
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Thursday, June 17, 2010 9:09 AM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Sorry - that was a cut-paste error - I don't have the action part - so I
am specifying key-value pairs. Since what I am trying to do seems like a
basic operation, I am wondering if it's something to do with my Serde -
unfortunately the error I see gives me no clue of what could be wrong -
any help would be greatly appreciated!
Thanks,
Pradeep
________________________________
From: yq he [mailto:hhh.heyq@gmail.com]
Sent: Wednesday, June 16, 2010 5:54 PM
To: hive-user@hadoop.apache.org
Subject: Re: alter table add partition error
Hi Pradeep,
partition definition need to be key-value pairs. partition key `action`
seems missed the value.
Thanks
Yongqiang
On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath <pr...@yahoo-inc.com>
wrote:
Hi,
I am trying to create an external table against already existing
data in sequencefile format. However I have written a custom Serde to
interpret the data. I am able to create the table fine but get the
exception shown in the session output below when I try to add partition
- any help would be greatly appreciated.
Thanks,
Pradeep
== session output ===
[pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER
TABLE mytable add partition(datestamp = '20091101', srcid = '10',action)
location '/user/pradeepk/mytable/20091101/10';"
10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException:
get_partition failed: unknown result
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
[pradeepk@chargesize:~/dev/howl]
== session output ===
/tmp/pradeepk/hive.log has:
2010-06-16 17:09:00,841 ERROR exec.DDLTask
(SessionState.java:printError(269)) - FAILED: Error in metadata:
org.apache.thrift.TApplicationException: get_partition failed: unknown
result
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.thrift.TApplicationException: get_partition failed: unknown
result
at
org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
at
org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
at
org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: org.apache.thrift.TApplicationException: get_partition
failed: unknown result
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get
_partition(ThriftHiveMetastore.java:931)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_part
ition(ThriftHiveMetastore.java:899)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMe
taStoreClient.java:500)
at
org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
... 15 more
The thrift server messages are:
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table :
db=default tbl=mytable
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store
with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize
called
10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition :
db=default tbl=mytable
RE: alter table add partition error
Posted by Paul Yang <py...@facebook.com>.
Hi Pradeep,
In trunk/metastore/src/gen-py/hive_metastore/, there is a script called ThriftHiveMetastore-remote that can be used to test out the thrift server independent of the CLI. As a quick test to narrow down the problem, after the partition is created, can you try running
ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name default dummy datestamp=20100602/srcid=100/action=view/testid=10
Cheers,
Paul
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Friday, June 18, 2010 11:26 AM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Hi Ning,
I have pasted below [1] the values for HADOOP_CLASSPATH and CLASSPATH from an "env" command printed just before the following line in execHiveCmd.sh
exec $HADOOP jar ${HIVE_LIB}/hive-cli-*.jar $CLASS $HIVE_OPTS "$@"
There is only one occurrence of libthrift.jar whose cksum matches the one from lib/libthrift.jar from a hive checkout build.
The hive_config_without_thrift is a conf directory containing a hive-site.xml with jdo settings to connect to the database directly. By default I have hive set up to use a hive-site.xml [2] which connects to a thrift server running on localhost.
Any pointers appreciated!
Thanks,
Pradeep
[1]
HADOOP_CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libexec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.jar:/home/y/libexec/hive_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:
CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libexec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.jar:/home/y/libexec/hive_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:
[2]
cat /home/y/libexec/hive_client/conf/hive-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<!-- Hive Configuration can either be stored in this file or in the hadoop configuration files -->
<!-- that are implied by Hadoop setup variables. -->
<!-- Aside from Hadoop setup variables - this file is provided as a convenience so that Hive -->
<!-- users do not have to edit hadoop configuration files (that may be managed as a centralized -->
<!-- resource). -->
<!-- Hive Execution Parameters -->
<property>
<name>hive.metastore.uris</name>
<value>thrift://localhost:9080</value>
<description>host and port for the thrift metastore server</description>
</property>
<property>
<name>hive.metastore.local</name>
<value>false</value>
<description>controls whether to connect to remove metastore server or open a new metastore server in Hive Client JVM</description>
</property>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/user/pradeepk/hive/warehouse</value>
<description>location of default database for the warehouse</description>
</property>
</configuration>
________________________________
From: Ning Zhang [mailto:nzhang@facebook.com]
Sent: Friday, June 18, 2010 10:50 AM
To: hive-user@hadoop.apache.org
Subject: Re: alter table add partition error
Pradeep,
I ran the commands you provided and it succeeded with the expected behavior.
One possibility is that there are multiple versions of libthrift.jar in your CLASSPATH (hadoop & hive). Can you check in the Hadoop & Hive CLASSPATH so that no other libthrift.jar is there? What is in hive_config_without_thrift?
Thanks,
Ning
On Jun 18, 2010, at 10:19 AM, Pradeep Kamath wrote:
I think there are two separate issues here - I want to open a jira for the first one since I am now able to reproduce it even with text format with builtin Serdes. Essentially this is a bug in the thrift code (not sure if it is in the client or server) since the same alter table statement works fine when the hive client does not use thrift. Here are the details:
cat create_dummy.sql
CREATE external TABLE if not exists dummy (
partition_name string
,partition_id int
)
PARTITIONED BY ( datestamp string, srcid string, action string, testid string )
row format delimited
stored as textfile
location '/user/pradeepk/dummy';
hive -f create_dummy.sql
10/06/18 10:13:36 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181013_184583537.txt
OK
Time taken: 0.627 seconds
hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid = '100',action='view',testid='10') location '/user/pradeepk/dummy/20100602/100/view/10';"
10/06/18 10:14:11 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_700722546.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
hive --config hive_conf_without_thrift -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid = '100',action='view',testid='10') location '/user/pradeepk/dummy/20100602/100/view/10';"
10/06/18 10:14:31 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_598649843.txt
OK
Time taken: 5.849 seconds
Is there some thrift setting I am missing or is this a bug? - If it is the latter, I can open a jira with the above details.
Thanks,
Pradeep
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Thursday, June 17, 2010 1:25 PM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: alter table add partition error
Here are the create table and alter table statements:
CREATE external TABLE if not exists mytable (
bc string
,src_spaceid string
,srcpvid string
,dstpvid string
,dst_spaceid string
,page_params map<string, string>
,clickinfo map<string, string>
,viewinfo array<map<string, string>>
)
PARTITIONED BY ( datestamp string, srcid string, action string, testid string )
row format serde 'com.yahoo.mySerde'
stored as inputformat 'org.apache.hadoop.mapred.SequenceFileInputFormat' outputformat 'org.apache.hadoop.mapred.SequenceFileOutputFormat'
location '/user/pradeepk/mytable';
hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '19174',action='click',testid='NOTESTID') location '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"
I get the following error:
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
If I don't use thrift and use a hive-site.xml to directly talk to the db, the alter table seems to succeed:
hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '19174',action='click',testid='NOTESTID') location '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"
However I get errors when I try to run a query:
[pradeepk@chargesize:~/dev]hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "select src_spaceid from ult_search_austria_ult where datestamp='20091101';"
10/06/17 13:22:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006171322_1913647383.txt
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
java.lang.IllegalArgumentException: Can not create a Path from an empty string
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)
at org.apache.hadoop.fs.Path.<init>(Path.java:90)
at org.apache.hadoop.fs.Path.<init>(Path.java:50)
at org.apache.hadoop.mapred.JobClient.copyRemoteFiles(JobClient.java:523)
at org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient.java:603)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:684)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
Any help is much appreciated.
Pradeep
________________________________
From: Ashish Thusoo [mailto:athusoo@facebook.com]
Sent: Thursday, June 17, 2010 11:15 AM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: alter table add partition error
hmm... Can you send the exact command and also the create table command for this table.
Ashish
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Thursday, June 17, 2010 9:09 AM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: alter table add partition error
Sorry - that was a cut-paste error - I don't have the action part - so I am specifying key-value pairs. Since what I am trying to do seems like a basic operation, I am wondering if it's something to do with my Serde - unfortunately the error I see gives me no clue of what could be wrong - any help would be greatly appreciated!
Thanks,
Pradeep
________________________________
From: yq he [mailto:hhh.heyq@gmail.com]
Sent: Wednesday, June 16, 2010 5:54 PM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: alter table add partition error
Hi Pradeep,
partition definition need to be key-value pairs. partition key `action` seems missed the value.
Thanks
Yongqiang
On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath <pr...@yahoo-inc.com>> wrote:
Hi,
I am trying to create an external table against already existing data in sequencefile format. However I have written a custom Serde to interpret the data. I am able to create the table fine but get the exception shown in the session output below when I try to add partition - any help would be greatly appreciated.
Thanks,
Pradeep
== session output ===
[pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '10',action) location '/user/pradeepk/mytable/20091101/10';"
10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
[pradeepk@chargesize:~/dev/howl]
== session output ===
/tmp/pradeepk/hive.log has:
2010-06-16 17:09:00,841 ERROR exec.DDLTask (SessionState.java:printError(269)) - FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.TApplicationException: get_partition failed: unknown result
at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
at org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: org.apache.thrift.TApplicationException: get_partition failed: unknown result
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition(ThriftHiveMetastore.java:931)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition(ThriftHiveMetastore.java:899)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:500)
at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
... 15 more
The thrift server messages are:
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table : db=default tbl=mytable
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize called
10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition : db=default tbl=mytable
RE: alter table add partition error
Posted by Pradeep Kamath <pr...@yahoo-inc.com>.
Yes - exactly - the create table works fine - I have been able to even
create external tables and run select queries on them with my current
thrift setup. I am using Oracle as a backend DB.
________________________________
From: Paul Yang [mailto:pyang@facebook.com]
Sent: Friday, June 18, 2010 12:00 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
If the JDO connectionURL wasn't set correctly on the Thrift server, then
the create table should fail as well. The problem is that the create
table succeeds, but the alter table fails, right?
From: Ning Zhang [mailto:nzhang@facebook.com]
Sent: Friday, June 18, 2010 11:52 AM
To: hive-user@hadoop.apache.org
Subject: Re: alter table add partition error
We also use thrift for hive.metastore.uris. Then at the thrift server
side, we use JDBC to mysql (javax.jdo.option.ConnectionURL). Can you
check your thrift server side to see the JDO connectionURL is set
correctly?
On Jun 18, 2010, at 11:25 AM, Pradeep Kamath wrote:
Hi Ning,
I have pasted below [1] the values for HADOOP_CLASSPATH and CLASSPATH
from an "env" command printed just before the following line in
execHiveCmd.sh
exec $HADOOP jar ${HIVE_LIB}/hive-cli-*.jar $CLASS $HIVE_OPTS "$@"
There is only one occurrence of libthrift.jar whose cksum matches the
one from lib/libthrift.jar from a hive checkout build.
The hive_config_without_thrift is a conf directory containing a
hive-site.xml with jdo settings to connect to the database directly. By
default I have hive set up to use a hive-site.xml [2] which connects to
a thrift server running on localhost.
Any pointers appreciated!
Thanks,
Pradeep
[1]
HADOOP_CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_c
lient/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.
1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home
/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_cl
ient/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/c
ommons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.j
ar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/lib
exec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_
client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucl
eus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus
-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1
.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home
/y/libexec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hba
se-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/hom
e/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive
_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-comm
on-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/li
bexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/li
b/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.
jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/
hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib
/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0
.jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexe
c/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/
lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/li
bexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/li
bfb303.jar:/home/y/libexec/hive_client/lib/libthrift.jar:/home/y/libexec
/hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6
.jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libe
xec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client
/lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.ja
r:
CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/l
ib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/
home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libe
xec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/li
b/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-
dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/hom
e/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hi
ve_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/
lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-con
nectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1
.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:
/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libe
xec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hbase-0.20
.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/lib
exec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client
/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.
0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/h
ive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_
hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/ho
me/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_cl
ient/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-s
erde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/h
ome/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_
client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jli
ne-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/h
ive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.
jar:/home/y/libexec/hive_client/lib/libthrift.jar:/home/y/libexec/hive_c
lient/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/h
ome/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hiv
e_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/ve
locity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:
[2]
cat /home/y/libexec/hive_client/conf/hive-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<!-- Hive Configuration can either be stored in this file or in the
hadoop configuration files -->
<!-- that are implied by Hadoop setup variables.
-->
<!-- Aside from Hadoop setup variables - this file is provided as a
convenience so that Hive -->
<!-- users do not have to edit hadoop configuration files (that may be
managed as a centralized -->
<!-- resource).
-->
<!-- Hive Execution Parameters -->
<property>
<name>hive.metastore.uris</name>
<value>thrift://localhost:9080</value>
<description>host and port for the thrift metastore
server</description>
</property>
<property>
<name>hive.metastore.local</name>
<value>false</value>
<description>controls whether to connect to remove metastore server or
open a new metastore server in Hive Client JVM</description>
</property>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/user/pradeepk/hive/warehouse</value>
<description>location of default database for the
warehouse</description>
</property>
</configuration>
________________________________
From: Ning Zhang [mailto:nzhang@facebook.com]
Sent: Friday, June 18, 2010 10:50 AM
To: hive-user@hadoop.apache.org
Subject: Re: alter table add partition error
Pradeep,
I ran the commands you provided and it succeeded with the expected
behavior.
One possibility is that there are multiple versions of libthrift.jar in
your CLASSPATH (hadoop & hive). Can you check in the Hadoop & Hive
CLASSPATH so that no other libthrift.jar is there? What is in
hive_config_without_thrift?
Thanks,
Ning
On Jun 18, 2010, at 10:19 AM, Pradeep Kamath wrote:
I think there are two separate issues here - I want to open a jira for
the first one since I am now able to reproduce it even with text format
with builtin Serdes. Essentially this is a bug in the thrift code (not
sure if it is in the client or server) since the same alter table
statement works fine when the hive client does not use thrift. Here are
the details:
cat create_dummy.sql
CREATE external TABLE if not exists dummy (
partition_name string
,partition_id int
)
PARTITIONED BY ( datestamp string, srcid string, action string, testid
string )
row format delimited
stored as textfile
location '/user/pradeepk/dummy';
hive -f create_dummy.sql
10/06/18 10:13:36 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006181013_184583537.txt
OK
Time taken: 0.627 seconds
hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid
= '100',action='view',testid='10') location
'/user/pradeepk/dummy/20100602/100/view/10';"
10/06/18 10:14:11 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_700722546.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException:
get_partition failed: unknown result
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
hive --config hive_conf_without_thrift -e "ALTER TABLE dummy add
partition(datestamp = '20100602', srcid =
'100',action='view',testid='10') location
'/user/pradeepk/dummy/20100602/100/view/10';"
10/06/18 10:14:31 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_598649843.txt
OK
Time taken: 5.849 seconds
Is there some thrift setting I am missing or is this a bug? - If it is
the latter, I can open a jira with the above details.
Thanks,
Pradeep
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Thursday, June 17, 2010 1:25 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Here are the create table and alter table statements:
CREATE external TABLE if not exists mytable (
bc string
,src_spaceid string
,srcpvid string
,dstpvid string
,dst_spaceid string
,page_params map<string, string>
,clickinfo map<string, string>
,viewinfo array<map<string, string>>
)
PARTITIONED BY ( datestamp string, srcid string, action string, testid
string )
row format serde 'com.yahoo.mySerde'
stored as inputformat 'org.apache.hadoop.mapred.SequenceFileInputFormat'
outputformat 'org.apache.hadoop.mapred.SequenceFileOutputFormat'
location '/user/pradeepk/mytable';
hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add
partition(datestamp = '20091101', srcid =
'19174',action='click',testid='NOTESTID') location
'/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"
I get the following error:
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException:
get_partition failed: unknown result
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
If I don't use thrift and use a hive-site.xml to directly talk to the
db, the alter table seems to succeed:
hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "ALTER
TABLE mytable add partition(datestamp = '20091101', srcid =
'19174',action='click',testid='NOTESTID') location
'/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"
However I get errors when I try to run a query:
[pradeepk@chargesize:~/dev]hive --auxpath ult-serde.jar --config
hive_conf_without_thrift -e "select src_spaceid from
ult_search_austria_ult where datestamp='20091101';"
10/06/17 13:22:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006171322_1913647383.txt
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
java.lang.IllegalArgumentException: Can not create a Path from an empty
string
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)
at org.apache.hadoop.fs.Path.<init>(Path.java:90)
at org.apache.hadoop.fs.Path.<init>(Path.java:50)
at
org.apache.hadoop.mapred.JobClient.copyRemoteFiles(JobClient.java:523)
at
org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient
.java:603)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)
at
org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
at
org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:684)
at
org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
Any help is much appreciated.
Pradeep
________________________________
From: Ashish Thusoo [mailto:athusoo@facebook.com]
Sent: Thursday, June 17, 2010 11:15 AM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
hmm... Can you send the exact command and also the create table command
for this table.
Ashish
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Thursday, June 17, 2010 9:09 AM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Sorry - that was a cut-paste error - I don't have the action part - so I
am specifying key-value pairs. Since what I am trying to do seems like a
basic operation, I am wondering if it's something to do with my Serde -
unfortunately the error I see gives me no clue of what could be wrong -
any help would be greatly appreciated!
Thanks,
Pradeep
________________________________
From: yq he [mailto:hhh.heyq@gmail.com]
Sent: Wednesday, June 16, 2010 5:54 PM
To: hive-user@hadoop.apache.org
Subject: Re: alter table add partition error
Hi Pradeep,
partition definition need to be key-value pairs. partition key `action`
seems missed the value.
Thanks
Yongqiang
On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath <pr...@yahoo-inc.com>
wrote:
Hi,
I am trying to create an external table against already existing
data in sequencefile format. However I have written a custom Serde to
interpret the data. I am able to create the table fine but get the
exception shown in the session output below when I try to add partition
- any help would be greatly appreciated.
Thanks,
Pradeep
== session output ===
[pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER
TABLE mytable add partition(datestamp = '20091101', srcid = '10',action)
location '/user/pradeepk/mytable/20091101/10';"
10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException:
get_partition failed: unknown result
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
[pradeepk@chargesize:~/dev/howl]
== session output ===
/tmp/pradeepk/hive.log has:
2010-06-16 17:09:00,841 ERROR exec.DDLTask
(SessionState.java:printError(269)) - FAILED: Error in metadata:
org.apache.thrift.TApplicationException: get_partition failed: unknown
result
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.thrift.TApplicationException: get_partition failed: unknown
result
at
org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
at
org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
at
org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: org.apache.thrift.TApplicationException: get_partition
failed: unknown result
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get
_partition(ThriftHiveMetastore.java:931)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_part
ition(ThriftHiveMetastore.java:899)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMe
taStoreClient.java:500)
at
org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
... 15 more
The thrift server messages are:
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table :
db=default tbl=mytable
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store
with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize
called
10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition :
db=default tbl=mytable
RE: alter table add partition error
Posted by Paul Yang <py...@facebook.com>.
If the JDO connectionURL wasn't set correctly on the Thrift server, then the create table should fail as well. The problem is that the create table succeeds, but the alter table fails, right?
From: Ning Zhang [mailto:nzhang@facebook.com]
Sent: Friday, June 18, 2010 11:52 AM
To: hive-user@hadoop.apache.org
Subject: Re: alter table add partition error
We also use thrift for hive.metastore.uris. Then at the thrift server side, we use JDBC to mysql (javax.jdo.option.ConnectionURL). Can you check your thrift server side to see the JDO connectionURL is set correctly?
On Jun 18, 2010, at 11:25 AM, Pradeep Kamath wrote:
Hi Ning,
I have pasted below [1] the values for HADOOP_CLASSPATH and CLASSPATH from an "env" command printed just before the following line in execHiveCmd.sh
exec $HADOOP jar ${HIVE_LIB}/hive-cli-*.jar $CLASS $HIVE_OPTS "$@"
There is only one occurrence of libthrift.jar whose cksum matches the one from lib/libthrift.jar from a hive checkout build.
The hive_config_without_thrift is a conf directory containing a hive-site.xml with jdo settings to connect to the database directly. By default I have hive set up to use a hive-site.xml [2] which connects to a thrift server running on localhost.
Any pointers appreciated!
Thanks,
Pradeep
[1]
HADOOP_CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libexec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.jar:/home/y/libexec/hive_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:
CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libexec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.jar:/home/y/libexec/hive_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:
[2]
cat /home/y/libexec/hive_client/conf/hive-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<!-- Hive Configuration can either be stored in this file or in the hadoop configuration files -->
<!-- that are implied by Hadoop setup variables. -->
<!-- Aside from Hadoop setup variables - this file is provided as a convenience so that Hive -->
<!-- users do not have to edit hadoop configuration files (that may be managed as a centralized -->
<!-- resource). -->
<!-- Hive Execution Parameters -->
<property>
<name>hive.metastore.uris</name>
<value>thrift://localhost:9080</value>
<description>host and port for the thrift metastore server</description>
</property>
<property>
<name>hive.metastore.local</name>
<value>false</value>
<description>controls whether to connect to remove metastore server or open a new metastore server in Hive Client JVM</description>
</property>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/user/pradeepk/hive/warehouse</value>
<description>location of default database for the warehouse</description>
</property>
</configuration>
________________________________
From: Ning Zhang [mailto:nzhang@facebook.com]
Sent: Friday, June 18, 2010 10:50 AM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: alter table add partition error
Pradeep,
I ran the commands you provided and it succeeded with the expected behavior.
One possibility is that there are multiple versions of libthrift.jar in your CLASSPATH (hadoop & hive). Can you check in the Hadoop & Hive CLASSPATH so that no other libthrift.jar is there? What is in hive_config_without_thrift?
Thanks,
Ning
On Jun 18, 2010, at 10:19 AM, Pradeep Kamath wrote:
I think there are two separate issues here - I want to open a jira for the first one since I am now able to reproduce it even with text format with builtin Serdes. Essentially this is a bug in the thrift code (not sure if it is in the client or server) since the same alter table statement works fine when the hive client does not use thrift. Here are the details:
cat create_dummy.sql
CREATE external TABLE if not exists dummy (
partition_name string
,partition_id int
)
PARTITIONED BY ( datestamp string, srcid string, action string, testid string )
row format delimited
stored as textfile
location '/user/pradeepk/dummy';
hive -f create_dummy.sql
10/06/18 10:13:36 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181013_184583537.txt
OK
Time taken: 0.627 seconds
hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid = '100',action='view',testid='10') location '/user/pradeepk/dummy/20100602/100/view/10';"
10/06/18 10:14:11 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_700722546.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
hive --config hive_conf_without_thrift -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid = '100',action='view',testid='10') location '/user/pradeepk/dummy/20100602/100/view/10';"
10/06/18 10:14:31 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_598649843.txt
OK
Time taken: 5.849 seconds
Is there some thrift setting I am missing or is this a bug? - If it is the latter, I can open a jira with the above details.
Thanks,
Pradeep
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Thursday, June 17, 2010 1:25 PM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: alter table add partition error
Here are the create table and alter table statements:
CREATE external TABLE if not exists mytable (
bc string
,src_spaceid string
,srcpvid string
,dstpvid string
,dst_spaceid string
,page_params map<string, string>
,clickinfo map<string, string>
,viewinfo array<map<string, string>>
)
PARTITIONED BY ( datestamp string, srcid string, action string, testid string )
row format serde 'com.yahoo.mySerde'
stored as inputformat 'org.apache.hadoop.mapred.SequenceFileInputFormat' outputformat 'org.apache.hadoop.mapred.SequenceFileOutputFormat'
location '/user/pradeepk/mytable';
hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '19174',action='click',testid='NOTESTID') location '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"
I get the following error:
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
If I don't use thrift and use a hive-site.xml to directly talk to the db, the alter table seems to succeed:
hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '19174',action='click',testid='NOTESTID') location '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"
However I get errors when I try to run a query:
[pradeepk@chargesize:~/dev]hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "select src_spaceid from ult_search_austria_ult where datestamp='20091101';"
10/06/17 13:22:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006171322_1913647383.txt
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
java.lang.IllegalArgumentException: Can not create a Path from an empty string
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)
at org.apache.hadoop.fs.Path.<init>(Path.java:90)
at org.apache.hadoop.fs.Path.<init>(Path.java:50)
at org.apache.hadoop.mapred.JobClient.copyRemoteFiles(JobClient.java:523)
at org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient.java:603)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:684)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
Any help is much appreciated.
Pradeep
________________________________
From: Ashish Thusoo [mailto:athusoo@facebook.com]
Sent: Thursday, June 17, 2010 11:15 AM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: alter table add partition error
hmm... Can you send the exact command and also the create table command for this table.
Ashish
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Thursday, June 17, 2010 9:09 AM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: alter table add partition error
Sorry - that was a cut-paste error - I don't have the action part - so I am specifying key-value pairs. Since what I am trying to do seems like a basic operation, I am wondering if it's something to do with my Serde - unfortunately the error I see gives me no clue of what could be wrong - any help would be greatly appreciated!
Thanks,
Pradeep
________________________________
From: yq he [mailto:hhh.heyq@gmail.com]
Sent: Wednesday, June 16, 2010 5:54 PM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: alter table add partition error
Hi Pradeep,
partition definition need to be key-value pairs. partition key `action` seems missed the value.
Thanks
Yongqiang
On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath <pr...@yahoo-inc.com>> wrote:
Hi,
I am trying to create an external table against already existing data in sequencefile format. However I have written a custom Serde to interpret the data. I am able to create the table fine but get the exception shown in the session output below when I try to add partition - any help would be greatly appreciated.
Thanks,
Pradeep
== session output ===
[pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '10',action) location '/user/pradeepk/mytable/20091101/10';"
10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
[pradeepk@chargesize:~/dev/howl]
== session output ===
/tmp/pradeepk/hive.log has:
2010-06-16 17:09:00,841 ERROR exec.DDLTask (SessionState.java:printError(269)) - FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.TApplicationException: get_partition failed: unknown result
at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
at org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: org.apache.thrift.TApplicationException: get_partition failed: unknown result
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition(ThriftHiveMetastore.java:931)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition(ThriftHiveMetastore.java:899)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:500)
at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
... 15 more
The thrift server messages are:
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table : db=default tbl=mytable
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize called
10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition : db=default tbl=mytable
Re: alter table add partition error
Posted by Ning Zhang <nz...@facebook.com>.
We also use thrift for hive.metastore.uris. Then at the thrift server side, we use JDBC to mysql (javax.jdo.option.ConnectionURL). Can you check your thrift server side to see the JDO connectionURL is set correctly?
On Jun 18, 2010, at 11:25 AM, Pradeep Kamath wrote:
Hi Ning,
I have pasted below [1] the values for HADOOP_CLASSPATH and CLASSPATH from an “env” command printed just before the following line in execHiveCmd.sh
exec $HADOOP jar ${HIVE_LIB}/hive-cli-*.jar $CLASS $HIVE_OPTS "$@"
There is only one occurrence of libthrift.jar whose cksum matches the one from lib/libthrift.jar from a hive checkout build.
The hive_config_without_thrift is a conf directory containing a hive-site.xml with jdo settings to connect to the database directly. By default I have hive set up to use a hive-site.xml [2] which connects to a thrift server running on localhost.
Any pointers appreciated!
Thanks,
Pradeep
[1]
HADOOP_CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libexec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.jar:/home/y/libexec/hive_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:
CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libexec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.jar:/home/y/libexec/hive_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:
[2]
cat /home/y/libexec/hive_client/conf/hive-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<!-- Hive Configuration can either be stored in this file or in the hadoop configuration files -->
<!-- that are implied by Hadoop setup variables. -->
<!-- Aside from Hadoop setup variables - this file is provided as a convenience so that Hive -->
<!-- users do not have to edit hadoop configuration files (that may be managed as a centralized -->
<!-- resource). -->
<!-- Hive Execution Parameters -->
<property>
<name>hive.metastore.uris</name>
<value>thrift://localhost:9080</value>
<description>host and port for the thrift metastore server</description>
</property>
<property>
<name>hive.metastore.local</name>
<value>false</value>
<description>controls whether to connect to remove metastore server or open a new metastore server in Hive Client JVM</description>
</property>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/user/pradeepk/hive/warehouse</value>
<description>location of default database for the warehouse</description>
</property>
</configuration>
________________________________
From: Ning Zhang [mailto:nzhang@facebook.com]
Sent: Friday, June 18, 2010 10:50 AM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: alter table add partition error
Pradeep,
I ran the commands you provided and it succeeded with the expected behavior.
One possibility is that there are multiple versions of libthrift.jar in your CLASSPATH (hadoop & hive). Can you check in the Hadoop & Hive CLASSPATH so that no other libthrift.jar is there? What is in hive_config_without_thrift?
Thanks,
Ning
On Jun 18, 2010, at 10:19 AM, Pradeep Kamath wrote:
I think there are two separate issues here – I want to open a jira for the first one since I am now able to reproduce it even with text format with builtin Serdes. Essentially this is a bug in the thrift code (not sure if it is in the client or server) since the same alter table statement works fine when the hive client does not use thrift. Here are the details:
cat create_dummy.sql
CREATE external TABLE if not exists dummy (
partition_name string
,partition_id int
)
PARTITIONED BY ( datestamp string, srcid string, action string, testid string )
row format delimited
stored as textfile
location '/user/pradeepk/dummy';
hive -f create_dummy.sql
10/06/18 10:13:36 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181013_184583537.txt
OK
Time taken: 0.627 seconds
hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid = '100',action='view',testid='10') location '/user/pradeepk/dummy/20100602/100/view/10';"
10/06/18 10:14:11 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_700722546.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
hive --config hive_conf_without_thrift -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid = '100',action='view',testid='10') location '/user/pradeepk/dummy/20100602/100/view/10';"
10/06/18 10:14:31 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_598649843.txt
OK
Time taken: 5.849 seconds
Is there some thrift setting I am missing or is this a bug? – If it is the latter, I can open a jira with the above details.
Thanks,
Pradeep
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Thursday, June 17, 2010 1:25 PM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: alter table add partition error
Here are the create table and alter table statements:
CREATE external TABLE if not exists mytable (
bc string
,src_spaceid string
,srcpvid string
,dstpvid string
,dst_spaceid string
,page_params map<string, string>
,clickinfo map<string, string>
,viewinfo array<map<string, string>>
)
PARTITIONED BY ( datestamp string, srcid string, action string, testid string )
row format serde 'com.yahoo.mySerde’
stored as inputformat 'org.apache.hadoop.mapred.SequenceFileInputFormat' outputformat 'org.apache.hadoop.mapred.SequenceFileOutputFormat'
location '/user/pradeepk/mytable’;
hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '19174',action='click',testid='NOTESTID') location '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"
I get the following error:
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
If I don’t use thrift and use a hive-site.xml to directly talk to the db, the alter table seems to succeed:
hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '19174',action='click',testid='NOTESTID') location '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"
However I get errors when I try to run a query:
[pradeepk@chargesize:~/dev]hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "select src_spaceid from ult_search_austria_ult where datestamp='20091101';"
10/06/17 13:22:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006171322_1913647383.txt
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
java.lang.IllegalArgumentException: Can not create a Path from an empty string
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)
at org.apache.hadoop.fs.Path.<init>(Path.java:90)
at org.apache.hadoop.fs.Path.<init>(Path.java:50)
at org.apache.hadoop.mapred.JobClient.copyRemoteFiles(JobClient.java:523)
at org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient.java:603)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:684)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
Any help is much appreciated.
Pradeep
________________________________
From: Ashish Thusoo [mailto:athusoo@facebook.com]
Sent: Thursday, June 17, 2010 11:15 AM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: alter table add partition error
hmm... Can you send the exact command and also the create table command for this table.
Ashish
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Thursday, June 17, 2010 9:09 AM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: alter table add partition error
Sorry – that was a cut-paste error – I don’t have the action part – so I am specifying key-value pairs. Since what I am trying to do seems like a basic operation, I am wondering if it’s something to do with my Serde – unfortunately the error I see gives me no clue of what could be wrong – any help would be greatly appreciated!
Thanks,
Pradeep
________________________________
From: yq he [mailto:hhh.heyq@gmail.com]
Sent: Wednesday, June 16, 2010 5:54 PM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: alter table add partition error
Hi Pradeep,
partition definition need to be key-value pairs. partition key `action` seems missed the value.
Thanks
Yongqiang
On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath <pr...@yahoo-inc.com>> wrote:
Hi,
I am trying to create an external table against already existing data in sequencefile format. However I have written a custom Serde to interpret the data. I am able to create the table fine but get the exception shown in the session output below when I try to add partition – any help would be greatly appreciated.
Thanks,
Pradeep
== session output ===
[pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = ‘10’,action) location '/user/pradeepk/mytable/20091101/10’;"
10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
[pradeepk@chargesize:~/dev/howl]
== session output ===
/tmp/pradeepk/hive.log has:
2010-06-16 17:09:00,841 ERROR exec.DDLTask (SessionState.java:printError(269)) - FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.TApplicationException: get_partition failed: unknown result
at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
at org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: org.apache.thrift.TApplicationException: get_partition failed: unknown result
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition(ThriftHiveMetastore.java:931)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition(ThriftHiveMetastore.java:899)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:500)
at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
... 15 more
The thrift server messages are:
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table : db=default tbl=mytable
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize called
10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition : db=default tbl=mytable
RE: alter table add partition error
Posted by Pradeep Kamath <pr...@yahoo-inc.com>.
Hi Ning,
I have pasted below [1] the values for HADOOP_CLASSPATH and CLASSPATH
from an "env" command printed just before the following line in
execHiveCmd.sh
exec $HADOOP jar ${HIVE_LIB}/hive-cli-*.jar $CLASS $HIVE_OPTS "$@"
There is only one occurrence of libthrift.jar whose cksum matches the
one from lib/libthrift.jar from a hive checkout build.
The hive_config_without_thrift is a conf directory containing a
hive-site.xml with jdo settings to connect to the database directly. By
default I have hive set up to use a hive-site.xml [2] which connects to
a thrift server running on localhost.
Any pointers appreciated!
Thanks,
Pradeep
[1]
HADOOP_CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_c
lient/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.
1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home
/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_cl
ient/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/c
ommons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.j
ar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/lib
exec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_
client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucl
eus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus
-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1
.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home
/y/libexec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hba
se-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/hom
e/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive
_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-comm
on-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/li
bexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/li
b/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.
jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/
hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib
/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0
.jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexe
c/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/
lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/li
bexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/li
bfb303.jar:/home/y/libexec/hive_client/lib/libthrift.jar:/home/y/libexec
/hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6
.jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libe
xec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client
/lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.ja
r:
CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/l
ib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/
home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libe
xec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/li
b/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-
dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/hom
e/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hi
ve_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/
lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-con
nectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1
.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:
/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libe
xec/hive_client/lib/derby.jar:/home/y/libexec/hive_client/lib/hbase-0.20
.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/lib
exec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client
/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.
0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/h
ive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_
hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/ho
me/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_cl
ient/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-s
erde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/h
ome/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_
client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jli
ne-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/h
ive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.
jar:/home/y/libexec/hive_client/lib/libthrift.jar:/home/y/libexec/hive_c
lient/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/h
ome/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hiv
e_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/ve
locity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:
[2]
cat /home/y/libexec/hive_client/conf/hive-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<!-- Hive Configuration can either be stored in this file or in the
hadoop configuration files -->
<!-- that are implied by Hadoop setup variables.
-->
<!-- Aside from Hadoop setup variables - this file is provided as a
convenience so that Hive -->
<!-- users do not have to edit hadoop configuration files (that may be
managed as a centralized -->
<!-- resource).
-->
<!-- Hive Execution Parameters -->
<property>
<name>hive.metastore.uris</name>
<value>thrift://localhost:9080</value>
<description>host and port for the thrift metastore
server</description>
</property>
<property>
<name>hive.metastore.local</name>
<value>false</value>
<description>controls whether to connect to remove metastore server or
open a new metastore server in Hive Client JVM</description>
</property>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/user/pradeepk/hive/warehouse</value>
<description>location of default database for the
warehouse</description>
</property>
</configuration>
________________________________
From: Ning Zhang [mailto:nzhang@facebook.com]
Sent: Friday, June 18, 2010 10:50 AM
To: hive-user@hadoop.apache.org
Subject: Re: alter table add partition error
Pradeep,
I ran the commands you provided and it succeeded with the expected
behavior.
One possibility is that there are multiple versions of libthrift.jar in
your CLASSPATH (hadoop & hive). Can you check in the Hadoop & Hive
CLASSPATH so that no other libthrift.jar is there? What is in
hive_config_without_thrift?
Thanks,
Ning
On Jun 18, 2010, at 10:19 AM, Pradeep Kamath wrote:
I think there are two separate issues here - I want to open a jira for
the first one since I am now able to reproduce it even with text format
with builtin Serdes. Essentially this is a bug in the thrift code (not
sure if it is in the client or server) since the same alter table
statement works fine when the hive client does not use thrift. Here are
the details:
cat create_dummy.sql
CREATE external TABLE if not exists dummy (
partition_name string
,partition_id int
)
PARTITIONED BY ( datestamp string, srcid string, action string, testid
string )
row format delimited
stored as textfile
location '/user/pradeepk/dummy';
hive -f create_dummy.sql
10/06/18 10:13:36 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006181013_184583537.txt
OK
Time taken: 0.627 seconds
hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid
= '100',action='view',testid='10') location
'/user/pradeepk/dummy/20100602/100/view/10';"
10/06/18 10:14:11 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_700722546.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException:
get_partition failed: unknown result
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
hive --config hive_conf_without_thrift -e "ALTER TABLE dummy add
partition(datestamp = '20100602', srcid =
'100',action='view',testid='10') location
'/user/pradeepk/dummy/20100602/100/view/10';"
10/06/18 10:14:31 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_598649843.txt
OK
Time taken: 5.849 seconds
Is there some thrift setting I am missing or is this a bug? - If it is
the latter, I can open a jira with the above details.
Thanks,
Pradeep
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Thursday, June 17, 2010 1:25 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Here are the create table and alter table statements:
CREATE external TABLE if not exists mytable (
bc string
,src_spaceid string
,srcpvid string
,dstpvid string
,dst_spaceid string
,page_params map<string, string>
,clickinfo map<string, string>
,viewinfo array<map<string, string>>
)
PARTITIONED BY ( datestamp string, srcid string, action string, testid
string )
row format serde 'com.yahoo.mySerde'
stored as inputformat 'org.apache.hadoop.mapred.SequenceFileInputFormat'
outputformat 'org.apache.hadoop.mapred.SequenceFileOutputFormat'
location '/user/pradeepk/mytable';
hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add
partition(datestamp = '20091101', srcid =
'19174',action='click',testid='NOTESTID') location
'/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"
I get the following error:
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException:
get_partition failed: unknown result
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
If I don't use thrift and use a hive-site.xml to directly talk to the
db, the alter table seems to succeed:
hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "ALTER
TABLE mytable add partition(datestamp = '20091101', srcid =
'19174',action='click',testid='NOTESTID') location
'/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"
However I get errors when I try to run a query:
[pradeepk@chargesize:~/dev]hive --auxpath ult-serde.jar --config
hive_conf_without_thrift -e "select src_spaceid from
ult_search_austria_ult where datestamp='20091101';"
10/06/17 13:22:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006171322_1913647383.txt
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
java.lang.IllegalArgumentException: Can not create a Path from an empty
string
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)
at org.apache.hadoop.fs.Path.<init>(Path.java:90)
at org.apache.hadoop.fs.Path.<init>(Path.java:50)
at
org.apache.hadoop.mapred.JobClient.copyRemoteFiles(JobClient.java:523)
at
org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient
.java:603)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)
at
org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
at
org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:684)
at
org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
Any help is much appreciated.
Pradeep
________________________________
From: Ashish Thusoo [mailto:athusoo@facebook.com]
Sent: Thursday, June 17, 2010 11:15 AM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
hmm... Can you send the exact command and also the create table command
for this table.
Ashish
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Thursday, June 17, 2010 9:09 AM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Sorry - that was a cut-paste error - I don't have the action part - so I
am specifying key-value pairs. Since what I am trying to do seems like a
basic operation, I am wondering if it's something to do with my Serde -
unfortunately the error I see gives me no clue of what could be wrong -
any help would be greatly appreciated!
Thanks,
Pradeep
________________________________
From: yq he [mailto:hhh.heyq@gmail.com]
Sent: Wednesday, June 16, 2010 5:54 PM
To: hive-user@hadoop.apache.org
Subject: Re: alter table add partition error
Hi Pradeep,
partition definition need to be key-value pairs. partition key `action`
seems missed the value.
Thanks
Yongqiang
On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath <pr...@yahoo-inc.com>
wrote:
Hi,
I am trying to create an external table against already existing
data in sequencefile format. However I have written a custom Serde to
interpret the data. I am able to create the table fine but get the
exception shown in the session output below when I try to add partition
- any help would be greatly appreciated.
Thanks,
Pradeep
== session output ===
[pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER
TABLE mytable add partition(datestamp = '20091101', srcid = '10',action)
location '/user/pradeepk/mytable/20091101/10';"
10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException:
get_partition failed: unknown result
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
[pradeepk@chargesize:~/dev/howl]
== session output ===
/tmp/pradeepk/hive.log has:
2010-06-16 17:09:00,841 ERROR exec.DDLTask
(SessionState.java:printError(269)) - FAILED: Error in metadata:
org.apache.thrift.TApplicationException: get_partition failed: unknown
result
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.thrift.TApplicationException: get_partition failed: unknown
result
at
org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
at
org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
at
org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: org.apache.thrift.TApplicationException: get_partition
failed: unknown result
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get
_partition(ThriftHiveMetastore.java:931)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_part
ition(ThriftHiveMetastore.java:899)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMe
taStoreClient.java:500)
at
org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
... 15 more
The thrift server messages are:
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table :
db=default tbl=mytable
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store
with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize
called
10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition :
db=default tbl=mytable
Re: alter table add partition error
Posted by Ning Zhang <nz...@facebook.com>.
Pradeep,
I ran the commands you provided and it succeeded with the expected behavior.
One possibility is that there are multiple versions of libthrift.jar in your CLASSPATH (hadoop & hive). Can you check in the Hadoop & Hive CLASSPATH so that no other libthrift.jar is there? What is in hive_config_without_thrift?
Thanks,
Ning
On Jun 18, 2010, at 10:19 AM, Pradeep Kamath wrote:
I think there are two separate issues here – I want to open a jira for the first one since I am now able to reproduce it even with text format with builtin Serdes. Essentially this is a bug in the thrift code (not sure if it is in the client or server) since the same alter table statement works fine when the hive client does not use thrift. Here are the details:
cat create_dummy.sql
CREATE external TABLE if not exists dummy (
partition_name string
,partition_id int
)
PARTITIONED BY ( datestamp string, srcid string, action string, testid string )
row format delimited
stored as textfile
location '/user/pradeepk/dummy';
hive -f create_dummy.sql
10/06/18 10:13:36 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181013_184583537.txt
OK
Time taken: 0.627 seconds
hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid = '100',action='view',testid='10') location '/user/pradeepk/dummy/20100602/100/view/10';"
10/06/18 10:14:11 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_700722546.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
hive --config hive_conf_without_thrift -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid = '100',action='view',testid='10') location '/user/pradeepk/dummy/20100602/100/view/10';"
10/06/18 10:14:31 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_598649843.txt
OK
Time taken: 5.849 seconds
Is there some thrift setting I am missing or is this a bug? – If it is the latter, I can open a jira with the above details.
Thanks,
Pradeep
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Thursday, June 17, 2010 1:25 PM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: alter table add partition error
Here are the create table and alter table statements:
CREATE external TABLE if not exists mytable (
bc string
,src_spaceid string
,srcpvid string
,dstpvid string
,dst_spaceid string
,page_params map<string, string>
,clickinfo map<string, string>
,viewinfo array<map<string, string>>
)
PARTITIONED BY ( datestamp string, srcid string, action string, testid string )
row format serde 'com.yahoo.mySerde’
stored as inputformat 'org.apache.hadoop.mapred.SequenceFileInputFormat' outputformat 'org.apache.hadoop.mapred.SequenceFileOutputFormat'
location '/user/pradeepk/mytable’;
hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '19174',action='click',testid='NOTESTID') location '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"
I get the following error:
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
If I don’t use thrift and use a hive-site.xml to directly talk to the db, the alter table seems to succeed:
hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '19174',action='click',testid='NOTESTID') location '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"
However I get errors when I try to run a query:
[pradeepk@chargesize:~/dev]hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "select src_spaceid from ult_search_austria_ult where datestamp='20091101';"
10/06/17 13:22:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006171322_1913647383.txt
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
java.lang.IllegalArgumentException: Can not create a Path from an empty string
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)
at org.apache.hadoop.fs.Path.<init>(Path.java:90)
at org.apache.hadoop.fs.Path.<init>(Path.java:50)
at org.apache.hadoop.mapred.JobClient.copyRemoteFiles(JobClient.java:523)
at org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient.java:603)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:684)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
Any help is much appreciated.
Pradeep
________________________________
From: Ashish Thusoo [mailto:athusoo@facebook.com]
Sent: Thursday, June 17, 2010 11:15 AM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: alter table add partition error
hmm... Can you send the exact command and also the create table command for this table.
Ashish
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Thursday, June 17, 2010 9:09 AM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: alter table add partition error
Sorry – that was a cut-paste error – I don’t have the action part – so I am specifying key-value pairs. Since what I am trying to do seems like a basic operation, I am wondering if it’s something to do with my Serde – unfortunately the error I see gives me no clue of what could be wrong – any help would be greatly appreciated!
Thanks,
Pradeep
________________________________
From: yq he [mailto:hhh.heyq@gmail.com]
Sent: Wednesday, June 16, 2010 5:54 PM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: alter table add partition error
Hi Pradeep,
partition definition need to be key-value pairs. partition key `action` seems missed the value.
Thanks
Yongqiang
On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath <pr...@yahoo-inc.com>> wrote:
Hi,
I am trying to create an external table against already existing data in sequencefile format. However I have written a custom Serde to interpret the data. I am able to create the table fine but get the exception shown in the session output below when I try to add partition – any help would be greatly appreciated.
Thanks,
Pradeep
== session output ===
[pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = ‘10’,action) location '/user/pradeepk/mytable/20091101/10’;"
10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
[pradeepk@chargesize:~/dev/howl]
== session output ===
/tmp/pradeepk/hive.log has:
2010-06-16 17:09:00,841 ERROR exec.DDLTask (SessionState.java:printError(269)) - FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.TApplicationException: get_partition failed: unknown result
at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
at org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: org.apache.thrift.TApplicationException: get_partition failed: unknown result
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition(ThriftHiveMetastore.java:931)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition(ThriftHiveMetastore.java:899)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:500)
at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
... 15 more
The thrift server messages are:
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table : db=default tbl=mytable
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize called
10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition : db=default tbl=mytable
RE: alter table add partition error
Posted by Pradeep Kamath <pr...@yahoo-inc.com>.
I think there are two separate issues here - I want to open a jira for
the first one since I am now able to reproduce it even with text format
with builtin Serdes. Essentially this is a bug in the thrift code (not
sure if it is in the client or server) since the same alter table
statement works fine when the hive client does not use thrift. Here are
the details:
cat create_dummy.sql
CREATE external TABLE if not exists dummy (
partition_name string
,partition_id int
)
PARTITIONED BY ( datestamp string, srcid string, action string, testid
string )
row format delimited
stored as textfile
location '/user/pradeepk/dummy';
hive -f create_dummy.sql
10/06/18 10:13:36 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006181013_184583537.txt
OK
Time taken: 0.627 seconds
hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid
= '100',action='view',testid='10') location
'/user/pradeepk/dummy/20100602/100/view/10';"
10/06/18 10:14:11 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_700722546.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException:
get_partition failed: unknown result
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
hive --config hive_conf_without_thrift -e "ALTER TABLE dummy add
partition(datestamp = '20100602', srcid =
'100',action='view',testid='10') location
'/user/pradeepk/dummy/20100602/100/view/10';"
10/06/18 10:14:31 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_598649843.txt
OK
Time taken: 5.849 seconds
Is there some thrift setting I am missing or is this a bug? - If it is
the latter, I can open a jira with the above details.
Thanks,
Pradeep
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Thursday, June 17, 2010 1:25 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Here are the create table and alter table statements:
CREATE external TABLE if not exists mytable (
bc string
,src_spaceid string
,srcpvid string
,dstpvid string
,dst_spaceid string
,page_params map<string, string>
,clickinfo map<string, string>
,viewinfo array<map<string, string>>
)
PARTITIONED BY ( datestamp string, srcid string, action string, testid
string )
row format serde 'com.yahoo.mySerde'
stored as inputformat 'org.apache.hadoop.mapred.SequenceFileInputFormat'
outputformat 'org.apache.hadoop.mapred.SequenceFileOutputFormat'
location '/user/pradeepk/mytable';
hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add
partition(datestamp = '20091101', srcid =
'19174',action='click',testid='NOTESTID') location
'/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"
I get the following error:
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException:
get_partition failed: unknown result
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
If I don't use thrift and use a hive-site.xml to directly talk to the
db, the alter table seems to succeed:
hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "ALTER
TABLE mytable add partition(datestamp = '20091101', srcid =
'19174',action='click',testid='NOTESTID') location
'/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"
However I get errors when I try to run a query:
[pradeepk@chargesize:~/dev]hive --auxpath ult-serde.jar --config
hive_conf_without_thrift -e "select src_spaceid from
ult_search_austria_ult where datestamp='20091101';"
10/06/17 13:22:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006171322_1913647383.txt
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
java.lang.IllegalArgumentException: Can not create a Path from an empty
string
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)
at org.apache.hadoop.fs.Path.<init>(Path.java:90)
at org.apache.hadoop.fs.Path.<init>(Path.java:50)
at
org.apache.hadoop.mapred.JobClient.copyRemoteFiles(JobClient.java:523)
at
org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient
.java:603)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)
at
org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
at
org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:684)
at
org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
Any help is much appreciated.
Pradeep
________________________________
From: Ashish Thusoo [mailto:athusoo@facebook.com]
Sent: Thursday, June 17, 2010 11:15 AM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
hmm... Can you send the exact command and also the create table command
for this table.
Ashish
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Thursday, June 17, 2010 9:09 AM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Sorry - that was a cut-paste error - I don't have the action part - so I
am specifying key-value pairs. Since what I am trying to do seems like a
basic operation, I am wondering if it's something to do with my Serde -
unfortunately the error I see gives me no clue of what could be wrong -
any help would be greatly appreciated!
Thanks,
Pradeep
________________________________
From: yq he [mailto:hhh.heyq@gmail.com]
Sent: Wednesday, June 16, 2010 5:54 PM
To: hive-user@hadoop.apache.org
Subject: Re: alter table add partition error
Hi Pradeep,
partition definition need to be key-value pairs. partition key `action`
seems missed the value.
Thanks
Yongqiang
On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath <pr...@yahoo-inc.com>
wrote:
Hi,
I am trying to create an external table against already existing
data in sequencefile format. However I have written a custom Serde to
interpret the data. I am able to create the table fine but get the
exception shown in the session output below when I try to add partition
- any help would be greatly appreciated.
Thanks,
Pradeep
== session output ===
[pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER
TABLE mytable add partition(datestamp = '20091101', srcid = '10',action)
location '/user/pradeepk/mytable/20091101/10';"
10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException:
get_partition failed: unknown result
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
[pradeepk@chargesize:~/dev/howl]
== session output ===
/tmp/pradeepk/hive.log has:
2010-06-16 17:09:00,841 ERROR exec.DDLTask
(SessionState.java:printError(269)) - FAILED: Error in metadata:
org.apache.thrift.TApplicationException: get_partition failed: unknown
result
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.thrift.TApplicationException: get_partition failed: unknown
result
at
org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
at
org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
at
org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: org.apache.thrift.TApplicationException: get_partition
failed: unknown result
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get
_partition(ThriftHiveMetastore.java:931)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_part
ition(ThriftHiveMetastore.java:899)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMe
taStoreClient.java:500)
at
org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
... 15 more
The thrift server messages are:
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table :
db=default tbl=mytable
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store
with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize
called
10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition :
db=default tbl=mytable
RE: alter table add partition error
Posted by Pradeep Kamath <pr...@yahoo-inc.com>.
Here are the create table and alter table statements:
CREATE external TABLE if not exists mytable (
bc string
,src_spaceid string
,srcpvid string
,dstpvid string
,dst_spaceid string
,page_params map<string, string>
,clickinfo map<string, string>
,viewinfo array<map<string, string>>
)
PARTITIONED BY ( datestamp string, srcid string, action string, testid
string )
row format serde 'com.yahoo.mySerde'
stored as inputformat 'org.apache.hadoop.mapred.SequenceFileInputFormat'
outputformat 'org.apache.hadoop.mapred.SequenceFileOutputFormat'
location '/user/pradeepk/mytable';
hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add
partition(datestamp = '20091101', srcid =
'19174',action='click',testid='NOTESTID') location
'/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"
I get the following error:
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException:
get_partition failed: unknown result
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
If I don't use thrift and use a hive-site.xml to directly talk to the
db, the alter table seems to succeed:
hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "ALTER
TABLE mytable add partition(datestamp = '20091101', srcid =
'19174',action='click',testid='NOTESTID') location
'/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"
However I get errors when I try to run a query:
[pradeepk@chargesize:~/dev]hive --auxpath ult-serde.jar --config
hive_conf_without_thrift -e "select src_spaceid from
ult_search_austria_ult where datestamp='20091101';"
10/06/17 13:22:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006171322_1913647383.txt
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
java.lang.IllegalArgumentException: Can not create a Path from an empty
string
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)
at org.apache.hadoop.fs.Path.<init>(Path.java:90)
at org.apache.hadoop.fs.Path.<init>(Path.java:50)
at
org.apache.hadoop.mapred.JobClient.copyRemoteFiles(JobClient.java:523)
at
org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient
.java:603)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)
at
org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
at
org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:684)
at
org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
Any help is much appreciated.
Pradeep
________________________________
From: Ashish Thusoo [mailto:athusoo@facebook.com]
Sent: Thursday, June 17, 2010 11:15 AM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
hmm... Can you send the exact command and also the create table command
for this table.
Ashish
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Thursday, June 17, 2010 9:09 AM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Sorry - that was a cut-paste error - I don't have the action part - so I
am specifying key-value pairs. Since what I am trying to do seems like a
basic operation, I am wondering if it's something to do with my Serde -
unfortunately the error I see gives me no clue of what could be wrong -
any help would be greatly appreciated!
Thanks,
Pradeep
________________________________
From: yq he [mailto:hhh.heyq@gmail.com]
Sent: Wednesday, June 16, 2010 5:54 PM
To: hive-user@hadoop.apache.org
Subject: Re: alter table add partition error
Hi Pradeep,
partition definition need to be key-value pairs. partition key `action`
seems missed the value.
Thanks
Yongqiang
On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath <pr...@yahoo-inc.com>
wrote:
Hi,
I am trying to create an external table against already existing
data in sequencefile format. However I have written a custom Serde to
interpret the data. I am able to create the table fine but get the
exception shown in the session output below when I try to add partition
- any help would be greatly appreciated.
Thanks,
Pradeep
== session output ===
[pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER
TABLE mytable add partition(datestamp = '20091101', srcid = '10',action)
location '/user/pradeepk/mytable/20091101/10';"
10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException:
get_partition failed: unknown result
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
[pradeepk@chargesize:~/dev/howl]
== session output ===
/tmp/pradeepk/hive.log has:
2010-06-16 17:09:00,841 ERROR exec.DDLTask
(SessionState.java:printError(269)) - FAILED: Error in metadata:
org.apache.thrift.TApplicationException: get_partition failed: unknown
result
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.thrift.TApplicationException: get_partition failed: unknown
result
at
org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
at
org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
at
org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: org.apache.thrift.TApplicationException: get_partition
failed: unknown result
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get
_partition(ThriftHiveMetastore.java:931)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_part
ition(ThriftHiveMetastore.java:899)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMe
taStoreClient.java:500)
at
org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
... 15 more
The thrift server messages are:
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table :
db=default tbl=mytable
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store
with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize
called
10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition :
db=default tbl=mytable
RE: alter table add partition error
Posted by Ashish Thusoo <at...@facebook.com>.
hmm... Can you send the exact command and also the create table command for this table.
Ashish
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Thursday, June 17, 2010 9:09 AM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Sorry - that was a cut-paste error - I don't have the action part - so I am specifying key-value pairs. Since what I am trying to do seems like a basic operation, I am wondering if it's something to do with my Serde - unfortunately the error I see gives me no clue of what could be wrong - any help would be greatly appreciated!
Thanks,
Pradeep
________________________________
From: yq he [mailto:hhh.heyq@gmail.com]
Sent: Wednesday, June 16, 2010 5:54 PM
To: hive-user@hadoop.apache.org
Subject: Re: alter table add partition error
Hi Pradeep,
partition definition need to be key-value pairs. partition key `action` seems missed the value.
Thanks
Yongqiang
On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath <pr...@yahoo-inc.com>> wrote:
Hi,
I am trying to create an external table against already existing data in sequencefile format. However I have written a custom Serde to interpret the data. I am able to create the table fine but get the exception shown in the session output below when I try to add partition - any help would be greatly appreciated.
Thanks,
Pradeep
== session output ===
[pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '10',action) location '/user/pradeepk/mytable/20091101/10';"
10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
[pradeepk@chargesize:~/dev/howl]
== session output ===
/tmp/pradeepk/hive.log has:
2010-06-16 17:09:00,841 ERROR exec.DDLTask (SessionState.java:printError(269)) - FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result
org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.TApplicationException: get_partition failed: unknown result
at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
at org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: org.apache.thrift.TApplicationException: get_partition failed: unknown result
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition(ThriftHiveMetastore.java:931)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition(ThriftHiveMetastore.java:899)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:500)
at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
... 15 more
The thrift server messages are:
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table : db=default tbl=mytable
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize called
10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition : db=default tbl=mytable
RE: alter table add partition error
Posted by Pradeep Kamath <pr...@yahoo-inc.com>.
Sorry - that was a cut-paste error - I don't have the action part - so I
am specifying key-value pairs. Since what I am trying to do seems like a
basic operation, I am wondering if it's something to do with my Serde -
unfortunately the error I see gives me no clue of what could be wrong -
any help would be greatly appreciated!
Thanks,
Pradeep
________________________________
From: yq he [mailto:hhh.heyq@gmail.com]
Sent: Wednesday, June 16, 2010 5:54 PM
To: hive-user@hadoop.apache.org
Subject: Re: alter table add partition error
Hi Pradeep,
partition definition need to be key-value pairs. partition key `action`
seems missed the value.
Thanks
Yongqiang
On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath <pr...@yahoo-inc.com>
wrote:
Hi,
I am trying to create an external table against already existing
data in sequencefile format. However I have written a custom Serde to
interpret the data. I am able to create the table fine but get the
exception shown in the session output below when I try to add partition
- any help would be greatly appreciated.
Thanks,
Pradeep
== session output ===
[pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER
TABLE mytable add partition(datestamp = '20091101', srcid = '10',action)
location '/user/pradeepk/mytable/20091101/10';"
10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException:
get_partition failed: unknown result
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
[pradeepk@chargesize:~/dev/howl]
== session output ===
/tmp/pradeepk/hive.log has:
2010-06-16 17:09:00,841 ERROR exec.DDLTask
(SessionState.java:printError(269)) - FAILED: Error in metadata:
org.apache.thrift.TApplicationException: get_partition failed: unknown
result
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.thrift.TApplicationException: get_partition failed: unknown
result
at
org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
at
org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
at
org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: org.apache.thrift.TApplicationException: get_partition
failed: unknown result
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get
_partition(ThriftHiveMetastore.java:931)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_part
ition(ThriftHiveMetastore.java:899)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMe
taStoreClient.java:500)
at
org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
... 15 more
The thrift server messages are:
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table :
db=default tbl=mytable
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store
with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize
called
10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition :
db=default tbl=mytable
Re: alter table add partition error
Posted by yq he <hh...@gmail.com>.
Hi Pradeep,
partition definition need to be key-value pairs. partition key `action`
seems missed the value.
Thanks
Yongqiang
On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath <pr...@yahoo-inc.com>wrote:
> Hi,
>
> I am trying to create an external table against already existing data
> in sequencefile format. However I have written a custom Serde to interpret
> the data. I am able to create the table fine but get the exception shown in
> the session output below when I try to add partition – any help would be
> greatly appreciated.
>
>
>
> Thanks,
>
> Pradeep
>
>
>
> == session output ===
>
>
>
> [pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER
> TABLE mytable add partition(datestamp = '20091101', srcid = ‘10’,action)
> location '/user/pradeepk/mytable/20091101/10’;"
>
> 10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
> found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use
> core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of
> core-default.xml, mapred-default.xml and hdfs-default.xml respectively
>
> Hive history
> file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
>
> *FAILED: Error in metadata: org.apache.thrift.TApplicationException:
> get_partition failed: unknown result*
>
> *FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask*
>
> [pradeepk@chargesize:~/dev/howl]
>
>
>
> == session output ===
>
>
>
> /tmp/pradeepk/hive.log has:
>
> 2010-06-16 17:09:00,841 ERROR exec.DDLTask
> (SessionState.java:printError(269)) - FAILED: Error in metadata:
> org.apache.thrift.TApplicationException: get_partition failed: unknown
> result
>
> org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.thrift.TApplicationException: get_partition failed: unknown
> result
>
> at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
>
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)
>
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)
>
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
>
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
>
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
>
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
>
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
>
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
>
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
>
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>
> at java.lang.reflect.Method.invoke(Method.java:597)
>
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>
> *Caused by: org.apache.thrift.TApplicationException: get_partition failed:
> unknown result*
>
> * at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition(ThriftHiveMetastore.java:931)
> *
>
> * at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition(ThriftHiveMetastore.java:899)
> *
>
> * at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:500)
> *
>
> * at
> org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)*
>
> * ... 15 more*
>
>
>
> The thrift server messages are:
>
> 10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table : db=default
> tbl=mytable
>
> 10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store with
> implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
>
> 10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize
> called
>
> 10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore
>
> 10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition :
> db=default tbl=mytable
>
>
>
>
>