You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Pradeep Kamath <pr...@yahoo-inc.com> on 2010/07/01 18:15:32 UTC
RE: alter table add partition error
I am having difficulty starting a thrift service instance in the junit
test for this patch -
https://issues.apache.org/jira/browse/HIVE-1428?focusedCommentId=1288361
4&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpane
l#action_12883614 - any pointers would be appreciated.
Thanks,
Pradeep
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Friday, June 25, 2010 1:37 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
I have posted a patch for this - not sure how to write unit tests for it
- so patch doesn't have tests.
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Wednesday, June 23, 2010 9:04 AM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
In the second step of
http://wiki.apache.org/hadoop/Hive/HowToContribute#Generating_Code
"Use the approved version of thrift. This is currently
thrift-instant-r790732, which you can obtain from
http://instant.thrift-rpc.org"
Looks like that site is down since it is not responding to requests - is
there an alternate url I can try? Also, is the approved version still
thrift-instant-r790732?
________________________________
From: Paul Yang [mailto:pyang@facebook.com]
Sent: Tuesday, June 22, 2010 6:06 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Hey Pradeep,
Yep, that's on the right path :) Since you made a change to
hive_metastore.thrift, you should run the thrift compiler by calling
'ant thriftif' within trunk/metastore. But before you can do that, the
thrift compiler has to be in your path. JVS has a nice write up on how
to do this:
http://wiki.apache.org/hadoop/Hive/HowToContribute#Generating_Code
Also, I've set up a JIRA for this issue at:
https://issues.apache.org/jira/browse/HIVE-1428
Can you post your patch there? It makes it a lot easier to track issues.
Thanks!
-Paul
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Tuesday, June 22, 2010 10:22 AM
To: hive-user@hadoop.apache.org
Subject: Re: alter table add partition error
Hi Paul,
I have attached a draft of a patch (incomplete). The problem I am
facing is that to make this change I seem to need to change the thrift
idl (hive_metastore.thrift). Looking at the code structure it seems like
the generate java files are checked in rather than generated through the
build (is this so that the build does not depend on the thrift
compiler?). I tried downloading and building thrift but hit issues in
the make install - wanted to check with you if the patch looks correct
so far or if I am on the wrong path. Also if the direction is right, I
need some guidance on generating the java files from the modified idl.
Thanks,
Pradeep
Paul Yang wrote:
Well, the section of code you are referring to is automatically
generated by the thrift compiler so we can't modify it. As the inability
to return null is a limitation of Thrift, we can work around that issue
by throwing an exception instead.
Check out this thread:
http://publists.facebook.com/pipermail/thrift/2007-September/000085.html
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Monday, June 21, 2010 5:46 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Hi Paul,
Yes I don't mind working on a patch though I have no knowledge of how
thrift code works. I was also confused by your suggestion for a fix. I
thought the fix would be in
ThriftHiveMetaStore.Client.recv_get_partition() - to first check
if(result.o1 != null) throw result.o1, then to just return
result.success and not throw the "unknown result" exception. I tried
something along these lines and the alter table worked. Am I missing
something?
Thanks,
Pradeep
________________________________
From: Paul Yang [mailto:pyang@facebook.com]
Sent: Monday, June 21, 2010 2:35 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Ah okay. So what's going on is that getPartition() is supposed to return
null if the specified partition doesn't exist. But because Thrift can't
handle null return values, we are seeing this exception. This isn't a
problem with a local metastore because Thrift isn't used.
One solution to this problem would be modify ObjectStore.getPartition()
to throw a NoSuchObjectException instead of returning null for
non-existent partitions. Then Hive.getPartition() can catch the
exception and return null to retain the original behavior.
Pradeep, are you interested in working on a patch for this problem?
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Monday, June 21, 2010 2:11 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
I tried debugging in code a little more. Here is what I found:
The code in ThriftHiveMetaStore eventually makes a call -
get_partition() passing the partition key values for the partition I am
trying to add using alter table. I assume this is to check that the
partition doesn't already exist.
I added a debug line in the following code:
public Partition recv_get_partition() throws MetaException, TException
{
TMessage msg = iprot_.readMessageBegin();
if (msg.type == TMessageType.EXCEPTION) {
TApplicationException x = TApplicationException.read(iprot_);
iprot_.readMessageEnd();
throw x;
}
get_partition_result result = new get_partition_result();
result.read(iprot_);
System.err.println("XXX: result:" + result);
iprot_.readMessageEnd();
if (result.isSetSuccess()) {
return result.success;
}
if (result.o1 != null) {
throw result.o1;
}
throw new
TApplicationException(TApplicationException.MISSING_RESULT,
"get_partition failed: unknown result");
}
I also put a debug statements in the read() method:
public void read(TProtocol iprot) throws TException {
TField field;
iprot.readStructBegin();
System.err.println("XXX: Reading TProtocol object:");
while (true)
{
field = iprot.readFieldBegin();
System.err.println("XXX: field just read:" + field);
if (field.type == TType.STOP) {
break;
I got
XXX: Reading TProtocol object:
XXX: field just read:<TField name:'' type:0 field-id:0>
XXX: result:get_partition_result(success:null, o1:null)
The field read in the thrift response message is of type "STOP" and with
id of type SUCCESS. This seems right since there are no existing
partitions. But the way the rest of the code handles this, results in
the exception.
Any pointers?
TIA,
Pradeep
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Friday, June 18, 2010 2:51 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Looks like the standalone script works fine against the existing
partition:
./ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name
default dummy datestamp=20100602/srcid=100/action=view/testid=10
Partition(parameters={'transient_lastDdlTime': '1276881277'},
tableName='dummy', createTime=1276881277, lastAccessTime=0,
values=['20100602', '100', 'view', '10'], dbName='default',
sd=StorageDescriptor(outputFormat='org.apache.hadoop.hive.ql.io.HiveIgno
reKeyTextOutputFormat', sortCols=[],
inputFormat='org.apache.hadoop.mapred.TextInputFormat',
cols=[FieldSchema(comment=None, type='string', name='partition_name'),
FieldSchema(comment=None, type='int', name='partition_id')],
compressed=False, bucketCols=[], numBuckets=-1, parameters={},
serdeInfo=SerDeInfo(serializationLib='org.apache.hadoop.hive.serde2.lazy
.LazySimpleSerDe', name=None, parameters={'serialization.format': '1'}),
location='hdfs://wilbur21.labs.corp.sp1.yahoo.com/user/pradeepk/dummy/20
100602/100/view/10'))
[pradeepk@chargesize:~/dev/howl/src/metastore/src/gen-py/hive_metastore]
However when I tried to add another partition with the hive cli using
thrift:
hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid
= '100',action='click',testid='10') location
'/user/pradeepk/dummy/20100602/100/click/10';"
10/06/18 14:49:13 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006181449_1158492515.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException:
get_partition failed: unknown result
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
tail -30 /tmp/pradeepk/hive.log
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
2010-06-18 14:49:14,124 ERROR exec.DDLTask
(SessionState.java:printError(277)) - FAILED: Error in metadata:
org.apache.thrift.TApplicationException: get_partition failed: unknown
result
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.thrift.TApplicationException: get_partition failed: unknown
result
at
org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
at
org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:255)
at
org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:169)
at
org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:633)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:506)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:384)
at
org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:267)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: org.apache.thrift.TApplicationException: get_partition
failed: unknown result
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get
_partition(ThriftHiveMetastore.java:931)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_part
ition(ThriftHiveMetastore.java:899)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMe
taStoreClient.java:500)
at
org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
... 15 more
2010-06-18 14:49:14,124 ERROR ql.Driver
(SessionState.java:printError(277)) - FAILED: Execution Error, return
code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
________________________________
From: Paul Yang [mailto:pyang@facebook.com]
Sent: Friday, June 18, 2010 2:19 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Looks like the thrift python libraries aren't in your path. run:
export PYTHONPATH=<path-to-trunk>/build/dist/lib/py/
before trying the remote command
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Friday, June 18, 2010 1:38 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Sorry, don't know python - so couldn't make sense out of the following
error when I run the command you suggested:
[src/metastore/src/gen-py/hive_metastore]./ThriftHiveMetastore-remote -h
localhost:9080 get_partition_by_name default dummy
datestamp=20100602/srcid=100/action=view/testid=10
Traceback (most recent call last):
File "./ThriftHiveMetastore-remote", line 11, in ?
from thrift.transport import TTransport
ImportError: No module named thrift.transport
Thanks,
Pradeep
________________________________
From: Paul Yang [mailto:pyang@facebook.com]
Sent: Friday, June 18, 2010 12:10 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Hi Pradeep,
In trunk/metastore/src/gen-py/hive_metastore/, there is a script called
ThriftHiveMetastore-remote that can be used to test out the thrift
server independent of the CLI. As a quick test to narrow down the
problem, after the partition is created, can you try running
ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name
default dummy datestamp=20100602/srcid=100/action=view/testid=10
Cheers,
Paul
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Friday, June 18, 2010 11:26 AM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Hi Ning,
I have pasted below [1] the values for HADOOP_CLASSPATH and CLASSPATH
from an "env" command printed just before the following line in
execHiveCmd.sh
exec $HADOOP jar ${HIVE_LIB}/hive-cli-*.jar $CLASS $HIVE_OPTS "$@"
There is only one occurrence of libthrift.jar whose cksum matches the
one from lib/libthrift.jar from a hive checkout build.
The hive_config_without_thrift is a conf directory containing a
hive-site.xml with jdo settings to connect to the database directly. By
default I have hive set up to use a hive-site.xml [2] which connects to
a thrift server running on localhost.
Any pointers appreciated!
Thanks,
Pradeep
[1]
HADOOP_CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_c
lient/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.
1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home
/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_cl
ient/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/c
ommons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.j
ar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/lib
exec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_
client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucl
eus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus
-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1
.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home
/y/libexec/hive_client/lib/derby.jar:/home/y/libex
ec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbas
e-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.ja
r:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hiv
e_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_
contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/
libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_clie
nt/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.
0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/l
ibexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/
lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.
6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/
y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/l
ib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libe
xec/hive_client/lib/libfb303.jar:/home/y/libexec/hi
ve_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15
.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_cli
ent/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtem
plate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y
/libexec/hive_client/lib/zookeeper-3.2.2.jar:
CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/l
ib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/
home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libe
xec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/li
b/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-
dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/hom
e/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hi
ve_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/
lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-con
nectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1
.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:
/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libe
xec/hive_client/lib/derby.jar:/home/y/libex
ec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbas
e-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.ja
r:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hiv
e_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_
contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/
libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_clie
nt/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.
0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/l
ibexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/
lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.
6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/
y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/l
ib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libe
xec/hive_client/lib/libfb303.jar:/home/y/libexec/hi
ve_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15
.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_cli
ent/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtem
plate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y
/libexec/hive_client/lib/zookeeper-3.2.2.jar:
[2]
cat /home/y/libexec/hive_client/conf/hive-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<!-- Hive Configuration can either be stored in this file or in the
hadoop configuration files -->
<!-- that are implied by Hadoop setup variables.
-->
<!-- Aside from Hadoop setup variables - this file is provided as a
convenience so that Hive -->
<!-- users do not have to edit hadoop configuration files (that may be
managed as a centralized -->
<!-- resource).
-->
<!-- Hive Execution Parameters -->
<property>
<name>hive.metastore.uris</name>
<value>thrift://localhost:9080</value>
<description>host and port for the thrift metastore
server</description>
</property>
<property>
<name>hive.metastore.local</name>
<value>false</value>
<description>controls whether to connect to remove metastore server or
open a new metastore server in Hive Client JVM</description>
</property>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/user/pradeepk/hive/warehouse</value>
<description>location of default database for the
warehouse</description>
</property>
</configuration>
________________________________
From: Ning Zhang [mailto:nzhang@facebook.com]
Sent: Friday, June 18, 2010 10:50 AM
To: hive-user@hadoop.apache.org
Subject: Re: alter table add partition error
Pradeep,
I ran the commands you provided and it succeeded with the expected
behavior.
One possibility is that there are multiple versions of libthrift.jar in
your CLASSPATH (hadoop & hive). Can you check in the Hadoop & Hive
CLASSPATH so that no other libthrift.jar is there? What is in
hive_config_without_thrift?
Thanks,
Ning
On Jun 18, 2010, at 10:19 AM, Pradeep Kamath wrote:
I think there are two separate issues here - I want to open a jira for
the first one since I am now able to reproduce it even with text format
with builtin Serdes. Essentially this is a bug in the thrift code (not
sure if it is in the client or server) since the same alter table
statement works fine when the hive client does not use thrift. Here are
the details:
cat create_dummy.sql
CREATE external TABLE if not exists dummy (
partition_name string
,partition_id int
)
PARTITIONED BY ( datestamp string, srcid string, action string, testid
string )
row format delimited
stored as textfile
location '/user/pradeepk/dummy';
hive -f create_dummy.sql
10/06/18 10:13:36 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006181013_184583537.txt
OK
Time taken: 0.627 seconds
hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid
= '100',action='view',testid='10') location
'/user/pradeepk/dummy/20100602/100/view/10';"
10/06/18 10:14:11 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_700722546.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException:
get_partition failed: unknown result
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
hive --config hive_conf_without_thrift -e "ALTER TABLE dummy add
partition(datestamp = '20100602', srcid =
'100',action='view',testid='10') location
'/user/pradeepk/dummy/20100602/100/view/10';"
10/06/18 10:14:31 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_598649843.txt
OK
Time taken: 5.849 seconds
Is there some thrift setting I am missing or is this a bug? - If it is
the latter, I can open a jira with the above details.
Thanks,
Pradeep
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Thursday, June 17, 2010 1:25 PM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Here are the create table and alter table statements:
CREATE external TABLE if not exists mytable (
bc string
,src_spaceid string
,srcpvid string
,dstpvid string
,dst_spaceid string
,page_params map<string, string>
,clickinfo map<string, string>
,viewinfo array<map<string, string>>
)
PARTITIONED BY ( datestamp string, srcid string, action string, testid
string )
row format serde 'com.yahoo.mySerde'
stored as inputformat 'org.apache.hadoop.mapred.SequenceFileInputFormat'
outputformat 'org.apache.hadoop.mapred.SequenceFileOutputFormat'
location '/user/pradeepk/mytable';
hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add
partition(datestamp = '20091101', srcid =
'19174',action='click',testid='NOTESTID') location
'/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"
I get the following error:
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException:
get_partition failed: unknown result
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
If I don't use thrift and use a hive-site.xml to directly talk to the
db, the alter table seems to succeed:
hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "ALTER
TABLE mytable add partition(datestamp = '20091101', srcid =
'19174',action='click',testid='NOTESTID') location
'/user/pradeepk/mytable/20091101/19174/click/NOTESTID';"
However I get errors when I try to run a query:
[pradeepk@chargesize:~/dev]hive --auxpath ult-serde.jar --config
hive_conf_without_thrift -e "select src_spaceid from
ult_search_austria_ult where datestamp='20091101';"
10/06/17 13:22:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006171322_1913647383.txt
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
java.lang.IllegalArgumentException: Can not create a Path from an empty
string
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)
at org.apache.hadoop.fs.Path.<init>(Path.java:90)
at org.apache.hadoop.fs.Path.<init>(Path.java:50)
at
org.apache.hadoop.mapred.JobClient.copyRemoteFiles(JobClient.java:523)
at
org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient
.java:603)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761)
at
org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
at
org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:684)
at
org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
Any help is much appreciated.
Pradeep
________________________________
From: Ashish Thusoo [mailto:athusoo@facebook.com]
Sent: Thursday, June 17, 2010 11:15 AM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
hmm... Can you send the exact command and also the create table command
for this table.
Ashish
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Thursday, June 17, 2010 9:09 AM
To: hive-user@hadoop.apache.org
Subject: RE: alter table add partition error
Sorry - that was a cut-paste error - I don't have the action part - so I
am specifying key-value pairs. Since what I am trying to do seems like a
basic operation, I am wondering if it's something to do with my Serde -
unfortunately the error I see gives me no clue of what could be wrong -
any help would be greatly appreciated!
Thanks,
Pradeep
________________________________
From: yq he [mailto:hhh.heyq@gmail.com]
Sent: Wednesday, June 16, 2010 5:54 PM
To: hive-user@hadoop.apache.org
Subject: Re: alter table add partition error
Hi Pradeep,
partition definition need to be key-value pairs. partition key `action`
seems missed the value.
Thanks
Yongqiang
On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath <pr...@yahoo-inc.com>
wrote:
Hi,
I am trying to create an external table against already existing
data in sequencefile format. However I have written a custom Serde to
interpret the data. I am able to create the table fine but get the
exception shown in the session output below when I try to add partition
- any help would be greatly appreciated.
Thanks,
Pradeep
== session output ===
[pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER
TABLE mytable add partition(datestamp = '20091101', srcid = '10',action)
location '/user/pradeepk/mytable/20091101/10';"
10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
found in the classpath. Usage of hadoop-site.xml is deprecated. Instead
use core-site.xml, mapred-site.xml and hdfs-site.xml to override
properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
Hive history
file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt
FAILED: Error in metadata: org.apache.thrift.TApplicationException:
get_partition failed: unknown result
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
[pradeepk@chargesize:~/dev/howl]
== session output ===
/tmp/pradeepk/hive.log has:
2010-06-16 17:09:00,841 ERROR exec.DDLTask
(SessionState.java:printError(269)) - FAILED: Error in metadata:
org.apache.thrift.TApplicationException: get_partition failed: unknown
result
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.thrift.TApplicationException: get_partition failed: unknown
result
at
org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)
at
org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:
55)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
at
org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: org.apache.thrift.TApplicationException: get_partition
failed: unknown result
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get
_partition(ThriftHiveMetastore.java:931)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_part
ition(ThriftHiveMetastore.java:899)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMe
taStoreClient.java:500)
at
org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)
... 15 more
The thrift server messages are:
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table :
db=default tbl=mytable
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store
with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize
called
10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore
10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition :
db=default tbl=mytable