You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2021/11/23 05:24:00 UTC
[jira] [Commented] (IMPALA-11025) Creation of functional.insert_only_transactional_table fails wIth 'illegal location for managed table'
[ https://issues.apache.org/jira/browse/IMPALA-11025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17447781#comment-17447781 ]
ASF subversion and git services commented on IMPALA-11025:
----------------------------------------------------------
Commit ee03727971f379fe8fb3161387783ce00a9f6b9d in impala's branch refs/heads/master from Andrew Sherman
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=ee03727 ]
IMPALA-11025: Transactional tables should use /test-warehouse/managed/databasename.db
Recent Hive releases seem to be enforcing that data for a managed table
is stored under the hive.metastore.warehouse.dir path property in a
folder path similar to databasename.db/tablename - see
https://cwiki.apache.org/confluence/display/Hive/Managed+vs.+External+Tables
Use this form /test-warehouse/managed/databasename.db in
generate-schema-statements.py when creating transactional tables.
Testing:
- A few small changes to tests that verify filesystem changes for acid
tables.
- Exhaustive tests pass.
Change-Id: Ib870ca802c9fa180e6be7a6f65bef35b227772db
Reviewed-on: http://gerrit.cloudera.org:8080/18046
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> Creation of functional.insert_only_transactional_table fails wIth 'illegal location for managed table'
> -------------------------------------------------------------------------------------------------------
>
> Key: IMPALA-11025
> URL: https://issues.apache.org/jira/browse/IMPALA-11025
> Project: IMPALA
> Issue Type: Bug
> Reporter: Andrew Sherman
> Assignee: Andrew Sherman
> Priority: Critical
> Attachments: IMPALA-11025_stack.txt
>
>
> Hive complains 'Illegal location for managed table' although location '/test-warehouse/managed/insert_only_transactional_table' appears to be within database's managed location.
> {code}
> INFO : Compiling command(queryId=jenkins_20211116193803_e342124e-7a94-4024-b11a-58578cdf2ce4): CREATE TABLE IF NOT EXISTS functional.insert_only_transactional_table (
> col1 int
> )
> STORED AS TEXTFILE
> LOCATION '/test-warehouse/managed/insert_only_transactional_table'
> TBLPROPERTIES (
> 'transactional_properties' = 'insert_only',
> 'transactional' = 'true'
> )
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Created Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO : Completed compiling command(queryId=jenkins_20211116193803_e342124e-7a94-4024-b11a-58578cdf2ce4); Time taken: 0.025 seconds
> INFO : Executing command(queryId=jenkins_20211116193803_e342124e-7a94-4024-b11a-58578cdf2ce4): CREATE TABLE IF NOT EXISTS functional.insert_only_transactional_table (
> col1 int
> )
> STORED AS TEXTFILE
> LOCATION '/test-warehouse/managed/insert_only_transactional_table'
> TBLPROPERTIES (
> 'transactional_properties' = 'insert_only',
> 'transactional' = 'true'
> )
> INFO : Starting task [Stage-0:DDL] in serial mode
> ERROR : Failed
> org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Illegal location for managed table, it has to be within database's managed location)
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:1170) ~[hive-exec-3.1.3000.7.1.8.0-393.jar:3.1.3000.7.1.8.0-393]
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:1175) ~[hive-exec-3.1.3000.7.1.8.0-393.jar:3.1.3000.7.1.8.0-393]
> {code}
> (For full hive stack see attachment)
> HMS log:
> {code}
> 2021-11-16T19:38:03,185 INFO [pool-9-thread-58] metastore.MetastoreDefaultTransformer: Starting translation for transformDatabase for processor HMSClient-@localhost with [EXTWRITE, EXTREAD, HIVEBUCKET2, HIVEFULLACIDREAD, HIVEFULLACIDWRITE, HIVECACHEINVALIDATE, HIVEMANAGESTATS, HIVEMANAGEDINSERTWRITE, HIVEMANAGEDINSERTREAD, HIVESQL, HIVEMQT, HIVEONLYMQTWRITE] on database functional locationUri=hdfs://localhost:20500/test-warehouse/functional.db managedLocationUri=hdfs://localhost:20500/test-warehouse/managed/functional.db
> 2021-11-16T19:38:03,185 INFO [pool-9-thread-58] metastore.MetastoreDefaultTransformer: Transformer returning database:Database(name:functional, description:null, locationUri:hdfs://localhost:20500/test-warehouse/functional.db, parameters:{}, ownerName:jenkins, ownerType:USER, catalogName:hive, createTime:1637119984, managedLocationUri:hdfs://localhost:20500/test-warehouse/managed/functional.db)
> 2021-11-16T19:38:03,323 INFO [pool-9-thread-58] metastore.HiveMetaStore: 63: source:127.0.0.1 create_table_req: Table(tableName:insert_only_transactional_table, dbName:functional, owner:jenkins, createTime:1637120283, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:col1, type:int, comment:null)], location:hdfs://localhost:20500/test-warehouse/managed/insert_only_transactional_table, inputFormat:org.apache.hadoop.mapred.TextInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, parameters:{serialization.format=1}), bucketCols:[], sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], skewedColValueLocationMaps:{}), storedAsSubDirectories:false), partitionKeys:[], parameters:{bucketing_version=2, transactional_properties=insert_only, transactional=true}, viewOriginalText:null, viewExpandedText:null, tableType:MANAGED_TABLE, privileges:PrincipalPrivilegeSet(userPrivileges:{jenkins=[PrivilegeGrantInfo(privilege:INSERT, createTime:-1, grantor:jenkins, grantorType:USER, grantOption:true), PrivilegeGrantInfo(privilege:SELECT, createTime:-1, grantor:jenkins, grantorType:USER, grantOption:true), PrivilegeGrantInfo(privilege:UPDATE, createTime:-1, grantor:jenkins, grantorType:USER, grantOption:true), PrivilegeGrantInfo(privilege:DELETE, createTime:-1, grantor:jenkins, grantorType:USER, grantOption:true)]}, groupPrivileges:null, rolePrivileges:null), temporary:false, catName:hive, ownerType:USER, writeId:0)
> 2021-11-16T19:38:03,323 INFO [pool-9-thread-58] HiveMetaStore.audit: ugi=jenkins ip=127.0.0.1 cmd=source:127.0.0.1 create_table_req: Table(tableName:insert_only_transactional_table, dbName:functional, owner:jenkins, createTime:1637120283, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:col1, type:int, comment:null)], location:hdfs://localhost:20500/test-warehouse/managed/insert_only_transactional_table, inputFormat:org.apache.hadoop.mapred.TextInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, parameters:{serialization.format=1}), bucketCols:[], sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], skewedColValueLocationMaps:{}), storedAsSubDirectories:false), partitionKeys:[], parameters:{bucketing_version=2, transactional_properties=insert_only, transactional=true}, viewOriginalText:null, viewExpandedText:null, tableType:MANAGED_TABLE, privileges:PrincipalPrivilegeSet(userPrivileges:{jenkins=[PrivilegeGrantInfo(privilege:INSERT, createTime:-1, grantor:jenkins, grantorType:USER, grantOption:true), PrivilegeGrantInfo(privilege:SELECT, createTime:-1, grantor:jenkins, grantorType:USER, grantOption:true), PrivilegeGrantInfo(privilege:UPDATE, createTime:-1, grantor:jenkins, grantorType:USER, grantOption:true), PrivilegeGrantInfo(privilege:DELETE, createTime:-1, grantor:jenkins, grantorType:USER, grantOption:true)]}, groupPrivileges:null, rolePrivileges:null), temporary:false, catName:hive, ownerType:USER, writeId:0)
> 2021-11-16T19:38:03,323 INFO [pool-9-thread-58] metastore.MetastoreDefaultTransformer: Starting translation for CreateTable for processor HMSClient-@localhost with [EXTWRITE, EXTREAD, HIVEBUCKET2, HIVEFULLACIDREAD, HIVEFULLACIDWRITE, HIVECACHEINVALIDATE, HIVEMANAGESTATS, HIVEMANAGEDINSERTWRITE, HIVEMANAGEDINSERTREAD, HIVESQL, HIVEMQT, HIVEONLYMQTWRITE] on table insert_only_transactional_table
> 2021-11-16T19:38:03,327 ERROR [pool-9-thread-58] metastore.RetryingHMSHandler: MetaException(message:Illegal location for managed table, it has to be within database's managed location)
> at org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer.validateTablePaths(MetastoreDefaultTransformer.java:886)
> at org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer.transformCreateTable(MetastoreDefaultTransformer.java:666)
> at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:2234)
> at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_req(HiveMetaStore.java:2544)
> at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy34.create_table_req(Unknown Source)
> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_req.getResult(ThriftHiveMetastore.java:16877)
> at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_req.getResult(ThriftHiveMetastore.java:16856)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)
> at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
> at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
> at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:313)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org