You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Wouter De Borger (JIRA)" <ji...@apache.org> on 2015/09/09 11:04:46 UTC
[jira] [Commented] (HIVE-11760) Make fs.azure.account.key style of
credential compatible with StorageBasedAuthorizationProvider
[ https://issues.apache.org/jira/browse/HIVE-11760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14736498#comment-14736498 ]
Wouter De Borger commented on HIVE-11760:
-----------------------------------------
upon further investigation, I can confirm that it is the StorageBasedAuthorizationProvider that causes the problem:
when I set hive.metastore.pre.event.listeners to empty, everything works as expected
> Make fs.azure.account.key style of credential compatible with StorageBasedAuthorizationProvider
> -----------------------------------------------------------------------------------------------
>
> Key: HIVE-11760
> URL: https://issues.apache.org/jira/browse/HIVE-11760
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
> Reporter: Wouter De Borger
> Priority: Minor
>
> When using Hive with Azure storage, credentials can be passed as a property to hiveserver2
> {code}
> set fs.azure.account.key.xxx.blob.core.windows.net=xxxxx
> create external table inputs(name string) STORED AS TEXTFILE LOCATION 'wasb://xx@xxx.blob.core.windows.net/’
> {code}
> By setting the key as a property, there is no need to put keys in the global hive config.
> This works as expected, except when the hive server is restarted. Then, all select queries return an error (see below).
> It would be nice if the properties were propagated to the metastore, so that it works as expected.
> {code}
> "*org.apache.hive.service.cli.HiveSQLException:Error while compiling statement: FAILED: SemanticException Unable to fetch table inputs. org.apache.hadoop.fs.azure.AzureException: org.apache.hadoop.fs.azure.AzureException: Container __container__ in account __account__.blob.core.windows.net not found, and we can't create it using anoynomous credentials.:17:16",
> 'org.apache.hive.service.cli.operation.Operation:toSQLException:Operation.java:315',
> 'org.apache.hive.service.cli.operation.SQLOperation:prepare:SQLOperation.java:112',
> 'org.apache.hive.service.cli.operation.SQLOperation:runInternal:SQLOperation.java:181',
> 'org.apache.hive.service.cli.operation.Operation:run:Operation.java:257',
> 'org.apache.hive.service.cli.session.HiveSessionImpl:executeStatementInternal:HiveSessionImpl.java:388',
> 'org.apache.hive.service.cli.session.HiveSessionImpl:executeStatementAsync:HiveSessionImpl.java:375',
> 'org.apache.hive.service.cli.CLIService:executeStatementAsync:CLIService.java:274',
> 'org.apache.hive.service.cli.thrift.ThriftCLIService:ExecuteStatement:ThriftCLIService.java:486',
> 'org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement:getResult:TCLIService.java:1313',
> 'org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement:getResult:TCLIService.java:1298',
> 'org.apache.thrift.ProcessFunction:process:ProcessFunction.java:39',
> 'org.apache.thrift.TBaseProcessor:process:TBaseProcessor.java:39',
> 'org.apache.hive.service.auth.TSetIpAddressProcessor:process:TSetIpAddressProcessor.java:56',
> 'org.apache.thrift.server.TThreadPoolServer$WorkerProcess:run:TThreadPoolServer.java:285',
> 'java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1145',
> 'java.util.concurrent.ThreadPoolExecutor$Worker:run:ThreadPoolExecutor.java:615',
> 'java.lang.Thread:run:Thread.java:745',
> "*org.apache.hadoop.hive.ql.parse.SemanticException:Unable to fetch table inputs. org.apache.hadoop.fs.azure.AzureException: org.apache.hadoop.fs.azure.AzureException: Container __container__ in account __account__.blob.core.windows.net not found, and we can't create it using anoynomous credentials.:26:10",
> 'org.apache.hadoop.hive.ql.parse.SemanticAnalyzer:getMetaData:SemanticAnalyzer.java:1868',
> 'org.apache.hadoop.hive.ql.parse.SemanticAnalyzer:getMetaData:SemanticAnalyzer.java:1545',
> 'org.apache.hadoop.hive.ql.parse.SemanticAnalyzer:genResolvedParseTree:SemanticAnalyzer.java:10077',
> 'org.apache.hadoop.hive.ql.parse.SemanticAnalyzer:analyzeInternal:SemanticAnalyzer.java:10128',
> 'org.apache.hadoop.hive.ql.parse.CalcitePlanner:analyzeInternal:CalcitePlanner.java:209',
> 'org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer:analyze:BaseSemanticAnalyzer.java:227',
> 'org.apache.hadoop.hive.ql.Driver:compile:Driver.java:424',
> 'org.apache.hadoop.hive.ql.Driver:compile:Driver.java:308',
> 'org.apache.hadoop.hive.ql.Driver:compileInternal:Driver.java:1122',
> 'org.apache.hadoop.hive.ql.Driver:compileAndRespond:Driver.java:1116',
> 'org.apache.hive.service.cli.operation.SQLOperation:prepare:SQLOperation.java:110',
> "*org.apache.hadoop.hive.ql.metadata.HiveException:Unable to fetch table inputs. org.apache.hadoop.fs.azure.AzureException: org.apache.hadoop.fs.azure.AzureException: Container __container__ in account __account__.blob.core.windows.net not found, and we can't create it using anoynomous credentials.:28:2",
> 'org.apache.hadoop.hive.ql.metadata.Hive:getTable:Hive.java:1123',
> 'org.apache.hadoop.hive.ql.metadata.Hive:getTable:Hive.java:1070',
> 'org.apache.hadoop.hive.ql.parse.SemanticAnalyzer:getMetaData:SemanticAnalyzer.java:1574',
>
> "*org.apache.hadoop.hive.metastore.api.MetaException:org.apache.hadoop.fs.azure.AzureException: org.apache.hadoop.fs.azure.AzureException: Container __container__ in account __account__.blob.core.windows.net not found, and we can't create it using anoynomous credentials.:47:19",
> 'org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener:metaException:AuthorizationPreEventListener.java:411',
> 'org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener:authorizeReadTable:AuthorizationPreEventListener.java:193',
> 'org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener:onEvent:AuthorizationPreEventListener.java:149',
> 'org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler:firePreEvent:HiveMetaStore.java:1992',
> 'org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler:get_table:HiveMetaStore.java:1779',
> 'sun.reflect.NativeMethodAccessorImpl:invoke0:NativeMethodAccessorImpl.java:-2',
> 'sun.reflect.NativeMethodAccessorImpl:invoke:NativeMethodAccessorImpl.java:57',
> 'sun.reflect.DelegatingMethodAccessorImpl:invoke:DelegatingMethodAccessorImpl.java:43',
> 'java.lang.reflect.Method:invoke:Method.java:606',
> 'org.apache.hadoop.hive.metastore.RetryingHMSHandler:invoke:RetryingHMSHandler.java:107',
> 'com.sun.proxy.$Proxy7:get_table::-1',
> 'org.apache.hadoop.hive.metastore.HiveMetaStoreClient:getTable:HiveMetaStoreClient.java:1208',
> 'org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient:getTable:SessionHiveMetaStoreClient.java:131',
> 'sun.reflect.NativeMethodAccessorImpl:invoke0:NativeMethodAccessorImpl.java:-2',
> 'sun.reflect.NativeMethodAccessorImpl:invoke:NativeMethodAccessorImpl.java:57',
> 'sun.reflect.DelegatingMethodAccessorImpl:invoke:DelegatingMethodAccessorImpl.java:43',
> 'java.lang.reflect.Method:invoke:Method.java:606',
> 'org.apache.hadoop.hive.metastore.RetryingMetaStoreClient:invoke:RetryingMetaStoreClient.java:156',
> 'com.sun.proxy.$Proxy8:getTable::-1',
> 'org.apache.hadoop.hive.ql.metadata.Hive:getTable:Hive.java:1115',
>
> "*org.apache.hadoop.hive.ql.metadata.HiveException:org.apache.hadoop.fs.azure.AzureException: org.apache.hadoop.fs.azure.AzureException: Container __container__ in account __account__.blob.core.windows.net not found, and we can't create it using anoynomous credentials.:49:3",
> 'org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider:hiveException:StorageBasedAuthorizationProvider.java:427',
> 'org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider:authorize:StorageBasedAuthorizationProvider.java:346',
> 'org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider:authorize:StorageBasedAuthorizationProvider.java:188',
> 'org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener:authorizeReadTable:AuthorizationPreEventListener.java:188',
> "*org.apache.hadoop.fs.azure.AzureException:org.apache.hadoop.fs.azure.AzureException: Container __container__ in account __account__.blob.core.windows.net not found, and we can't create it using anoynomous credentials.:58:10",
> 'org.apache.hadoop.fs.azure.AzureNativeFileSystemStore:createAzureStorageSession:AzureNativeFileSystemStore.java:943',
> 'org.apache.hadoop.fs.azure.AzureNativeFileSystemStore:initialize:AzureNativeFileSystemStore.java:440',
> 'org.apache.hadoop.fs.azure.NativeAzureFileSystem:initialize:NativeAzureFileSystem.java:1048',
> 'org.apache.hadoop.fs.FileSystem:createFileSystem:FileSystem.java:2653',
> 'org.apache.hadoop.fs.FileSystem:access$200:FileSystem.java:92',
> 'org.apache.hadoop.fs.FileSystem$Cache:getInternal:FileSystem.java:2687',
> 'org.apache.hadoop.fs.FileSystem$Cache:get:FileSystem.java:2669',
> 'org.apache.hadoop.fs.FileSystem:get:FileSystem.java:371',
> 'org.apache.hadoop.fs.Path:getFileSystem:Path.java:295',
> 'org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider:checkPermissions:StorageBasedAuthorizationProvider.java:362',
> 'org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider:authorize:StorageBasedAuthorizationProvider.java:339',
>
> "*org.apache.hadoop.fs.azure.AzureException:Container __container__ in account __account__.blob.core.windows.net not found, and we can't create it using anoynomous credentials.:59:1",
> 'org.apache.hadoop.fs.azure.AzureNativeFileSystemStore:connectUsingAnonymousCredentials:AzureNativeFileSystemStore.java:735',
> 'org.apache.hadoop.fs.azure.AzureNativeFileSystemStore:createAzureStorageSession:AzureNativeFileSystemStore.java:938'
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)