You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Jason Dere (JIRA)" <ji...@apache.org> on 2014/08/19 04:49:18 UTC
[jira] [Commented] (HIVE-7774) Issues with location path for
temporary external tables
[ https://issues.apache.org/jira/browse/HIVE-7774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14101773#comment-14101773 ]
Jason Dere commented on HIVE-7774:
----------------------------------
For non-temp tables (and temp non-external tables), it looks like Warehouse.getDnsPath() is used to properly format the table location string before the location is saved. This needs to be done for temp external tables as well.
> Issues with location path for temporary external tables
> -------------------------------------------------------
>
> Key: HIVE-7774
> URL: https://issues.apache.org/jira/browse/HIVE-7774
> Project: Hive
> Issue Type: Bug
> Reporter: Jason Dere
> Assignee: Jason Dere
>
> Depending on the location string passed into temp external table, a query requiring a map/reduce job will fail. Example:
> {noformat}
> create temporary external table tmp1 (c1 string) location '/tmp/tmp1';
> describe extended tmp1;
> select count(*) from tmp1;
> {noformat}
> Will result in the following error:
> {noformat}
> Diagnostic Messages for this Task:
> Error: java.lang.RuntimeException: Error in configuring object
> at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
> at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:426)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
> ... 9 more
> Caused by: java.lang.RuntimeException: Error in configuring object
> at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
> at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
> at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38)
> ... 14 more
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
> ... 17 more
> Caused by: java.lang.RuntimeException: Map operator initialization failed
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:154)
> ... 22 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: Configuration and input path are inconsistent
> at org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:404)
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:123)
> ... 22 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Configuration and input path are inconsistent
> at org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:398)
> ... 23 more
> FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> {noformat}
> If the location is set to 'hdfs:/tmp/tmp1', it gets the following error:
> {noformat}
> java.io.IOException: cannot find dir = hdfs://node-1.example.com:8020/tmp/tmp1/tmp1.txt in pathToPartitionInfo: [hdfs:/tmp/tmp1]
> at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getPartitionDescFromPathRecursively(HiveFileFormatUtils.java:344)
> at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getPartitionDescFromPathRecursively(HiveFileFormatUtils.java:306)
> at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat$CombineHiveInputSplit.<init>(CombineHiveInputFormat.java:108)
> at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:455)
> at org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:520)
> at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:512)
> at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:394)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
> at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:562)
> at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:557)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:557)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:548)
> at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:420)
> at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:136)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:161)
> at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1555)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1322)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1136)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:960)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:950)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:265)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:217)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:427)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:363)
> at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:460)
> at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:476)
> at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:756)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:694)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:633)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> Job Submission failed with exception 'java.io.IOException(cannot find dir = hdfs://node-1.example.com:8020/tmp/tmp1/tmp1.txt in pathToPartitionInfo: [hdfs:/tmp/tmp1])'
> FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> {noformat}
> The only location string value that works properly here is 'hdfs://hostname/tmp/tmp1'.
--
This message was sent by Atlassian JIRA
(v6.2#6252)