You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by wenxing zheng <we...@gmail.com> on 2017/01/17 06:44:06 UTC
File not found of TEZ libraries with tez.lib.uris configuration
Dear all,
I met an issue in the TEZ configuration for HIVE, as from the HIVE logs
file:
> *Caused by: java.io.FileNotFoundException: File does not exist:
> hdfs://hdfscluster/apps/tez-0.8.4/tez.tar.gz*
> * at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1309)
> ~[hadoop-hdfs-2.7.3.jar:?]*
> * at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
> ~[hadoop-hdfs-2.7.3.jar:?]*
> * at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> ~[hadoop-common-2.7.3.jar:?]*
> * at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1301)
> ~[hadoop-hdfs-2.7.3.jar:?]*
> * at
> org.apache.hadoop.fs.FileSystem.resolvePath(FileSystem.java:753)
> ~[hadoop-common-2.7.3.jar:?]*
> * at
> org.apache.tez.client.TezClientUtils.addLocalResources(TezClientUtils.java:218)
> ~[tez-api-0.8.4.jar:0.8.4]*
> * at
> org.apache.tez.client.TezClientUtils.setupTezJarsLocalResources(TezClientUtils.java:183)
> ~[tez-api-0.8.4.jar:0.8.4]*
> * at
> org.apache.tez.client.TezClient.getTezJarResources(TezClient.java:1057)
> ~[tez-api-0.8.4.jar:0.8.4]*
> * at org.apache.tez.client.TezClient.start(TezClient.java:447)
> ~[tez-api-0.8.4.jar:0.8.4]*
> * at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.startSessionAndContainers(TezSessionState.java:396)
> ~[hive-exec-2.1.1.jar:2.1.1]*
> * at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.access$000(TezSessionState.java:97)
> ~[hive-exec-2.1.1.jar:2.1.1]*
> * at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState$1.call(TezSessionState.java:333)
> ~[hive-exec-2.1.1.jar:2.1.1]*
> * at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState$1.call(TezSessionState.java:329)
> ~[hive-exec-2.1.1.jar:2.1.1]*
> * at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_101]** at java.lang.Thread.run(Thread.java:745)
> ~[?:1.8.0_101]*
But in our configuration file, we already set the tez.lib.uris as below:
> <property>
> <name>tez.lib.uris</name>
> <value>*hdfs://hdfscluster/apps/tez-0.8.4/tez.tar.gz*
> </value>
> </property>
> <property>
> <name>hive.execution.engine</name>
> <value>tez</value>
> </property>
And we did copy the tar.gz file to the HDFS filesystem:
>
> *[hadoop@hdfs-hdp-202 tez]$ hadoop fs -ls /apps//tez-0.8.4/tez.tar.gz**-rw-r--r--
> 3 hadoop supergroup 42892768 2016-12-21 18:36
> /apps/tez-0.8.4/tez.tar.gz*
Appreciated for any advice.
Kind Regards, Wenxing
Re: File not found of TEZ libraries with tez.lib.uris configuration
Posted by Jörn Franke <jo...@gmail.com>.
Sorry never mind my previous mail... in the stack it seems to look exactly for this file. Can you try to download the file? Can you check if these are all files needed? I think you need to extract the .tar.gz and point to the jars (check the Tez web site for the confit).
> On 17 Jan 2017, at 07:44, wenxing zheng <we...@gmail.com> wrote:
>
> Dear all,
>
> I met an issue in the TEZ configuration for HIVE, as from the HIVE logs file:
>
>> Caused by: java.io.FileNotFoundException: File does not exist: hdfs://hdfscluster/apps/tez-0.8.4/tez.tar.gz
>> at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1309) ~[hadoop-hdfs-2.7.3.jar:?]
>> at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301) ~[hadoop-hdfs-2.7.3.jar:?]
>> at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) ~[hadoop-common-2.7.3.jar:?]
>> at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1301) ~[hadoop-hdfs-2.7.3.jar:?]
>> at org.apache.hadoop.fs.FileSystem.resolvePath(FileSystem.java:753) ~[hadoop-common-2.7.3.jar:?]
>> at org.apache.tez.client.TezClientUtils.addLocalResources(TezClientUtils.java:218) ~[tez-api-0.8.4.jar:0.8.4]
>> at org.apache.tez.client.TezClientUtils.setupTezJarsLocalResources(TezClientUtils.java:183) ~[tez-api-0.8.4.jar:0.8.4]
>> at org.apache.tez.client.TezClient.getTezJarResources(TezClient.java:1057) ~[tez-api-0.8.4.jar:0.8.4]
>> at org.apache.tez.client.TezClient.start(TezClient.java:447) ~[tez-api-0.8.4.jar:0.8.4]
>> at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.startSessionAndContainers(TezSessionState.java:396) ~[hive-exec-2.1.1.jar:2.1.1]
>> at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.access$000(TezSessionState.java:97) ~[hive-exec-2.1.1.jar:2.1.1]
>> at org.apache.hadoop.hive.ql.exec.tez.TezSessionState$1.call(TezSessionState.java:333) ~[hive-exec-2.1.1.jar:2.1.1]
>> at org.apache.hadoop.hive.ql.exec.tez.TezSessionState$1.call(TezSessionState.java:329) ~[hive-exec-2.1.1.jar:2.1.1]
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_101]
>> at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_101]
>
>
> But in our configuration file, we already set the tez.lib.uris as below:
>> <property>
>> <name>tez.lib.uris</name>
>> <value>hdfs://hdfscluster/apps/tez-0.8.4/tez.tar.gz</value>
>> </property>
>> <property>
>> <name>hive.execution.engine</name>
>> <value>tez</value>
>> </property>
>
>
> And we did copy the tar.gz file to the HDFS filesystem:
>> [hadoop@hdfs-hdp-202 tez]$ hadoop fs -ls /apps//tez-0.8.4/tez.tar.gz
>> -rw-r--r-- 3 hadoop supergroup 42892768 2016-12-21 18:36 /apps/tez-0.8.4/tez.tar.gz
>
>
> Appreciated for any advice.
> Kind Regards, Wenxing
Re: File not found of TEZ libraries with tez.lib.uris configuration
Posted by Jörn Franke <jo...@gmail.com>.
Maybe the wrong configuration file is picked up?
> On 17 Jan 2017, at 07:44, wenxing zheng <we...@gmail.com> wrote:
>
> Dear all,
>
> I met an issue in the TEZ configuration for HIVE, as from the HIVE logs file:
>
>> Caused by: java.io.FileNotFoundException: File does not exist: hdfs://hdfscluster/apps/tez-0.8.4/tez.tar.gz
>> at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1309) ~[hadoop-hdfs-2.7.3.jar:?]
>> at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301) ~[hadoop-hdfs-2.7.3.jar:?]
>> at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) ~[hadoop-common-2.7.3.jar:?]
>> at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1301) ~[hadoop-hdfs-2.7.3.jar:?]
>> at org.apache.hadoop.fs.FileSystem.resolvePath(FileSystem.java:753) ~[hadoop-common-2.7.3.jar:?]
>> at org.apache.tez.client.TezClientUtils.addLocalResources(TezClientUtils.java:218) ~[tez-api-0.8.4.jar:0.8.4]
>> at org.apache.tez.client.TezClientUtils.setupTezJarsLocalResources(TezClientUtils.java:183) ~[tez-api-0.8.4.jar:0.8.4]
>> at org.apache.tez.client.TezClient.getTezJarResources(TezClient.java:1057) ~[tez-api-0.8.4.jar:0.8.4]
>> at org.apache.tez.client.TezClient.start(TezClient.java:447) ~[tez-api-0.8.4.jar:0.8.4]
>> at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.startSessionAndContainers(TezSessionState.java:396) ~[hive-exec-2.1.1.jar:2.1.1]
>> at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.access$000(TezSessionState.java:97) ~[hive-exec-2.1.1.jar:2.1.1]
>> at org.apache.hadoop.hive.ql.exec.tez.TezSessionState$1.call(TezSessionState.java:333) ~[hive-exec-2.1.1.jar:2.1.1]
>> at org.apache.hadoop.hive.ql.exec.tez.TezSessionState$1.call(TezSessionState.java:329) ~[hive-exec-2.1.1.jar:2.1.1]
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_101]
>> at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_101]
>
>
> But in our configuration file, we already set the tez.lib.uris as below:
>> <property>
>> <name>tez.lib.uris</name>
>> <value>hdfs://hdfscluster/apps/tez-0.8.4/tez.tar.gz</value>
>> </property>
>> <property>
>> <name>hive.execution.engine</name>
>> <value>tez</value>
>> </property>
>
>
> And we did copy the tar.gz file to the HDFS filesystem:
>> [hadoop@hdfs-hdp-202 tez]$ hadoop fs -ls /apps//tez-0.8.4/tez.tar.gz
>> -rw-r--r-- 3 hadoop supergroup 42892768 2016-12-21 18:36 /apps/tez-0.8.4/tez.tar.gz
>
>
> Appreciated for any advice.
> Kind Regards, Wenxing