You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Puneet Khatod <pu...@tavant.com> on 2012/05/31 10:35:09 UTC

Error in Hive execution on cluster : Wrong FS: hdfs

Hi,

I am facing below error when I am firing any query in Hive.
My Hive setup is present on the master node in my cluster. Hadoop is configured using IP_addresses in configuration xmls and in master, slave files and it is running fine. The error only arises when hive query is executed which had table location on HDFS. It seems hive is expecting configurations to be done using hostnames.

Please help me in configuring hive such that it can understand IP address based configuration. I am using Hadoop 0.20.2 and Hive 0.7.1.

Regards,
Puneet

____________________  ______________________  ______________________________  ______________________________

hive> show tables;
OK
test_page_view
test_page_view_stg
Time taken: 131.309 seconds
hive>
    > select * from test_page_view_stg;
FAILED: Hive Internal Error: java.lang.RuntimeException(Error while making MR scratch directory - check filesystem config (null))
java.lang.RuntimeException: Error while making MR scratch directory - check filesystem config (null)
        at org.apache.hadoop.hive.ql.Context.getMRScratchDir(Context.java:196)
        at org.apache.hadoop.hive.ql.Context.getMRTmpFileURI(Context.java:247)
        at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:900)
        at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:6594)
        at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:238)
        at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:340)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:736)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:164)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:241)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:456)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: java.lang.IllegalArgumentException: Wrong FS: hdfs://<IP_ADDRESS>:9100/tmp/hive-hadoop/hive_2012-05-31_11-29-19_844_3368974040204630542, expected: hdfs://<hostname>.local:9100
        at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:310)
        at org.apache.hadoop.hdfs.DistributedFileSystem.checkPath(DistributedFileSystem.java:99)
        at org.apache.hadoop.fs.FileSystem.makeQualified(FileSystem.java:222)
        at org.apache.hadoop.hdfs.DistributedFileSystem.makeQualified(DistributedFileSystem.java:116)
        at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:146)
        at org.apache.hadoop.hive.ql.Context.getMRScratchDir(Context.java:190)
        ... 14 more

Any comments or statements made in this email are not necessarily those of Tavant Technologies.
The information transmitted is intended only for the person or entity to which it is addressed and may
contain confidential and/or privileged material. If you have received this in error, please contact the
sender and delete the material from any computer. All e-mails sent from or to Tavant Technologies
may be subject to our monitoring procedures.

Re: Error in Hive execution on cluster : Wrong FS: hdfs

Posted by Abhishek Pratap Singh <ma...@gmail.com>.
Can you check the configuration in your Hadoop's core-site.xml file.
Also are you able to fire fs command from hadoop successfully by using the
fully qualified directory structure?

Seems a problem in your file system config.


Regards,
Abhishek

On Thu, May 31, 2012 at 1:35 AM, Puneet Khatod <pu...@tavant.com>wrote:

>  Hi,****
>
> ** **
>
> I am facing below error when I am firing any query in Hive.****
>
> My Hive setup is present on the master node in my cluster. Hadoop is
> configured using IP_addresses in configuration xmls and in master, slave
> files and it is running fine. The error only arises when hive query is
> executed which had table location on HDFS. It seems hive is expecting
> configurations to be done using hostnames.****
>
> ** **
>
> Please help me in configuring hive such that it can understand IP address
> based configuration. I am using Hadoop 0.20.2 and Hive 0.7.1.****
>
> ** **
>
> Regards,****
>
> Puneet****
>
> ** **
>
> ____________________  ______________________
> ______________________________  ______________________________****
>
> ** **
>
> hive> show tables;****
>
> OK****
>
> test_page_view****
>
> test_page_view_stg****
>
> Time taken: 131.309 seconds****
>
> hive>****
>
>     > select * from test_page_view_stg;****
>
> FAILED: Hive Internal Error: java.lang.RuntimeException(Error while making
> MR scratch directory - check filesystem config (null))****
>
> java.lang.RuntimeException: Error while making MR scratch directory -
> check filesystem config (null)****
>
>         at
> org.apache.hadoop.hive.ql.Context.getMRScratchDir(Context.java:196)****
>
>         at
> org.apache.hadoop.hive.ql.Context.getMRTmpFileURI(Context.java:247)****
>
>         at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:900)
> ****
>
>         at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:6594)
> ****
>
>         at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:238)
> ****
>
>         at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:340)****
>
>         at org.apache.hadoop.hive.ql.Driver.run(Driver.java:736)****
>
>         at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:164)****
>
>         at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:241)****
>
>         at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:456)**
> **
>
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)****
>
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> ****
>
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ****
>
>         at java.lang.reflect.Method.invoke(Method.java:601)****
>
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:156)****
>
> *Caused by: java.lang.IllegalArgumentException: Wrong FS:
> hdfs://<IP_ADDRESS>:9100/tmp/hive-hadoop/hive_2012-05-31_11-29-19_844_3368974040204630542,
> expected: hdfs://<hostname>.local:9100*
>
>         at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:310)*
> ***
>
>         at
> org.apache.hadoop.hdfs.DistributedFileSystem.checkPath(DistributedFileSystem.java:99)
> ****
>
>         at
> org.apache.hadoop.fs.FileSystem.makeQualified(FileSystem.java:222)****
>
>         at
> org.apache.hadoop.hdfs.DistributedFileSystem.makeQualified(DistributedFileSystem.java:116)
> ****
>
>         at
> org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:146)****
>
>         at
> org.apache.hadoop.hive.ql.Context.getMRScratchDir(Context.java:190)****
>
>         ... 14 more****
>
> ** **
>
> Any comments or statements made in this email are not necessarily those of
> Tavant Technologies.
> The information transmitted is intended only for the person or entity to
> which it is addressed and may
> contain confidential and/or privileged material. If you have received this
> in error, please contact the
> sender and delete the material from any computer. All e-mails sent from or
> to Tavant Technologies
> may be subject to our monitoring procedures.
>