You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by "Garg, Rinku" <Ri...@fisglobal.com> on 2013/09/16 10:03:28 UTC
Issue while quering Hive
Hi All,
I have setup Hadoop, hive setup and trying to load gzip file in hadoop cluster. Files are loaded successfully and can be view on web UI. While executing Select query it gives me the below mentioned error.
ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:nxtbig (auth:SIMPLE) cause:java.io.IOException: java.lang.reflect.InvocationTargetException
2013-09-16 09:11:18,971 WARN org.apache.hadoop.mapred.Child: Error running child
java.io.IOException: java.lang.reflect.InvocationTargetException
at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:369)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.<init>(HadoopShimsSecure.java:316)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(HadoopShimsSecure.java:430)
at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:540)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:395)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1407)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:355)
... 10 more
Caused by: java.io.IOException: hdfs://localhost:54310/user/hive/warehouse/cpj_tbl/cpj.csv.gz not a SequenceFile
at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1805)
at org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:1765)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1714)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
at org.apache.hadoop.mapred.SequenceFileRecordReader.<init>(SequenceFileRecordReader.java:43)
at org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:59)
at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.<init>(CombineHiveRecordReader.java:65)
... 15 more
Can anybody help me on this.
Thanks & Regards,
Rinku Garg
_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.
Hiveserver2 Authentication (openLDAP) and Authorization (using
Sentry)
Posted by Sanjay Subramanian <Sa...@wizecommerce.com>.
Hi guys
************************************************************************************************************************************
DISCLAIMER
==========
I have no affiliations to Cloudera and I am writing this mail of my own free will, with the hope to help fellow Hive users who will be implementing security around Hive
************************************************************************************************************************************
I have implemented Hiveserver2 Authentication (openLDAP) and Authorization (using Cloudera Sentry).
I am super-excited because we know can open our Hive Data Platform in "read only" mode to remote clients in the company and SAS clients !
We are using Cloudera Manager Standard distribution (still can't afford support costs !)
Thanks
Sanjay
The stack I am planning to push to production after my hive/yarn/oozie tests succeed are as follows
Group 1 (CDH4)
Hosts
dev-thdp5.corp.nextag.com
Component Version CDH Version
Impala 1.1.1 Not applicable
Yarn (CDH4 only) 2.0.0+1475 CDH4
HDFS (CDH4 only) 2.0.0+1475 CDH4
Hue Plugins 2.5.0+139 CDH4
Sqoop2 (CDH4 only) 1.99.2+85 CDH4
MapReduce 2 (CDH4 only) 2.0.0+1475 CDH4
HBase 0.94.6+132 CDH4
HCatalog (CDH4 only) Unavailable Not installed or path incorrect
Oozie 3.3.2+92 CDH4
Zookeeper 3.4.5+23 CDH4
Hue 2.5.0+139 CDH4
MapReduce 1 (CDH4 only) 2.0.0+1475 CDH4
Pig 0.11.0+33 CDH4
HttpFS (CDH4 only) 2.0.0+1475 CDH4
Hadoop 2.0.0+1475 CDH4
Hive 0.10.0+198 CDH4
Lily HBase Indexer (CDH4 only) 1.2+2 CDH4
Solr (CDH4 only) 4.4.0+69 CDH4
Flume NG 1.4.0+23 CDH4
Java java version "1.6.0_31" Java(TM) SE Runtime Environment (build 1.6.0_31-b04) Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01, mixed mode) Not applicable
Cloudera Manager Agent 4.7.0 Not applicable
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
Re: Issue while quering Hive
Posted by Sanjay Subramanian <Sa...@wizecommerce.com>.
With regards to splitting an compression there are 2 options really as of now
If u r using Sequence Files , then Snappy
If u r using TXT files then LZO us great (u have to cross a few minor hoops to get LZO to work and I can provide guidance on that)
Please don't use GZ (not splittable) / or worse BZ2 (took slow to compress/decompress) - too slow for comfort
The only compelling reason u want to use GZIP as I am using in production is that my log files are MULTIPLE LINES…so if I use regular TXT files then splitting can happen between records
sanjay
From: Nitin Pawar <ni...@gmail.com>>
Reply-To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Date: Monday, September 16, 2013 5:07 AM
To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Subject: Re: Issue while quering Hive
As per my understanding, hadoop 1.x does not provide you any help on processing compressing files in parallel manner. (Atleast this was the case few months back).
This bzip2 splitting etc is added in hadoop2.x as per my understanding.
On Mon, Sep 16, 2013 at 5:18 PM, Garg, Rinku <Ri...@fisglobal.com>> wrote:
Thanks Nitin,
That way it worked, But in that case Hadoop will not be able to split my file into chunks/blocks and run multiple maps in parallel. This can cause under-utilization of my cluster's 'mapping' power. Is that rue??
Thanks & Regards,
Rinku Garg
From: Nitin Pawar [mailto:nitinpawar432@gmail.com<ma...@gmail.com>]
Sent: 16 September 2013 15:57
To: user@hive.apache.org<ma...@hive.apache.org>
Subject: Re: Issue while quering Hive
Does your .gz file contains the data in sequencefile ? or its a plain csv?
I think looking at the filename its a plain csv file, so I would recommend that you create a normal table with TextInputFormat (the default) and load data in the new table and give it a try.
On Mon, Sep 16, 2013 at 3:36 PM, Garg, Rinku <Ri...@fisglobal.com>> wrote:
Hi Nitin,
Yes, I created the table with sequencefile.
Thanks & Regards,
Rinku Garg
From: Nitin Pawar [mailto:nitinpawar432@gmail.com<ma...@gmail.com>]
Sent: 16 September 2013 14:19
To: user@hive.apache.org<ma...@hive.apache.org>
Subject: Re: Issue while quering Hive
Look at the error message
Caused by: java.io.IOException: hdfs://localhost:54310/user/hive/warehouse/cpj_tbl/cpj.csv.gz not a SequenceFile
Did you create table with sequencefile ?
On Mon, Sep 16, 2013 at 1:33 PM, Garg, Rinku <Ri...@fisglobal.com>> wrote:
Hi All,
I have setup Hadoop, hive setup and trying to load gzip file in hadoop cluster. Files are loaded successfully and can be view on web UI. While executing Select query it gives me the below mentioned error.
ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:nxtbig (auth:SIMPLE) cause:java.io.IOException: java.lang.reflect.InvocationTargetException
2013-09-16 09:11:18,971 WARN org.apache.hadoop.mapred.Child: Error running child
java.io.IOException: java.lang.reflect.InvocationTargetException
at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:369)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.<init>(HadoopShimsSecure.java:316)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(HadoopShimsSecure.java:430)
at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:540)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:395)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1407)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:355)
... 10 more
Caused by: java.io.IOException: hdfs://localhost:54310/user/hive/warehouse/cpj_tbl/cpj.csv.gz not a SequenceFile
at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1805)
at org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:1765)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1714)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
at org.apache.hadoop.mapred.SequenceFileRecordReader.<init>(SequenceFileRecordReader.java:43)
at org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:59)
at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.<init>(CombineHiveRecordReader.java:65)
... 15 more
Can anybody help me on this.
Thanks & Regards,
Rinku Garg
_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.
--
Nitin Pawar
_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.
--
Nitin Pawar
_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.
--
Nitin Pawar
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
Re: Issue while quering Hive
Posted by Nitin Pawar <ni...@gmail.com>.
As per my understanding, hadoop 1.x does not provide you any help on
processing compressing files in parallel manner. (Atleast this was the case
few months back).
This bzip2 splitting etc is added in hadoop2.x as per my understanding.
On Mon, Sep 16, 2013 at 5:18 PM, Garg, Rinku <Ri...@fisglobal.com>wrote:
> Thanks Nitin,****
>
> ** **
>
> That way it worked, But in that case Hadoop will not be able to split my
> file into chunks/blocks and run multiple maps in parallel. This can cause
> under-utilization of my cluster's 'mapping' power. Is that rue??****
>
> ** **
>
> Thanks & Regards,****
>
> *Rinku Garg*
>
> ****
>
> *From:* Nitin Pawar [mailto:nitinpawar432@gmail.com]
> *Sent:* 16 September 2013 15:57
>
> *To:* user@hive.apache.org
> *Subject:* Re: Issue while quering Hive****
>
> ** **
>
> Does your .gz file contains the data in sequencefile ? or its a plain csv?
> ****
>
> ** **
>
> I think looking at the filename its a plain csv file, so I would recommend
> that you create a normal table with TextInputFormat (the default) and load
> data in the new table and give it a try. ****
>
> ** **
>
> ** **
>
> On Mon, Sep 16, 2013 at 3:36 PM, Garg, Rinku <Ri...@fisglobal.com>
> wrote:****
>
> Hi Nitin,****
>
> ****
>
> Yes, I created the table with sequencefile.****
>
> ****
>
> Thanks & Regards,****
>
> *Rinku Garg*****
>
> ****
>
> ****
>
> ****
>
> *From:* Nitin Pawar [mailto:nitinpawar432@gmail.com]
> *Sent:* 16 September 2013 14:19
> *To:* user@hive.apache.org
> *Subject:* Re: Issue while quering Hive****
>
> ****
>
> Look at the error message ****
>
> ****
>
> Caused by: java.io.IOException:
> hdfs://localhost:54310/user/hive/warehouse/cpj_tbl/cpj.csv.gz not a
> SequenceFile****
>
> ****
>
> Did you create table with sequencefile ? ****
>
> ****
>
> On Mon, Sep 16, 2013 at 1:33 PM, Garg, Rinku <Ri...@fisglobal.com>
> wrote:****
>
> Hi All,****
>
> ****
>
> I have setup Hadoop, hive setup and trying to load gzip file in hadoop
> cluster. Files are loaded successfully and can be view on web UI. While
> executing Select query it gives me the below mentioned error.****
>
> ****
>
> ERROR org.apache.hadoop.security.UserGroupInformation:
> PriviledgedActionException as:nxtbig (auth:SIMPLE)
> cause:java.io.IOException: java.lang.reflect.InvocationTargetException****
>
> 2013-09-16 09:11:18,971 WARN org.apache.hadoop.mapred.Child: Error running
> child****
>
> java.io.IOException: java.lang.reflect.InvocationTargetException****
>
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
> ****
>
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
> ****
>
> at
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:369)
> ****
>
> at
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.<init>(HadoopShimsSecure.java:316)
> ****
>
> at
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(HadoopShimsSecure.java:430)
> ****
>
> at
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:540)
> ****
>
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:395)
> ****
>
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333)****
>
> at org.apache.hadoop.mapred.Child$4.run(Child.java:268)****
>
> at java.security.AccessController.doPrivileged(Native Method)****
>
> at javax.security.auth.Subject.doAs(Subject.java:396)****
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1407)
> ****
>
> at org.apache.hadoop.mapred.Child.main(Child.java:262)****
>
> Caused by: java.lang.reflect.InvocationTargetException****
>
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)****
>
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> ****
>
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> ****
>
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> ****
>
> at
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:355)
> ****
>
> ... 10 more****
>
> Caused by: java.io.IOException:
> hdfs://localhost:54310/user/hive/warehouse/cpj_tbl/cpj.csv.gz not a
> SequenceFile****
>
> at
> org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1805)****
>
> at
> org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:1765)
> ****
>
> at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1714)***
> *
>
> at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)***
> *
>
> at
> org.apache.hadoop.mapred.SequenceFileRecordReader.<init>(SequenceFileRecordReader.java:43)
> ****
>
> at
> org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:59)
> ****
>
> at
> org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.<init>(CombineHiveRecordReader.java:65)
> ****
>
> ... 15 more****
>
> ****
>
> Can anybody help me on this.****
>
> ****
>
> Thanks & Regards,****
>
> *Rinku Garg*****
>
> ****
>
> ****
>
> _____________
> The information contained in this message is proprietary and/or
> confidential. If you are not the intended recipient, please: (i) delete the
> message and all copies; (ii) do not disclose, distribute or use the message
> in any manner; and (iii) notify the sender immediately. In addition, please
> be aware that any message addressed to our domain is subject to archiving
> and review by persons other than the intended recipient. Thank you.****
>
>
>
> ****
>
> ****
>
> --
> Nitin Pawar****
>
> _____________
> The information contained in this message is proprietary and/or
> confidential. If you are not the intended recipient, please: (i) delete the
> message and all copies; (ii) do not disclose, distribute or use the message
> in any manner; and (iii) notify the sender immediately. In addition, please
> be aware that any message addressed to our domain is subject to archiving
> and review by persons other than the intended recipient. Thank you.****
>
>
>
> ****
>
> ** **
>
> --
> Nitin Pawar****
> _____________
> The information contained in this message is proprietary and/or
> confidential. If you are not the intended recipient, please: (i) delete the
> message and all copies; (ii) do not disclose, distribute or use the message
> in any manner; and (iii) notify the sender immediately. In addition, please
> be aware that any message addressed to our domain is subject to archiving
> and review by persons other than the intended recipient. Thank you.
>
--
Nitin Pawar
RE: Issue while quering Hive
Posted by "Garg, Rinku" <Ri...@fisglobal.com>.
Thanks Nitin,
That way it worked, But in that case Hadoop will not be able to split my file into chunks/blocks and run multiple maps in parallel. This can cause under-utilization of my cluster's 'mapping' power. Is that rue??
Thanks & Regards,
Rinku Garg
From: Nitin Pawar [mailto:nitinpawar432@gmail.com]
Sent: 16 September 2013 15:57
To: user@hive.apache.org
Subject: Re: Issue while quering Hive
Does your .gz file contains the data in sequencefile ? or its a plain csv?
I think looking at the filename its a plain csv file, so I would recommend that you create a normal table with TextInputFormat (the default) and load data in the new table and give it a try.
On Mon, Sep 16, 2013 at 3:36 PM, Garg, Rinku <Ri...@fisglobal.com>> wrote:
Hi Nitin,
Yes, I created the table with sequencefile.
Thanks & Regards,
Rinku Garg
From: Nitin Pawar [mailto:nitinpawar432@gmail.com<ma...@gmail.com>]
Sent: 16 September 2013 14:19
To: user@hive.apache.org<ma...@hive.apache.org>
Subject: Re: Issue while quering Hive
Look at the error message
Caused by: java.io.IOException: hdfs://localhost:54310/user/hive/warehouse/cpj_tbl/cpj.csv.gz not a SequenceFile
Did you create table with sequencefile ?
On Mon, Sep 16, 2013 at 1:33 PM, Garg, Rinku <Ri...@fisglobal.com>> wrote:
Hi All,
I have setup Hadoop, hive setup and trying to load gzip file in hadoop cluster. Files are loaded successfully and can be view on web UI. While executing Select query it gives me the below mentioned error.
ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:nxtbig (auth:SIMPLE) cause:java.io.IOException: java.lang.reflect.InvocationTargetException
2013-09-16 09:11:18,971 WARN org.apache.hadoop.mapred.Child: Error running child
java.io.IOException: java.lang.reflect.InvocationTargetException
at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:369)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.<init>(HadoopShimsSecure.java:316)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(HadoopShimsSecure.java:430)
at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:540)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:395)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1407)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:355)
... 10 more
Caused by: java.io.IOException: hdfs://localhost:54310/user/hive/warehouse/cpj_tbl/cpj.csv.gz not a SequenceFile
at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1805)
at org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:1765)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1714)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
at org.apache.hadoop.mapred.SequenceFileRecordReader.<init>(SequenceFileRecordReader.java:43)
at org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:59)
at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.<init>(CombineHiveRecordReader.java:65)
... 15 more
Can anybody help me on this.
Thanks & Regards,
Rinku Garg
_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.
--
Nitin Pawar
_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.
--
Nitin Pawar
_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.
Re: Issue while quering Hive
Posted by Nitin Pawar <ni...@gmail.com>.
Does your .gz file contains the data in sequencefile ? or its a plain csv?
I think looking at the filename its a plain csv file, so I would recommend
that you create a normal table with TextInputFormat (the default) and load
data in the new table and give it a try.
On Mon, Sep 16, 2013 at 3:36 PM, Garg, Rinku <Ri...@fisglobal.com>wrote:
> Hi Nitin,****
>
> ** **
>
> Yes, I created the table with sequencefile.****
>
> ** **
>
> Thanks & Regards,****
>
> *Rinku Garg*
>
> ****
>
> ** **
>
> ** **
>
> ** **
>
> *From:* Nitin Pawar [mailto:nitinpawar432@gmail.com]
> *Sent:* 16 September 2013 14:19
> *To:* user@hive.apache.org
> *Subject:* Re: Issue while quering Hive****
>
> ** **
>
> Look at the error message ****
>
> ** **
>
> Caused by: java.io.IOException:
> hdfs://localhost:54310/user/hive/warehouse/cpj_tbl/cpj.csv.gz not a
> SequenceFile****
>
> ** **
>
> Did you create table with sequencefile ? ****
>
> ** **
>
> On Mon, Sep 16, 2013 at 1:33 PM, Garg, Rinku <Ri...@fisglobal.com>
> wrote:****
>
> Hi All,****
>
> ****
>
> I have setup Hadoop, hive setup and trying to load gzip file in hadoop
> cluster. Files are loaded successfully and can be view on web UI. While
> executing Select query it gives me the below mentioned error.****
>
> ****
>
> ERROR org.apache.hadoop.security.UserGroupInformation:
> PriviledgedActionException as:nxtbig (auth:SIMPLE)
> cause:java.io.IOException: java.lang.reflect.InvocationTargetException****
>
> 2013-09-16 09:11:18,971 WARN org.apache.hadoop.mapred.Child: Error running
> child****
>
> java.io.IOException: java.lang.reflect.InvocationTargetException****
>
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
> ****
>
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
> ****
>
> at
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:369)
> ****
>
> at
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.<init>(HadoopShimsSecure.java:316)
> ****
>
> at
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(HadoopShimsSecure.java:430)
> ****
>
> at
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:540)
> ****
>
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:395)
> ****
>
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333)****
>
> at org.apache.hadoop.mapred.Child$4.run(Child.java:268)****
>
> at java.security.AccessController.doPrivileged(Native Method)****
>
> at javax.security.auth.Subject.doAs(Subject.java:396)****
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1407)
> ****
>
> at org.apache.hadoop.mapred.Child.main(Child.java:262)****
>
> Caused by: java.lang.reflect.InvocationTargetException****
>
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)****
>
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> ****
>
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> ****
>
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> ****
>
> at
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:355)
> ****
>
> ... 10 more****
>
> Caused by: java.io.IOException:
> hdfs://localhost:54310/user/hive/warehouse/cpj_tbl/cpj.csv.gz not a
> SequenceFile****
>
> at
> org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1805)****
>
> at
> org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:1765)
> ****
>
> at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1714)***
> *
>
> at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)***
> *
>
> at
> org.apache.hadoop.mapred.SequenceFileRecordReader.<init>(SequenceFileRecordReader.java:43)
> ****
>
> at
> org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:59)
> ****
>
> at
> org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.<init>(CombineHiveRecordReader.java:65)
> ****
>
> ... 15 more****
>
> ****
>
> Can anybody help me on this.****
>
> ****
>
> Thanks & Regards,****
>
> *Rinku Garg*****
>
> ****
>
> ****
>
> _____________
> The information contained in this message is proprietary and/or
> confidential. If you are not the intended recipient, please: (i) delete the
> message and all copies; (ii) do not disclose, distribute or use the message
> in any manner; and (iii) notify the sender immediately. In addition, please
> be aware that any message addressed to our domain is subject to archiving
> and review by persons other than the intended recipient. Thank you.****
>
>
>
> ****
>
> ** **
>
> --
> Nitin Pawar****
> _____________
> The information contained in this message is proprietary and/or
> confidential. If you are not the intended recipient, please: (i) delete the
> message and all copies; (ii) do not disclose, distribute or use the message
> in any manner; and (iii) notify the sender immediately. In addition, please
> be aware that any message addressed to our domain is subject to archiving
> and review by persons other than the intended recipient. Thank you.
>
--
Nitin Pawar
RE: Issue while quering Hive
Posted by "Garg, Rinku" <Ri...@fisglobal.com>.
Hi Nitin,
Yes, I created the table with sequencefile.
Thanks & Regards,
Rinku Garg
From: Nitin Pawar [mailto:nitinpawar432@gmail.com]
Sent: 16 September 2013 14:19
To: user@hive.apache.org
Subject: Re: Issue while quering Hive
Look at the error message
Caused by: java.io.IOException: hdfs://localhost:54310/user/hive/warehouse/cpj_tbl/cpj.csv.gz not a SequenceFile
Did you create table with sequencefile ?
On Mon, Sep 16, 2013 at 1:33 PM, Garg, Rinku <Ri...@fisglobal.com>> wrote:
Hi All,
I have setup Hadoop, hive setup and trying to load gzip file in hadoop cluster. Files are loaded successfully and can be view on web UI. While executing Select query it gives me the below mentioned error.
ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:nxtbig (auth:SIMPLE) cause:java.io.IOException: java.lang.reflect.InvocationTargetException
2013-09-16 09:11:18,971 WARN org.apache.hadoop.mapred.Child: Error running child
java.io.IOException: java.lang.reflect.InvocationTargetException
at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:369)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.<init>(HadoopShimsSecure.java:316)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(HadoopShimsSecure.java:430)
at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:540)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:395)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1407)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:355)
... 10 more
Caused by: java.io.IOException: hdfs://localhost:54310/user/hive/warehouse/cpj_tbl/cpj.csv.gz not a SequenceFile
at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1805)
at org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:1765)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1714)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
at org.apache.hadoop.mapred.SequenceFileRecordReader.<init>(SequenceFileRecordReader.java:43)
at org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:59)
at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.<init>(CombineHiveRecordReader.java:65)
... 15 more
Can anybody help me on this.
Thanks & Regards,
Rinku Garg
_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.
--
Nitin Pawar
_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.
Re: Issue while quering Hive
Posted by Nitin Pawar <ni...@gmail.com>.
Look at the error message
Caused by: java.io.IOException: hdfs://localhost:54310/user/
hive/warehouse/cpj_tbl/cpj.csv.gz not a SequenceFile
Did you create table with sequencefile ?
On Mon, Sep 16, 2013 at 1:33 PM, Garg, Rinku <Ri...@fisglobal.com>wrote:
> Hi All,****
>
> ** **
>
> I have setup Hadoop, hive setup and trying to load gzip file in hadoop
> cluster. Files are loaded successfully and can be view on web UI. While
> executing Select query it gives me the below mentioned error.****
>
> ** **
>
> ERROR org.apache.hadoop.security.UserGroupInformation:
> PriviledgedActionException as:nxtbig (auth:SIMPLE)
> cause:java.io.IOException: java.lang.reflect.InvocationTargetException****
>
> 2013-09-16 09:11:18,971 WARN org.apache.hadoop.mapred.Child: Error running
> child****
>
> java.io.IOException: java.lang.reflect.InvocationTargetException****
>
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
> ****
>
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
> ****
>
> at
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:369)
> ****
>
> at
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.<init>(HadoopShimsSecure.java:316)
> ****
>
> at
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(HadoopShimsSecure.java:430)
> ****
>
> at
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:540)
> ****
>
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:395)
> ****
>
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333)****
>
> at org.apache.hadoop.mapred.Child$4.run(Child.java:268)****
>
> at java.security.AccessController.doPrivileged(Native Method)****
>
> at javax.security.auth.Subject.doAs(Subject.java:396)****
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1407)
> ****
>
> at org.apache.hadoop.mapred.Child.main(Child.java:262)****
>
> Caused by: java.lang.reflect.InvocationTargetException****
>
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)****
>
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> ****
>
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> ****
>
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> ****
>
> at
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:355)
> ****
>
> ... 10 more****
>
> Caused by: java.io.IOException:
> hdfs://localhost:54310/user/hive/warehouse/cpj_tbl/cpj.csv.gz not a
> SequenceFile****
>
> at
> org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1805)****
>
> at
> org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:1765)
> ****
>
> at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1714)***
> *
>
> at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)***
> *
>
> at
> org.apache.hadoop.mapred.SequenceFileRecordReader.<init>(SequenceFileRecordReader.java:43)
> ****
>
> at
> org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:59)
> ****
>
> at
> org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.<init>(CombineHiveRecordReader.java:65)
> ****
>
> ... 15 more****
>
> ** **
>
> Can anybody help me on this.****
>
> ** **
>
> Thanks & Regards,****
>
> *Rinku Garg*
>
> ****
>
> ** **
>
> ** **
> _____________
> The information contained in this message is proprietary and/or
> confidential. If you are not the intended recipient, please: (i) delete the
> message and all copies; (ii) do not disclose, distribute or use the message
> in any manner; and (iii) notify the sender immediately. In addition, please
> be aware that any message addressed to our domain is subject to archiving
> and review by persons other than the intended recipient. Thank you.
>
--
Nitin Pawar