You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Manish <ma...@rocketmail.com> on 2012/09/20 17:10:00 UTC
Hive custom inputfomat error.
I have created custom input format in java. Now I want to use that input
format when creating table in Hive.
Here are my steps:
1. Created jar file of the program.
2. Copied jar in /usr/lib/hadoop.
3. Added the path in CLASSPATH.
now here is my code:
Create table hitdata_fnl
(
col_0 string,
col_1 string,
col_2 string,
col_3 string,
col_4 string,
col_5 string,
col_6 string,
col_7 string,
col_8 string,
col_9 string,
col_10 string
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'
STORED AS INPUTFORMAT 'com.tgam.hadoop.mapreduce.inputfilerecordreader'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat';
Error :
FAILED: Error in metadata: Class not found:
com.tgam.hadoop.mapreduce.inputfilerecordreader
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
Please help on this error.
Thank You,
Manish.
RE: Hive custom inputfomat error.
Posted by "Connell, Chuck" <Ch...@nuance.com>.
Good tips. Thanks.
Everyone using an extra JAR should keep in mind that there is a difference between the Hive shell seeing a jar (which allows Hive commands like CREATE TABLE … SERDE to work) and a generated MR job seeing a jar (allowing the job to run correctly). Some of these settings affect one, and some the other.
Chuck Connell
Nuance R&D Data Team
Burlington, MA
From: richin.jain@nokia.com [mailto:richin.jain@nokia.com]
Sent: Thursday, September 20, 2012 11:26 AM
To: user@hive.apache.org
Subject: RE: Hive custom inputfomat error.
You can always use : ADD JAR <Path-to-your-jar-file> to your HQL file or run this command on Hive shell.
OR
I found this in a previous thread
Add following property to your hive-site.xml
<property>
<name>hive.aux.jars.path</name>
<value>file:///home/me/my.jar,file:///home/you/your.jar,file:///home/us/our.jar</value<file:///\\home\me\my.jar,file:\home\you\your.jar,file:\home\us\our.jar%3c\value>>
</property>
Hope this helps.
Richin
From: ext Connell, Chuck [mailto:Chuck.Connell@nuance.com]
Sent: Thursday, September 20, 2012 11:16 AM
To: user@hive.apache.org<ma...@hive.apache.org>
Subject: RE: Hive custom inputfomat error.
You might try adding “ --auxpath /path/to/jar/dir “ to the Hive command line.
Chuck Connell
Nuance R&D Data Team
Burlington, MA
From: Manish [mailto:manishbhoge@rocketmail.com]<mailto:[mailto:manishbhoge@rocketmail.com]>
Sent: Thursday, September 20, 2012 11:10 AM
To: user
Cc: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Hive custom inputfomat error.
I have created custom input format in java. Now I want to use that input format when creating table in Hive.
Here are my steps:
1. Created jar file of the program.
2. Copied jar in /usr/lib/hadoop.
3. Added the path in CLASSPATH.
now here is my code:
Create table hitdata_fnl
(
col_0 string,
col_1 string,
col_2 string,
col_3 string,
col_4 string,
col_5 string,
col_6 string,
col_7 string,
col_8 string,
col_9 string,
col_10 string
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'
STORED AS INPUTFORMAT 'com.tgam.hadoop.mapreduce.inputfilerecordreader' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat';
Error :
FAILED: Error in metadata: Class not found: com.tgam.hadoop.mapreduce.inputfilerecordreader
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
Please help on this error.
Thank You,
Manish.
Re: Hive custom inputfomat error.
Posted by Edward Capriolo <ed...@gmail.com>.
InputFormats need to be in auxlib. Add jar is not good enough. You can
either edit your config file or create an auxlib folder and put these
jars in that folder.
On Thu, Sep 20, 2012 at 11:25 AM, <ri...@nokia.com> wrote:
> You can always use : ADD JAR <Path-to-your-jar-file> to your HQL file or run
> this command on Hive shell.
>
>
>
> OR
>
>
>
> I found this in a previous thread
>
>
>
> Add following property to your hive-site.xml
>
> <property>
>
> <name>hive.aux.jars.path</name>
>
> <value>file:///home/me/my.jar,file:///home/you/your.jar,file:///home/us/our.jar</value>
>
> </property>
>
> Hope this helps.
>
>
>
> Richin
>
>
>
> From: ext Connell, Chuck [mailto:Chuck.Connell@nuance.com]
> Sent: Thursday, September 20, 2012 11:16 AM
> To: user@hive.apache.org
> Subject: RE: Hive custom inputfomat error.
>
>
>
> You might try adding “ --auxpath /path/to/jar/dir “ to the Hive command
> line.
>
>
>
> Chuck Connell
>
> Nuance R&D Data Team
>
> Burlington, MA
>
>
>
>
>
> From: Manish [mailto:manishbhoge@rocketmail.com]
> Sent: Thursday, September 20, 2012 11:10 AM
> To: user
> Cc: user@hadoop.apache.org
> Subject: Hive custom inputfomat error.
>
>
>
> I have created custom input format in java. Now I want to use that input
> format when creating table in Hive.
>
> Here are my steps:
> 1. Created jar file of the program.
> 2. Copied jar in /usr/lib/hadoop.
> 3. Added the path in CLASSPATH.
>
> now here is my code:
>
> Create table hitdata_fnl
> (
> col_0 string,
> col_1 string,
> col_2 string,
> col_3 string,
> col_4 string,
> col_5 string,
> col_6 string,
> col_7 string,
> col_8 string,
> col_9 string,
> col_10 string
> )
> ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'
> STORED AS INPUTFORMAT 'com.tgam.hadoop.mapreduce.inputfilerecordreader'
> OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat';
>
> Error :
> FAILED: Error in metadata: Class not found:
> com.tgam.hadoop.mapreduce.inputfilerecordreader
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask
>
> Please help on this error.
>
> Thank You,
> Manish.
RE: Hive custom inputfomat error.
Posted by ri...@nokia.com.
You can always use : ADD JAR <Path-to-your-jar-file> to your HQL file or run this command on Hive shell.
OR
I found this in a previous thread
Add following property to your hive-site.xml
<property>
<name>hive.aux.jars.path</name>
<value>file:///home/me/my.jar,file:///home/you/your.jar,file:///home/us/our.jar</value>
</property>
Hope this helps.
Richin
From: ext Connell, Chuck [mailto:Chuck.Connell@nuance.com]
Sent: Thursday, September 20, 2012 11:16 AM
To: user@hive.apache.org
Subject: RE: Hive custom inputfomat error.
You might try adding “ --auxpath /path/to/jar/dir “ to the Hive command line.
Chuck Connell
Nuance R&D Data Team
Burlington, MA
From: Manish [mailto:manishbhoge@rocketmail.com]<mailto:[mailto:manishbhoge@rocketmail.com]>
Sent: Thursday, September 20, 2012 11:10 AM
To: user
Cc: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Hive custom inputfomat error.
I have created custom input format in java. Now I want to use that input format when creating table in Hive.
Here are my steps:
1. Created jar file of the program.
2. Copied jar in /usr/lib/hadoop.
3. Added the path in CLASSPATH.
now here is my code:
Create table hitdata_fnl
(
col_0 string,
col_1 string,
col_2 string,
col_3 string,
col_4 string,
col_5 string,
col_6 string,
col_7 string,
col_8 string,
col_9 string,
col_10 string
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'
STORED AS INPUTFORMAT 'com.tgam.hadoop.mapreduce.inputfilerecordreader' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat';
Error :
FAILED: Error in metadata: Class not found: com.tgam.hadoop.mapreduce.inputfilerecordreader
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
Please help on this error.
Thank You,
Manish.
RE: Hive custom inputfomat error.
Posted by Manish <ma...@rocketmail.com>.
Thanks. Got the solution, i was copying my jar in wrong folder. I should
have copied in /usr/lib/hive/lib folder.
Thank You,
Manish
On Thu, 2012-09-20 at 15:15 +0000, Connell, Chuck wrote:
> You might try adding “ --auxpath /path/to/jar/dir “ to the Hive
> command line.
>
>
>
>
> Chuck Connell
>
> Nuance R&D Data Team
>
> Burlington, MA
>
>
>
>
>
>
> From: Manish [mailto:manishbhoge@rocketmail.com]
> Sent: Thursday, September 20, 2012 11:10 AM
> To: user
> Cc: user@hadoop.apache.org
> Subject: Hive custom inputfomat error.
>
>
>
>
>
> I have created custom input format in java. Now I want to use that
> input format when creating table in Hive.
>
> Here are my steps:
> 1. Created jar file of the program.
> 2. Copied jar in /usr/lib/hadoop.
> 3. Added the path in CLASSPATH.
>
> now here is my code:
>
> Create table hitdata_fnl
> (
> col_0 string,
> col_1 string,
> col_2 string,
> col_3 string,
> col_4 string,
> col_5 string,
> col_6 string,
> col_7 string,
> col_8 string,
> col_9 string,
> col_10 string
> )
> ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'
> STORED AS INPUTFORMAT
> 'com.tgam.hadoop.mapreduce.inputfilerecordreader' OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat';
>
> Error :
> FAILED: Error in metadata: Class not found:
> com.tgam.hadoop.mapreduce.inputfilerecordreader
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask
>
> Please help on this error.
>
> Thank You,
> Manish.
>
>
RE: Hive custom inputfomat error.
Posted by "Connell, Chuck" <Ch...@nuance.com>.
You might try adding “ --auxpath /path/to/jar/dir “ to the Hive command line.
Chuck Connell
Nuance R&D Data Team
Burlington, MA
From: Manish [mailto:manishbhoge@rocketmail.com]
Sent: Thursday, September 20, 2012 11:10 AM
To: user
Cc: user@hadoop.apache.org
Subject: Hive custom inputfomat error.
I have created custom input format in java. Now I want to use that input format when creating table in Hive.
Here are my steps:
1. Created jar file of the program.
2. Copied jar in /usr/lib/hadoop.
3. Added the path in CLASSPATH.
now here is my code:
Create table hitdata_fnl
(
col_0 string,
col_1 string,
col_2 string,
col_3 string,
col_4 string,
col_5 string,
col_6 string,
col_7 string,
col_8 string,
col_9 string,
col_10 string
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'
STORED AS INPUTFORMAT 'com.tgam.hadoop.mapreduce.inputfilerecordreader' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat';
Error :
FAILED: Error in metadata: Class not found: com.tgam.hadoop.mapreduce.inputfilerecordreader
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
Please help on this error.
Thank You,
Manish.