You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by YouPeng Yang <yy...@gmail.com> on 2013/04/22 11:01:36 UTC
Exception comes out when counting the rows
Hi hive users
Sorry for missing the title on the previous mail.
This is my first time to post a question here.
I have gotten an exception when I count the rows of my hive table after I
have loaded the data:
hive>create EXTERNAL TABLE NMS_CMTS_CPU_CDX_TEST (CMTSID INT,MSEQ
INT,GOTTIME BIGINT,CMTSINDEX INT,CPUTOTAL INT,DESCR STRING) ROW FORMAT
DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n' STORED AS
TEXTFILE;
hive>load data inpath '/user/sqoop/NMS_CMTS_CPU_CDX3/NMS_CMTS_CPU_CDX3'
into table NMS_CMTS_CPU_CDX_TEST;
hive> select count(1) from NMS_CMTS_CPU_CDX_TEST;
I get an exception on step 3,logs are as follows.
Any helps will be gratefull.
Regards
-----------------------------------------------------------------------------------
....
WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please
use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties
files.
Execution log at:
/tmp/hive/hive_20130422162020_791a7b61-6ba0-466d-99ba-5c2556bafaa4.log
Job running in-process (local Hadoop)
Hadoop job information for null: number of mappers: 1; number of reducers: 1
2013-04-22 16:24:21,604 null map = 0%, reduce = 0%
2013-04-22 16:25:21,965 null map = 0%, reduce = 0%
2013-04-22 16:26:22,902 null map = 0%, reduce = 0%
2013-04-22 16:26:27,312 null map = 100%, reduce = 0%
Ended Job = job_1364348895095_0055 with errors
Error during job, obtaining debugging information...
Examining task ID: task_1364348895095_0055_m_000000 (and more) from job
job_1364348895095_0055
Unable to retrieve URL for Hadoop Task logs. Does not contain a valid
host:port authority: local
Task with the most failures(4):
-----
Task ID:
task_1364348895095_0055_m_000000
URL:
Unavailable
-----
Diagnostic Messages for this Task:
Error: java.lang.RuntimeException: java.io.FileNotFoundException:
/tmp/hive/hive_2013-04-22_16-20-45_720_3839682514463028560/-mr-10001/89dd576e-fb9d-409a-8b46-2e46b7d21160
(No such file or directory)
at
org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:224)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:381)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:374)
at
org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:536)
at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:160)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:381)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:152)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:147)
Caused by: java.io.FileNotFoundException:
/tmp/hive/hive_2013-04-22_16-20-45_720_3839682514463028560/-mr-10001/89dd576e-fb9d-409a-8b46-2e46b7d21160
(No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:120)
at java.io.FileInputStream.<init>(FileInputStream.java:79)
at
org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:215)
... 12 more
Execution failed with exit status: 2
13/04/22 16:26:28 ERROR exec.Task: Execution failed with exit status: 2
Obtaining error information
13/04/22 16:26:28 ERROR exec.Task: Obtaining error information
Task failed!
Task ID:
Stage-1
Logs:
13/04/22 16:26:28 ERROR exec.Task:
Task failed!
Task ID:
Stage-1
Logs:
13/04/22 16:26:28 ERROR exec.ExecDriver: Execution failed with exit status:
2
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.MapRedTask
13/04/22 16:26:28 ERROR ql.Driver: FAILED: Execution Error, return code 2
from org.apache.hadoop.hive.ql.exec.MapRedTask
13/04/22 16:26:28 INFO ql.Driver: </PERFLOG method=Driver.execute
start=1366618840573 end=1366619188646 duration=348073>
13/04/22 16:26:28 INFO ql.Driver: <PERFLOG method=releaseLocks>
13/04/22 16:26:28 INFO ql.Driver: </PERFLOG method=releaseLocks
start=1366619188646 end=1366619188646 duration=0>
Re: Exception comes out when counting the rows
Posted by YouPeng Yang <yy...@gmail.com>.
Hi All
I notice that there are lots of fimilar issues ,however there is no
perfect responses to these issues.
I post this mail to close the case because I happen to get the reason .
Hope to be helpfull.
The exception is due to the configuration variable mapred.job.tracker =
local by default.
I set mapred.job.tracker = MyJobtrackerIP:post,and it works.
In addition,I have an declaration:
Anyone should close the issue if he or she gets the answers that
resolve the issues posted by himself in this mail list.
We should help each other if we get the answer since we are in the
community
Regards
2013/4/23 Ramki Palle <ra...@gmail.com>
> Normally, you do not use load data when you use external table. You either
> specify location in the create external table query or use alter table
> <tablename> set location syntax to set the location of the files that the
> external table stores the data.
>
> In this case, try describe extended NMS_CMTS_CPU_CDX_TEST and see what is
> the value for location.
>
> It is better to use the location as I mentioned above for external tables
> instead of using load data. This way there is no confusion between managed
> tables and external tables locations.
>
> Regards,
> Ramki.
>
>
> On Mon, Apr 22, 2013 at 2:01 AM, YouPeng Yang <yy...@gmail.com>wrote:
>
>>
>> Hi hive users
>>
>> Sorry for missing the title on the previous mail.
>>
>> This is my first time to post a question here.
>>
>> I have gotten an exception when I count the rows of my hive table after
>> I have loaded the data:
>>
>> hive>create EXTERNAL TABLE NMS_CMTS_CPU_CDX_TEST (CMTSID INT,MSEQ
>> INT,GOTTIME BIGINT,CMTSINDEX INT,CPUTOTAL INT,DESCR STRING) ROW FORMAT
>> DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n' STORED AS
>> TEXTFILE;
>>
>> hive>load data inpath '/user/sqoop/NMS_CMTS_CPU_CDX3/NMS_CMTS_CPU_CDX3'
>> into table NMS_CMTS_CPU_CDX_TEST;
>>
>> hive> select count(1) from NMS_CMTS_CPU_CDX_TEST;
>> I get an exception on step 3,logs are as follows.
>>
>> Any helps will be gratefull.
>>
>> Regards
>>
>> -----------------------------------------------------------------------------------
>> ....
>> WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please
>> use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties
>> files.
>> Execution log at:
>> /tmp/hive/hive_20130422162020_791a7b61-6ba0-466d-99ba-5c2556bafaa4.log
>> Job running in-process (local Hadoop)
>> Hadoop job information for null: number of mappers: 1; number of
>> reducers: 1
>> 2013-04-22 16:24:21,604 null map = 0%, reduce = 0%
>> 2013-04-22 16:25:21,965 null map = 0%, reduce = 0%
>> 2013-04-22 16:26:22,902 null map = 0%, reduce = 0%
>> 2013-04-22 16:26:27,312 null map = 100%, reduce = 0%
>> Ended Job = job_1364348895095_0055 with errors
>> Error during job, obtaining debugging information...
>> Examining task ID: task_1364348895095_0055_m_000000 (and more) from job
>> job_1364348895095_0055
>> Unable to retrieve URL for Hadoop Task logs. Does not contain a valid
>> host:port authority: local
>>
>> Task with the most failures(4):
>> -----
>> Task ID:
>> task_1364348895095_0055_m_000000
>>
>> URL:
>> Unavailable
>> -----
>> Diagnostic Messages for this Task:
>> Error: java.lang.RuntimeException: java.io.FileNotFoundException:
>> /tmp/hive/hive_2013-04-22_16-20-45_720_3839682514463028560/-mr-10001/89dd576e-fb9d-409a-8b46-2e46b7d21160
>> (No such file or directory)
>> at
>> org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:224)
>> at
>> org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255)
>> at
>> org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:381)
>> at
>> org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:374)
>> at
>> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:536)
>> at
>> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:160)
>> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:381)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334)
>> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:152)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
>> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:147)
>> Caused by: java.io.FileNotFoundException:
>> /tmp/hive/hive_2013-04-22_16-20-45_720_3839682514463028560/-mr-10001/89dd576e-fb9d-409a-8b46-2e46b7d21160
>> (No such file or directory)
>> at java.io.FileInputStream.open(Native Method)
>> at java.io.FileInputStream.<init>(FileInputStream.java:120)
>> at java.io.FileInputStream.<init>(FileInputStream.java:79)
>> at
>> org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:215)
>> ... 12 more
>>
>>
>> Execution failed with exit status: 2
>> 13/04/22 16:26:28 ERROR exec.Task: Execution failed with exit status: 2
>> Obtaining error information
>> 13/04/22 16:26:28 ERROR exec.Task: Obtaining error information
>>
>> Task failed!
>> Task ID:
>> Stage-1
>>
>> Logs:
>>
>> 13/04/22 16:26:28 ERROR exec.Task:
>> Task failed!
>> Task ID:
>> Stage-1
>>
>> Logs:
>>
>> 13/04/22 16:26:28 ERROR exec.ExecDriver: Execution failed with exit
>> status: 2
>> FAILED: Execution Error, return code 2 from
>> org.apache.hadoop.hive.ql.exec.MapRedTask
>> 13/04/22 16:26:28 ERROR ql.Driver: FAILED: Execution Error, return code 2
>> from org.apache.hadoop.hive.ql.exec.MapRedTask
>> 13/04/22 16:26:28 INFO ql.Driver: </PERFLOG method=Driver.execute
>> start=1366618840573 end=1366619188646 duration=348073>
>> 13/04/22 16:26:28 INFO ql.Driver: <PERFLOG method=releaseLocks>
>> 13/04/22 16:26:28 INFO ql.Driver: </PERFLOG method=releaseLocks
>> start=1366619188646 end=1366619188646 duration=0>
>>
>>
>>
>
Re: Exception comes out when counting the rows
Posted by Ramki Palle <ra...@gmail.com>.
Normally, you do not use load data when you use external table. You either
specify location in the create external table query or use alter table
<tablename> set location syntax to set the location of the files that the
external table stores the data.
In this case, try describe extended NMS_CMTS_CPU_CDX_TEST and see what is
the value for location.
It is better to use the location as I mentioned above for external tables
instead of using load data. This way there is no confusion between managed
tables and external tables locations.
Regards,
Ramki.
On Mon, Apr 22, 2013 at 2:01 AM, YouPeng Yang <yy...@gmail.com>wrote:
>
> Hi hive users
>
> Sorry for missing the title on the previous mail.
>
> This is my first time to post a question here.
>
> I have gotten an exception when I count the rows of my hive table after I
> have loaded the data:
>
> hive>create EXTERNAL TABLE NMS_CMTS_CPU_CDX_TEST (CMTSID INT,MSEQ
> INT,GOTTIME BIGINT,CMTSINDEX INT,CPUTOTAL INT,DESCR STRING) ROW FORMAT
> DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n' STORED AS
> TEXTFILE;
>
> hive>load data inpath '/user/sqoop/NMS_CMTS_CPU_CDX3/NMS_CMTS_CPU_CDX3'
> into table NMS_CMTS_CPU_CDX_TEST;
>
> hive> select count(1) from NMS_CMTS_CPU_CDX_TEST;
> I get an exception on step 3,logs are as follows.
>
> Any helps will be gratefull.
>
> Regards
>
> -----------------------------------------------------------------------------------
> ....
> WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please
> use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties
> files.
> Execution log at:
> /tmp/hive/hive_20130422162020_791a7b61-6ba0-466d-99ba-5c2556bafaa4.log
> Job running in-process (local Hadoop)
> Hadoop job information for null: number of mappers: 1; number of reducers:
> 1
> 2013-04-22 16:24:21,604 null map = 0%, reduce = 0%
> 2013-04-22 16:25:21,965 null map = 0%, reduce = 0%
> 2013-04-22 16:26:22,902 null map = 0%, reduce = 0%
> 2013-04-22 16:26:27,312 null map = 100%, reduce = 0%
> Ended Job = job_1364348895095_0055 with errors
> Error during job, obtaining debugging information...
> Examining task ID: task_1364348895095_0055_m_000000 (and more) from job
> job_1364348895095_0055
> Unable to retrieve URL for Hadoop Task logs. Does not contain a valid
> host:port authority: local
>
> Task with the most failures(4):
> -----
> Task ID:
> task_1364348895095_0055_m_000000
>
> URL:
> Unavailable
> -----
> Diagnostic Messages for this Task:
> Error: java.lang.RuntimeException: java.io.FileNotFoundException:
> /tmp/hive/hive_2013-04-22_16-20-45_720_3839682514463028560/-mr-10001/89dd576e-fb9d-409a-8b46-2e46b7d21160
> (No such file or directory)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:224)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:381)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:374)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:536)
> at
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:160)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:381)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:152)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:147)
> Caused by: java.io.FileNotFoundException:
> /tmp/hive/hive_2013-04-22_16-20-45_720_3839682514463028560/-mr-10001/89dd576e-fb9d-409a-8b46-2e46b7d21160
> (No such file or directory)
> at java.io.FileInputStream.open(Native Method)
> at java.io.FileInputStream.<init>(FileInputStream.java:120)
> at java.io.FileInputStream.<init>(FileInputStream.java:79)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:215)
> ... 12 more
>
>
> Execution failed with exit status: 2
> 13/04/22 16:26:28 ERROR exec.Task: Execution failed with exit status: 2
> Obtaining error information
> 13/04/22 16:26:28 ERROR exec.Task: Obtaining error information
>
> Task failed!
> Task ID:
> Stage-1
>
> Logs:
>
> 13/04/22 16:26:28 ERROR exec.Task:
> Task failed!
> Task ID:
> Stage-1
>
> Logs:
>
> 13/04/22 16:26:28 ERROR exec.ExecDriver: Execution failed with exit
> status: 2
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.MapRedTask
> 13/04/22 16:26:28 ERROR ql.Driver: FAILED: Execution Error, return code 2
> from org.apache.hadoop.hive.ql.exec.MapRedTask
> 13/04/22 16:26:28 INFO ql.Driver: </PERFLOG method=Driver.execute
> start=1366618840573 end=1366619188646 duration=348073>
> 13/04/22 16:26:28 INFO ql.Driver: <PERFLOG method=releaseLocks>
> 13/04/22 16:26:28 INFO ql.Driver: </PERFLOG method=releaseLocks
> start=1366619188646 end=1366619188646 duration=0>
>
>
>