You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Thiago Henrique dos Santos Bento <th...@b2wdigital.com> on 2014/10/01 16:41:56 UTC
Join error with ORC Hive tables
Hi!
I'm trying to run a simple join with two tables stored as ORC File. (the same query runs normally without ORC File)
I got this error while map-reduce job:
Driver returned: 2. Errors: OK
Query ID = hue_20141001071313_33e31d68-3b4a-49d5-acc0-30c0a42276ab
Total jobs = 1
Stage-1 is selected by condition resolver.
Launching Job 1 out of 1
Number of reduce tasks not specified. Estimated from input data size: 12
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Starting Job = job_1411750973876_0099, Tracking URL = http://arnaldo02:8088/proxy/application_1411750973876_0099/
Kill Command = /usr/lib/hadoop/bin/hadoop job -kill job_1411750973876_0099
Hadoop job information for Stage-1: number of mappers: 42; number of reducers: 12
2014-10-01 07:13:51,965 Stage-1 map = 0%, reduce = 0%
2014-10-01 07:14:18,882 Stage-1 map = 2%, reduce = 0%, Cumulative CPU 37.32 sec
2014-10-01 07:14:19,990 Stage-1 map = 3%, reduce = 0%, Cumulative CPU 40.31 sec
2014-10-01 07:14:21,100 Stage-1 map = 5%, reduce = 0%, Cumulative CPU 43.33 sec
2014-10-01 07:14:25,544 Stage-1 map = 7%, reduce = 0%, Cumulative CPU 57.49 sec
2014-10-01 07:14:39,996 Stage-1 map = 8%, reduce = 0%, Cumulative CPU 73.08 sec
2014-10-01 07:14:43,338 Stage-1 map = 9%, reduce = 0%, Cumulative CPU 76.76 sec
2014-10-01 07:14:48,882 Stage-1 map = 11%, reduce = 0%, Cumulative CPU 92.88 sec
2014-10-01 07:14:51,107 Stage-1 map = 12%, reduce = 0%, Cumulative CPU 100.93 sec
2014-10-01 07:14:52,212 Stage-1 map = 13%, reduce = 0%, Cumulative CPU 106.04 sec
2014-10-01 07:14:54,426 Stage-1 map = 14%, reduce = 0%, Cumulative CPU 111.02 sec
2014-10-01 07:14:57,751 Stage-1 map = 15%, reduce = 0%, Cumulative CPU 126.41 sec
2014-10-01 07:15:05,507 Stage-1 map = 17%, reduce = 0%, Cumulative CPU 135.71 sec
2014-10-01 07:15:10,023 Stage-1 map = 18%, reduce = 0%, Cumulative CPU 146.69 sec
2014-10-01 07:15:19,987 Stage-1 map = 19%, reduce = 0%, Cumulative CPU 156.43 sec
2014-10-01 07:15:29,957 Stage-1 map = 19%, reduce = 1%, Cumulative CPU 168.3 sec
2014-10-01 07:15:37,703 Stage-1 map = 21%, reduce = 1%, Cumulative CPU 179.77 sec
2014-10-01 07:15:45,511 Stage-1 map = 23%, reduce = 1%, Cumulative CPU 192.87 sec
2014-10-01 07:15:49,948 Stage-1 map = 25%, reduce = 1%, Cumulative CPU 204.72 sec
2014-10-01 07:15:53,287 Stage-1 map = 26%, reduce = 1%, Cumulative CPU 211.83 sec
2014-10-01 07:16:05,456 Stage-1 map = 28%, reduce = 1%, Cumulative CPU 231.45 sec
2014-10-01 07:16:07,664 Stage-1 map = 29%, reduce = 1%, Cumulative CPU 233.46 sec
2014-10-01 07:16:09,876 Stage-1 map = 29%, reduce = 2%, Cumulative CPU 233.74 sec
2014-10-01 07:16:15,408 Stage-1 map = 32%, reduce = 2%, Cumulative CPU 255.51 sec
2014-10-01 07:16:18,727 Stage-1 map = 33%, reduce = 2%, Cumulative CPU 258.57 sec
2014-10-01 07:16:19,830 Stage-1 map = 33%, reduce = 3%, Cumulative CPU 261.95 sec
2014-10-01 07:16:35,288 Stage-1 map = 35%, reduce = 3%, Cumulative CPU 288.59 sec
2014-10-01 07:16:40,806 Stage-1 map = 37%, reduce = 3%, Cumulative CPU 298.89 sec
2014-10-01 07:16:45,278 Stage-1 map = 38%, reduce = 3%, Cumulative CPU 306.72 sec
2014-10-01 07:16:58,531 Stage-1 map = 38%, reduce = 4%, Cumulative CPU 312.76 sec
2014-10-01 07:17:02,945 Stage-1 map = 40%, reduce = 4%, Cumulative CPU 321.87 sec
2014-10-01 07:17:22,828 Stage-1 map = 42%, reduce = 4%, Cumulative CPU 336.71 sec
2014-10-01 07:17:28,341 Stage-1 map = 43%, reduce = 5%, Cumulative CPU 343.66 sec
2014-10-01 07:17:39,370 Stage-1 map = 44%, reduce = 5%, Cumulative CPU 350.7 sec
2014-10-01 07:17:51,541 Stage-1 map = 47%, reduce = 5%, Cumulative CPU 365.91 sec
2014-10-01 07:18:08,097 Stage-1 map = 49%, reduce = 5%, Cumulative CPU 375.35 sec
2014-10-01 07:18:22,452 Stage-1 map = 49%, reduce = 7%, Cumulative CPU 383.34 sec
2014-10-01 07:18:25,756 Stage-1 map = 50%, reduce = 7%, Cumulative CPU 383.69 sec
2014-10-01 07:19:26,488 Stage-1 map = 50%, reduce = 7%, Cumulative CPU 389.38 sec
2014-10-01 07:19:50,793 Stage-1 map = 52%, reduce = 7%, Cumulative CPU 391.28 sec
2014-10-01 07:19:58,551 Stage-1 map = 50%, reduce = 7%, Cumulative CPU 386.39 sec
2014-10-01 07:20:50,430 Stage-1 map = 52%, reduce = 7%, Cumulative CPU 393.79 sec
2014-10-01 07:21:51,114 Stage-1 map = 52%, reduce = 7%, Cumulative CPU 411.94 sec
2014-10-01 07:22:19,773 Stage-1 map = 54%, reduce = 7%, Cumulative CPU 417.31 sec
2014-10-01 07:23:20,493 Stage-1 map = 54%, reduce = 7%, Cumulative CPU 423.21 sec
2014-10-01 07:24:21,168 Stage-1 map = 54%, reduce = 7%, Cumulative CPU 425.94 sec
2014-10-01 07:25:21,814 Stage-1 map = 54%, reduce = 7%, Cumulative CPU 441.2 sec
2014-10-01 07:25:36,145 Stage-1 map = 55%, reduce = 7%, Cumulative CPU 442.7 sec
2014-10-01 07:26:36,837 Stage-1 map = 55%, reduce = 7%, Cumulative CPU 448.75 sec
2014-10-01 07:26:40,140 Stage-1 map = 55%, reduce = 8%, Cumulative CPU 449.35 sec
2014-10-01 07:27:11,064 Stage-1 map = 56%, reduce = 8%, Cumulative CPU 455.24 sec
2014-10-01 07:27:19,871 Stage-1 map = 57%, reduce = 8%, Cumulative CPU 460.35 sec
2014-10-01 07:27:56,277 Stage-1 map = 58%, reduce = 8%, Cumulative CPU 481.34 sec
2014-10-01 07:28:40,332 Stage-1 map = 59%, reduce = 8%, Cumulative CPU 504.81 sec
2014-10-01 07:29:40,957 Stage-1 map = 59%, reduce = 8%, Cumulative CPU 510.71 sec
2014-10-01 07:30:41,638 Stage-1 map = 59%, reduce = 8%, Cumulative CPU 513.74 sec
2014-10-01 07:31:00,419 Stage-1 map = 60%, reduce = 8%, Cumulative CPU 514.6 sec
2014-10-01 07:32:01,081 Stage-1 map = 60%, reduce = 8%, Cumulative CPU 518.18 sec
2014-10-01 07:32:50,724 Stage-1 map = 62%, reduce = 8%, Cumulative CPU 520.43 sec
2014-10-01 07:33:00,648 Stage-1 map = 60%, reduce = 8%, Cumulative CPU 521.71 sec
2014-10-01 07:33:14,974 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 459.27 sec
MapReduce Total cumulative CPU time: 7 minutes 39 seconds 270 msec
Ended Job = job_1411750973876_0099 with errors
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:
Job 0: Map: 42 Reduce: 12 Cumulative CPU: 459.27 sec HDFS Read: 1989875843 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 7 minutes 39 seconds 270 msec
Any idea?
Tks,
Thiago Henrique S. Bento
B2W Digital / B2W Labs | BIT SP
thiago.bento@b2wdigital.com<ma...@b2wdigital.com>
Tel.: +55 (11) 3454-3024
.............................................................................................................................................................
B2W DIGITAL #Americanas.com<http://americanas.com/> #Shoptime.com<http://shoptime.com/> #Submarino.com<http://submarino.com/> #SouBarato.com<http://soubarato.com/>
________________________________
Esta mensagem pode conter informa??es confidenciais e somente o indiv?duo ou entidade a quem foi destinada pode utiliz?-la. A transmiss?o incorreta da mensagem n?o acarreta a perda de sua confidencialidade. Caso esta mensagem tenha sido recebida por engano, solicitamos que o fato seja comunicado ao remetente e que a mensagem seja eliminada de seu sistema imediatamente. ? vedado a qualquer pessoa que n?o seja o destinat?rio usar, revelar, distribuir ou copiar qualquer parte desta mensagem. Ambiente de comunica??o sujeito a monitoramento.
This message may include confidential information and only the intended addresses have the right to use it as is, or any part of it. A wrong transmission does not break its confidentiality. If you've received it because of a mistake or erroneous transmission, please notify the sender and delete it from your system immediately. This communication environment is controlled and monitored.
B2W Digital
Re: Join error with ORC Hive tables
Posted by Nagarjuna Vissarapu <na...@gmail.com>.
Set -m flag to 1 programmatically
On Oct 1, 2014 8:13 PM, "Thiago Henrique dos Santos Bento" <
thiago.bento@b2wdigital.com> wrote:
> Hi!
>
>
>
> I'm trying to run a simple join with two tables stored as ORC File. (the
> same query runs normally without ORC File)
>
> I got this error while map-reduce job:
>
> Driver returned: 2. Errors: OK
>
> Query ID = hue_20141001071313_33e31d68-3b4a-49d5-acc0-30c0a42276ab
>
> Total jobs = 1
>
> Stage-1 is selected by condition resolver.
>
> Launching Job 1 out of 1
>
> Number of reduce tasks not specified. Estimated from input data size: 12
>
> In order to change the average load for a reducer (in bytes):
>
> set hive.exec.reducers.bytes.per.reducer=<number>
>
> In order to limit the maximum number of reducers:
>
> set hive.exec.reducers.max=<number>
>
> In order to set a constant number of reducers:
>
> set mapreduce.job.reduces=<number>
>
> Starting Job = job_1411750973876_0099, Tracking URL =
> http://arnaldo02:8088/proxy/application_1411750973876_0099/
>
> Kill Command = /usr/lib/hadoop/bin/hadoop job -kill job_1411750973876_0099
>
> Hadoop job information for Stage-1: number of mappers: 42; number of
> reducers: 12
>
> 2014-10-01 07:13:51,965 Stage-1 map = 0%, reduce = 0%
>
> 2014-10-01 07:14:18,882 Stage-1 map = 2%, reduce = 0%, Cumulative CPU
> 37.32 sec
>
> 2014-10-01 07:14:19,990 Stage-1 map = 3%, reduce = 0%, Cumulative CPU
> 40.31 sec
>
> 2014-10-01 07:14:21,100 Stage-1 map = 5%, reduce = 0%, Cumulative CPU
> 43.33 sec
>
> 2014-10-01 07:14:25,544 Stage-1 map = 7%, reduce = 0%, Cumulative CPU
> 57.49 sec
>
> 2014-10-01 07:14:39,996 Stage-1 map = 8%, reduce = 0%, Cumulative CPU
> 73.08 sec
>
> 2014-10-01 07:14:43,338 Stage-1 map = 9%, reduce = 0%, Cumulative CPU
> 76.76 sec
>
> 2014-10-01 07:14:48,882 Stage-1 map = 11%, reduce = 0%, Cumulative CPU
> 92.88 sec
>
> 2014-10-01 07:14:51,107 Stage-1 map = 12%, reduce = 0%, Cumulative CPU
> 100.93 sec
>
> 2014-10-01 07:14:52,212 Stage-1 map = 13%, reduce = 0%, Cumulative CPU
> 106.04 sec
>
> 2014-10-01 07:14:54,426 Stage-1 map = 14%, reduce = 0%, Cumulative CPU
> 111.02 sec
>
> 2014-10-01 07:14:57,751 Stage-1 map = 15%, reduce = 0%, Cumulative CPU
> 126.41 sec
>
> 2014-10-01 07:15:05,507 Stage-1 map = 17%, reduce = 0%, Cumulative CPU
> 135.71 sec
>
> 2014-10-01 07:15:10,023 Stage-1 map = 18%, reduce = 0%, Cumulative CPU
> 146.69 sec
>
> 2014-10-01 07:15:19,987 Stage-1 map = 19%, reduce = 0%, Cumulative CPU
> 156.43 sec
>
> 2014-10-01 07:15:29,957 Stage-1 map = 19%, reduce = 1%, Cumulative CPU
> 168.3 sec
>
> 2014-10-01 07:15:37,703 Stage-1 map = 21%, reduce = 1%, Cumulative CPU
> 179.77 sec
>
> 2014-10-01 07:15:45,511 Stage-1 map = 23%, reduce = 1%, Cumulative CPU
> 192.87 sec
>
> 2014-10-01 07:15:49,948 Stage-1 map = 25%, reduce = 1%, Cumulative CPU
> 204.72 sec
>
> 2014-10-01 07:15:53,287 Stage-1 map = 26%, reduce = 1%, Cumulative CPU
> 211.83 sec
>
> 2014-10-01 07:16:05,456 Stage-1 map = 28%, reduce = 1%, Cumulative CPU
> 231.45 sec
>
> 2014-10-01 07:16:07,664 Stage-1 map = 29%, reduce = 1%, Cumulative CPU
> 233.46 sec
>
> 2014-10-01 07:16:09,876 Stage-1 map = 29%, reduce = 2%, Cumulative CPU
> 233.74 sec
>
> 2014-10-01 07:16:15,408 Stage-1 map = 32%, reduce = 2%, Cumulative CPU
> 255.51 sec
>
> 2014-10-01 07:16:18,727 Stage-1 map = 33%, reduce = 2%, Cumulative CPU
> 258.57 sec
>
> 2014-10-01 07:16:19,830 Stage-1 map = 33%, reduce = 3%, Cumulative CPU
> 261.95 sec
>
> 2014-10-01 07:16:35,288 Stage-1 map = 35%, reduce = 3%, Cumulative CPU
> 288.59 sec
>
> 2014-10-01 07:16:40,806 Stage-1 map = 37%, reduce = 3%, Cumulative CPU
> 298.89 sec
>
> 2014-10-01 07:16:45,278 Stage-1 map = 38%, reduce = 3%, Cumulative CPU
> 306.72 sec
>
> 2014-10-01 07:16:58,531 Stage-1 map = 38%, reduce = 4%, Cumulative CPU
> 312.76 sec
>
> 2014-10-01 07:17:02,945 Stage-1 map = 40%, reduce = 4%, Cumulative CPU
> 321.87 sec
>
> 2014-10-01 07:17:22,828 Stage-1 map = 42%, reduce = 4%, Cumulative CPU
> 336.71 sec
>
> 2014-10-01 07:17:28,341 Stage-1 map = 43%, reduce = 5%, Cumulative CPU
> 343.66 sec
>
> 2014-10-01 07:17:39,370 Stage-1 map = 44%, reduce = 5%, Cumulative CPU
> 350.7 sec
>
> 2014-10-01 07:17:51,541 Stage-1 map = 47%, reduce = 5%, Cumulative CPU
> 365.91 sec
>
> 2014-10-01 07:18:08,097 Stage-1 map = 49%, reduce = 5%, Cumulative CPU
> 375.35 sec
>
> 2014-10-01 07:18:22,452 Stage-1 map = 49%, reduce = 7%, Cumulative CPU
> 383.34 sec
>
> 2014-10-01 07:18:25,756 Stage-1 map = 50%, reduce = 7%, Cumulative CPU
> 383.69 sec
>
> 2014-10-01 07:19:26,488 Stage-1 map = 50%, reduce = 7%, Cumulative CPU
> 389.38 sec
>
> 2014-10-01 07:19:50,793 Stage-1 map = 52%, reduce = 7%, Cumulative CPU
> 391.28 sec
>
> 2014-10-01 07:19:58,551 Stage-1 map = 50%, reduce = 7%, Cumulative CPU
> 386.39 sec
>
> 2014-10-01 07:20:50,430 Stage-1 map = 52%, reduce = 7%, Cumulative CPU
> 393.79 sec
>
> 2014-10-01 07:21:51,114 Stage-1 map = 52%, reduce = 7%, Cumulative CPU
> 411.94 sec
>
> 2014-10-01 07:22:19,773 Stage-1 map = 54%, reduce = 7%, Cumulative CPU
> 417.31 sec
>
> 2014-10-01 07:23:20,493 Stage-1 map = 54%, reduce = 7%, Cumulative CPU
> 423.21 sec
>
> 2014-10-01 07:24:21,168 Stage-1 map = 54%, reduce = 7%, Cumulative CPU
> 425.94 sec
>
> 2014-10-01 07:25:21,814 Stage-1 map = 54%, reduce = 7%, Cumulative CPU
> 441.2 sec
>
> 2014-10-01 07:25:36,145 Stage-1 map = 55%, reduce = 7%, Cumulative CPU
> 442.7 sec
>
> 2014-10-01 07:26:36,837 Stage-1 map = 55%, reduce = 7%, Cumulative CPU
> 448.75 sec
>
> 2014-10-01 07:26:40,140 Stage-1 map = 55%, reduce = 8%, Cumulative CPU
> 449.35 sec
>
> 2014-10-01 07:27:11,064 Stage-1 map = 56%, reduce = 8%, Cumulative CPU
> 455.24 sec
>
> 2014-10-01 07:27:19,871 Stage-1 map = 57%, reduce = 8%, Cumulative CPU
> 460.35 sec
>
> 2014-10-01 07:27:56,277 Stage-1 map = 58%, reduce = 8%, Cumulative CPU
> 481.34 sec
>
> 2014-10-01 07:28:40,332 Stage-1 map = 59%, reduce = 8%, Cumulative CPU
> 504.81 sec
>
> 2014-10-01 07:29:40,957 Stage-1 map = 59%, reduce = 8%, Cumulative CPU
> 510.71 sec
>
> 2014-10-01 07:30:41,638 Stage-1 map = 59%, reduce = 8%, Cumulative CPU
> 513.74 sec
>
> 2014-10-01 07:31:00,419 Stage-1 map = 60%, reduce = 8%, Cumulative CPU
> 514.6 sec
>
> 2014-10-01 07:32:01,081 Stage-1 map = 60%, reduce = 8%, Cumulative CPU
> 518.18 sec
>
> 2014-10-01 07:32:50,724 Stage-1 map = 62%, reduce = 8%, Cumulative CPU
> 520.43 sec
>
> 2014-10-01 07:33:00,648 Stage-1 map = 60%, reduce = 8%, Cumulative CPU
> 521.71 sec
>
> 2014-10-01 07:33:14,974 Stage-1 map = 100%, reduce = 100%, Cumulative CPU
> 459.27 sec
>
> MapReduce Total cumulative CPU time: 7 minutes 39 seconds 270 msec
>
> Ended Job = job_1411750973876_0099 with errors
>
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask
>
> MapReduce Jobs Launched:
>
> Job 0: Map: 42 Reduce: 12 Cumulative CPU: 459.27 sec HDFS Read:
> 1989875843 HDFS Write: 0 FAIL
>
> Total MapReduce CPU Time Spent: 7 minutes 39 seconds 270 msec
>
>
>
>
>
> Any idea?
>
>
>
> Tks,
>
> *Thiago Henrique S. Bento*
>
> B2W Digital / B2W Labs | BIT SP
>
> thiago.bento@b2wdigital.com
>
> Tel.: +55 (11) 3454-3024
>
>
> .............................................................................................................................................................
>
> *B**2**W DIGITAL* #Americanas.com <http://americanas.com/> #Shoptime.com
> <http://shoptime.com/> #Submarino.com <http://submarino.com/> #
> SouBarato.com <http://soubarato.com/>
>
>
>
>
> ------------------------------
>
> Esta mensagem pode conter informações confidenciais e somente o indivíduo
> ou entidade a quem foi destinada pode utilizá-la. A transmissão incorreta
> da mensagem não acarreta a perda de sua confidencialidade. Caso esta
> mensagem tenha sido recebida por engano, solicitamos que o fato seja
> comunicado ao remetente e que a mensagem seja eliminada de seu sistema
> imediatamente. É vedado a qualquer pessoa que não seja o destinatário usar,
> revelar, distribuir ou copiar qualquer parte desta mensagem. Ambiente de
> comunicação sujeito a monitoramento.
>
> This message may include confidential information and only the intended
> addresses have the right to use it as is, or any part of it. A wrong
> transmission does not break its confidentiality. If you've received it
> because of a mistake or erroneous transmission, please notify the sender
> and delete it from your system immediately. This communication environment
> is controlled and monitored.
>
> B2W Digital
>
>
>
RES: RES: Join error with ORC Hive tables
Posted by Thiago Henrique dos Santos Bento <th...@b2wdigital.com>.
I created the table with stored as ORC, and I did a sqoop job from oracle to load data to this table.
This table have something like 65 million lines but when I load 10 million, the query runs normally.
I think that is problems with memory, my cluster doesn't have much memory.
De: Prasanth Jayachandran [mailto:pjayachandran@hortonworks.com]
Enviada em: Thursday, October 2, 2014 4:53 PM
Para: user@hive.apache.org
Assunto: Re: RES: Join error with ORC Hive tables
Caused by: java.io.IOException: Malformed ORC file hdfs://arnaldo01:8020/apps/hive/warehouse/mds.db/ds_customer_qg/part-m-00001. Invalid postscript.
>From the exception it looks like the file it is trying to read from table stored as ORC but the above file is not an ORC file. How is the ORC table loaded? My suspicion is some non-ORC file ended up in ORC table.
Thanks
Prasanth Jayachandran
On Oct 2, 2014, at 12:26 PM, Thiago Henrique dos Santos Bento <th...@b2wdigital.com>> wrote:
Erro:
2014-10-02 15:59:08,654 FATAL [IPC Server handler 7 on 46438] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1411750973876_0130_m_000023_1000 - exited : java.io.IOException: java.lang.reflect.InvocationTargetException
at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:302)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.<init>(HadoopShimsSecure.java:249)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(HadoopShimsSecure.java:363)
at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:591)
at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:168)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:409)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:288)
... 11 more
Caused by: java.io.IOException: Malformed ORC file hdfs://arnaldo01:8020/apps/hive/warehouse/mds.db/ds_customer_qg/part-m-00001. Invalid postscript.
at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.ensureOrcFooter(ReaderImpl.java:226)
at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.extractMetaInfoFromFooter(ReaderImpl.java:336)
at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.<init>(ReaderImpl.java:292)
at org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:197)
at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:999)
at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.<init>(CombineHiveRecordReader.java:65)
... 16 more
Hive Version: 0.13
Query:
SELECT t1.ID_TP_CLIENTE
, t1.COD_CLIENTE
, t1.DESC_MARCA
, t1.DOCUMENT_ID_NR
, t1.NM_CLIENTE
, t1.DT_NASCIMENTO
, t1.ID_TP_SEXO
, t1.DT_CRIACAO_CARGA
, t1.DT_LAST_CARGA
--
FROM ds_customer_qg t1
join ds_customer_qg t2
on t1.COD_CLIENTE = t2.COD_CLIENTE
De: Prasanth Jayachandran [mailto:pjayachandran@hortonworks.com]
Enviada em: Wednesday, October 1, 2014 3:24 PM
Para: user@hive.apache.org<ma...@hive.apache.org>
Assunto: Re: Join error with ORC Hive tables
Hi
Can you post the exception stacktrace from hadoop execution logs? What version of hive are you using?
Can you provide the join query that you are using?
Thanks
Prasanth Jayachandran
On Oct 1, 2014, at 7:41 AM, Thiago Henrique dos Santos Bento <th...@b2wdigital.com>> wrote:
Hi!
I'm trying to run a simple join with two tables stored as ORC File. (the same query runs normally without ORC File)
I got this error while map-reduce job:
Driver returned: 2. Errors: OK
Query ID = hue_20141001071313_33e31d68-3b4a-49d5-acc0-30c0a42276ab
Total jobs = 1
Stage-1 is selected by condition resolver.
Launching Job 1 out of 1
Number of reduce tasks not specified. Estimated from input data size: 12
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Starting Job = job_1411750973876_0099, Tracking URL = http://arnaldo02:8088/proxy/application_1411750973876_0099/
Kill Command = /usr/lib/hadoop/bin/hadoop job -kill job_1411750973876_0099
Hadoop job information for Stage-1: number of mappers: 42; number of reducers: 12
2014-10-01 07:13:51,965 Stage-1 map = 0%, reduce = 0%
2014-10-01 07:14:18,882 Stage-1 map = 2%, reduce = 0%, Cumulative CPU 37.32 sec
2014-10-01 07:14:19,990 Stage-1 map = 3%, reduce = 0%, Cumulative CPU 40.31 sec
2014-10-01 07:14:21,100 Stage-1 map = 5%, reduce = 0%, Cumulative CPU 43.33 sec
2014-10-01 07:14:25,544 Stage-1 map = 7%, reduce = 0%, Cumulative CPU 57.49 sec
2014-10-01 07:14:39,996 Stage-1 map = 8%, reduce = 0%, Cumulative CPU 73.08 sec
2014-10-01 07:14:43,338 Stage-1 map = 9%, reduce = 0%, Cumulative CPU 76.76 sec
2014-10-01 07:14:48,882 Stage-1 map = 11%, reduce = 0%, Cumulative CPU 92.88 sec
2014-10-01 07:14:51,107 Stage-1 map = 12%, reduce = 0%, Cumulative CPU 100.93 sec
2014-10-01 07:14:52,212 Stage-1 map = 13%, reduce = 0%, Cumulative CPU 106.04 sec
2014-10-01 07:14:54,426 Stage-1 map = 14%, reduce = 0%, Cumulative CPU 111.02 sec
2014-10-01 07:14:57,751 Stage-1 map = 15%, reduce = 0%, Cumulative CPU 126.41 sec
2014-10-01 07:15:05,507 Stage-1 map = 17%, reduce = 0%, Cumulative CPU 135.71 sec
2014-10-01 07:15:10,023 Stage-1 map = 18%, reduce = 0%, Cumulative CPU 146.69 sec
2014-10-01 07:15:19,987 Stage-1 map = 19%, reduce = 0%, Cumulative CPU 156.43 sec
2014-10-01 07:15:29,957 Stage-1 map = 19%, reduce = 1%, Cumulative CPU 168.3 sec
2014-10-01 07:15:37,703 Stage-1 map = 21%, reduce = 1%, Cumulative CPU 179.77 sec
2014-10-01 07:15:45,511 Stage-1 map = 23%, reduce = 1%, Cumulative CPU 192.87 sec
2014-10-01 07:15:49,948 Stage-1 map = 25%, reduce = 1%, Cumulative CPU 204.72 sec
2014-10-01 07:15:53,287 Stage-1 map = 26%, reduce = 1%, Cumulative CPU 211.83 sec
2014-10-01 07:16:05,456 Stage-1 map = 28%, reduce = 1%, Cumulative CPU 231.45 sec
2014-10-01 07:16:07,664 Stage-1 map = 29%, reduce = 1%, Cumulative CPU 233.46 sec
2014-10-01 07:16:09,876 Stage-1 map = 29%, reduce = 2%, Cumulative CPU 233.74 sec
2014-10-01 07:16:15,408 Stage-1 map = 32%, reduce = 2%, Cumulative CPU 255.51 sec
2014-10-01 07:16:18,727 Stage-1 map = 33%, reduce = 2%, Cumulative CPU 258.57 sec
2014-10-01 07:16:19,830 Stage-1 map = 33%, reduce = 3%, Cumulative CPU 261.95 sec
2014-10-01 07:16:35,288 Stage-1 map = 35%, reduce = 3%, Cumulative CPU 288.59 sec
2014-10-01 07:16:40,806 Stage-1 map = 37%, reduce = 3%, Cumulative CPU 298.89 sec
2014-10-01 07:16:45,278 Stage-1 map = 38%, reduce = 3%, Cumulative CPU 306.72 sec
2014-10-01 07:16:58,531 Stage-1 map = 38%, reduce = 4%, Cumulative CPU 312.76 sec
2014-10-01 07:17:02,945 Stage-1 map = 40%, reduce = 4%, Cumulative CPU 321.87 sec
2014-10-01 07:17:22,828 Stage-1 map = 42%, reduce = 4%, Cumulative CPU 336.71 sec
2014-10-01 07:17:28,341 Stage-1 map = 43%, reduce = 5%, Cumulative CPU 343.66 sec
2014-10-01 07:17:39,370 Stage-1 map = 44%, reduce = 5%, Cumulative CPU 350.7 sec
2014-10-01 07:17:51,541 Stage-1 map = 47%, reduce = 5%, Cumulative CPU 365.91 sec
2014-10-01 07:18:08,097 Stage-1 map = 49%, reduce = 5%, Cumulative CPU 375.35 sec
2014-10-01 07:18:22,452 Stage-1 map = 49%, reduce = 7%, Cumulative CPU 383.34 sec
2014-10-01 07:18:25,756 Stage-1 map = 50%, reduce = 7%, Cumulative CPU 383.69 sec
2014-10-01 07:19:26,488 Stage-1 map = 50%, reduce = 7%, Cumulative CPU 389.38 sec
2014-10-01 07:19:50,793 Stage-1 map = 52%, reduce = 7%, Cumulative CPU 391.28 sec
2014-10-01 07:19:58,551 Stage-1 map = 50%, reduce = 7%, Cumulative CPU 386.39 sec
2014-10-01 07:20:50,430 Stage-1 map = 52%, reduce = 7%, Cumulative CPU 393.79 sec
2014-10-01 07:21:51,114 Stage-1 map = 52%, reduce = 7%, Cumulative CPU 411.94 sec
2014-10-01 07:22:19,773 Stage-1 map = 54%, reduce = 7%, Cumulative CPU 417.31 sec
2014-10-01 07:23:20,493 Stage-1 map = 54%, reduce = 7%, Cumulative CPU 423.21 sec
2014-10-01 07:24:21,168 Stage-1 map = 54%, reduce = 7%, Cumulative CPU 425.94 sec
2014-10-01 07:25:21,814 Stage-1 map = 54%, reduce = 7%, Cumulative CPU 441.2 sec
2014-10-01 07:25:36,145 Stage-1 map = 55%, reduce = 7%, Cumulative CPU 442.7 sec
2014-10-01 07:26:36,837 Stage-1 map = 55%, reduce = 7%, Cumulative CPU 448.75 sec
2014-10-01 07:26:40,140 Stage-1 map = 55%, reduce = 8%, Cumulative CPU 449.35 sec
2014-10-01 07:27:11,064 Stage-1 map = 56%, reduce = 8%, Cumulative CPU 455.24 sec
2014-10-01 07:27:19,871 Stage-1 map = 57%, reduce = 8%, Cumulative CPU 460.35 sec
2014-10-01 07:27:56,277 Stage-1 map = 58%, reduce = 8%, Cumulative CPU 481.34 sec
2014-10-01 07:28:40,332 Stage-1 map = 59%, reduce = 8%, Cumulative CPU 504.81 sec
2014-10-01 07:29:40,957 Stage-1 map = 59%, reduce = 8%, Cumulative CPU 510.71 sec
2014-10-01 07:30:41,638 Stage-1 map = 59%, reduce = 8%, Cumulative CPU 513.74 sec
2014-10-01 07:31:00,419 Stage-1 map = 60%, reduce = 8%, Cumulative CPU 514.6 sec
2014-10-01 07:32:01,081 Stage-1 map = 60%, reduce = 8%, Cumulative CPU 518.18 sec
2014-10-01 07:32:50,724 Stage-1 map = 62%, reduce = 8%, Cumulative CPU 520.43 sec
2014-10-01 07:33:00,648 Stage-1 map = 60%, reduce = 8%, Cumulative CPU 521.71 sec
2014-10-01 07:33:14,974 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 459.27 sec
MapReduce Total cumulative CPU time: 7 minutes 39 seconds 270 msec
Ended Job = job_1411750973876_0099 with errors
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:
Job 0: Map: 42 Reduce: 12 Cumulative CPU: 459.27 sec HDFS Read: 1989875843 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 7 minutes 39 seconds 270 msec
Any idea?
Tks,
Thiago Henrique S. Bento
B2W Digital / B2W Labs | BIT SP
thiago.bento@b2wdigital.com<ma...@b2wdigital.com>
Tel.: +55 (11) 3454-3024
.............................................................................................................................................................
B2W DIGITAL #Americanas.com<http://americanas.com/> #Shoptime.com<http://shoptime.com/> #Submarino.com<http://submarino.com/> #SouBarato.com<http://soubarato.com/>
________________________________
Esta mensagem pode conter informações confidenciais e somente o indivíduo ou entidade a quem foi destinada pode utilizá-la. A transmissão incorreta da mensagem não acarreta a perda de sua confidencialidade. Caso esta mensagem tenha sido recebida por engano, solicitamos que o fato seja comunicado ao remetente e que a mensagem seja eliminada de seu sistema imediatamente. É vedado a qualquer pessoa que não seja o destinatário usar, revelar, distribuir ou copiar qualquer parte desta mensagem. Ambiente de comunicação sujeito a monitoramento.
This message may include confidential information and only the intended addresses have the right to use it as is, or any part of it. A wrong transmission does not break its confidentiality. If you've received it because of a mistake or erroneous transmission, please notify the sender and delete it from your system immediately. This communication environment is controlled and monitored.
B2W Digital
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: RES: Join error with ORC Hive tables
Posted by Prasanth Jayachandran <pj...@hortonworks.com>.
Caused by: java.io.IOException: Malformed ORC file hdfs://arnaldo01:8020/apps/hive/warehouse/mds.db/ds_customer_qg/part-m-00001. Invalid postscript.
>From the exception it looks like the file it is trying to read from table stored as ORC but the above file is not an ORC file. How is the ORC table loaded? My suspicion is some non-ORC file ended up in ORC table.
Thanks
Prasanth Jayachandran
On Oct 2, 2014, at 12:26 PM, Thiago Henrique dos Santos Bento <th...@b2wdigital.com> wrote:
> Erro:
>
> 2014-10-02 15:59:08,654 FATAL [IPC Server handler 7 on 46438] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1411750973876_0130_m_000023_1000 - exited : java.io.IOException: java.lang.reflect.InvocationTargetException
>
> at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
>
> at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
>
> at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:302)
>
> at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.<init>(HadoopShimsSecure.java:249)
>
> at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(HadoopShimsSecure.java:363)
>
> at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:591)
>
> at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:168)
>
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:409)
>
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
>
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:415)
>
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
>
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
>
> Caused by: java.lang.reflect.InvocationTargetException
>
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>
> at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:288)
>
> ... 11 more
>
> Caused by: java.io.IOException: Malformed ORC file hdfs://arnaldo01:8020/apps/hive/warehouse/mds.db/ds_customer_qg/part-m-00001. Invalid postscript.
>
> at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.ensureOrcFooter(ReaderImpl.java:226)
>
> at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.extractMetaInfoFromFooter(ReaderImpl.java:336)
>
> at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.<init>(ReaderImpl.java:292)
>
> at org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:197)
>
> at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:999)
>
> at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.<init>(CombineHiveRecordReader.java:65)
>
> ... 16 more
>
>
>
>
> Hive Version: 0.13
>
> Query:
> SELECT t1.ID_TP_CLIENTE
> , t1.COD_CLIENTE
> , t1.DESC_MARCA
> , t1.DOCUMENT_ID_NR
> , t1.NM_CLIENTE
> , t1.DT_NASCIMENTO
> , t1.ID_TP_SEXO
> , t1.DT_CRIACAO_CARGA
> , t1.DT_LAST_CARGA
> --
> FROM ds_customer_qg t1
> join ds_customer_qg t2
> on t1.COD_CLIENTE = t2.COD_CLIENTE
>
>
>
>
> De: Prasanth Jayachandran [mailto:pjayachandran@hortonworks.com]
> Enviada em: Wednesday, October 1, 2014 3:24 PM
> Para: user@hive.apache.org
> Assunto: Re: Join error with ORC Hive tables
>
> Hi
>
> Can you post the exception stacktrace from hadoop execution logs? What version of hive are you using?
> Can you provide the join query that you are using?
>
> Thanks
> Prasanth Jayachandran
>
> On Oct 1, 2014, at 7:41 AM, Thiago Henrique dos Santos Bento <th...@b2wdigital.com> wrote:
>
>
> Hi!
>
> I’m trying to run a simple join with two tables stored as ORC File. (the same query runs normally without ORC File)
> I got this error while map-reduce job:
> Driver returned: 2. Errors: OK
>
> Query ID = hue_20141001071313_33e31d68-3b4a-49d5-acc0-30c0a42276ab
>
> Total jobs = 1
>
> Stage-1 is selected by condition resolver.
>
> Launching Job 1 out of 1
>
> Number of reduce tasks not specified. Estimated from input data size: 12
>
> In order to change the average load for a reducer (in bytes):
>
> set hive.exec.reducers.bytes.per.reducer=<number>
>
> In order to limit the maximum number of reducers:
>
> set hive.exec.reducers.max=<number>
>
> In order to set a constant number of reducers:
>
> set mapreduce.job.reduces=<number>
>
> Starting Job = job_1411750973876_0099, Tracking URL = http://arnaldo02:8088/proxy/application_1411750973876_0099/
>
> Kill Command = /usr/lib/hadoop/bin/hadoop job -kill job_1411750973876_0099
>
> Hadoop job information for Stage-1: number of mappers: 42; number of reducers: 12
>
> 2014-10-01 07:13:51,965 Stage-1 map = 0%, reduce = 0%
>
> 2014-10-01 07:14:18,882 Stage-1 map = 2%, reduce = 0%, Cumulative CPU 37.32 sec
>
> 2014-10-01 07:14:19,990 Stage-1 map = 3%, reduce = 0%, Cumulative CPU 40.31 sec
>
> 2014-10-01 07:14:21,100 Stage-1 map = 5%, reduce = 0%, Cumulative CPU 43.33 sec
>
> 2014-10-01 07:14:25,544 Stage-1 map = 7%, reduce = 0%, Cumulative CPU 57.49 sec
>
> 2014-10-01 07:14:39,996 Stage-1 map = 8%, reduce = 0%, Cumulative CPU 73.08 sec
>
> 2014-10-01 07:14:43,338 Stage-1 map = 9%, reduce = 0%, Cumulative CPU 76.76 sec
>
> 2014-10-01 07:14:48,882 Stage-1 map = 11%, reduce = 0%, Cumulative CPU 92.88 sec
>
> 2014-10-01 07:14:51,107 Stage-1 map = 12%, reduce = 0%, Cumulative CPU 100.93 sec
>
> 2014-10-01 07:14:52,212 Stage-1 map = 13%, reduce = 0%, Cumulative CPU 106.04 sec
>
> 2014-10-01 07:14:54,426 Stage-1 map = 14%, reduce = 0%, Cumulative CPU 111.02 sec
>
> 2014-10-01 07:14:57,751 Stage-1 map = 15%, reduce = 0%, Cumulative CPU 126.41 sec
>
> 2014-10-01 07:15:05,507 Stage-1 map = 17%, reduce = 0%, Cumulative CPU 135.71 sec
>
> 2014-10-01 07:15:10,023 Stage-1 map = 18%, reduce = 0%, Cumulative CPU 146.69 sec
>
> 2014-10-01 07:15:19,987 Stage-1 map = 19%, reduce = 0%, Cumulative CPU 156.43 sec
>
> 2014-10-01 07:15:29,957 Stage-1 map = 19%, reduce = 1%, Cumulative CPU 168.3 sec
>
> 2014-10-01 07:15:37,703 Stage-1 map = 21%, reduce = 1%, Cumulative CPU 179.77 sec
>
> 2014-10-01 07:15:45,511 Stage-1 map = 23%, reduce = 1%, Cumulative CPU 192.87 sec
>
> 2014-10-01 07:15:49,948 Stage-1 map = 25%, reduce = 1%, Cumulative CPU 204.72 sec
>
> 2014-10-01 07:15:53,287 Stage-1 map = 26%, reduce = 1%, Cumulative CPU 211.83 sec
>
> 2014-10-01 07:16:05,456 Stage-1 map = 28%, reduce = 1%, Cumulative CPU 231.45 sec
>
> 2014-10-01 07:16:07,664 Stage-1 map = 29%, reduce = 1%, Cumulative CPU 233.46 sec
>
> 2014-10-01 07:16:09,876 Stage-1 map = 29%, reduce = 2%, Cumulative CPU 233.74 sec
>
> 2014-10-01 07:16:15,408 Stage-1 map = 32%, reduce = 2%, Cumulative CPU 255.51 sec
>
> 2014-10-01 07:16:18,727 Stage-1 map = 33%, reduce = 2%, Cumulative CPU 258.57 sec
>
> 2014-10-01 07:16:19,830 Stage-1 map = 33%, reduce = 3%, Cumulative CPU 261.95 sec
>
> 2014-10-01 07:16:35,288 Stage-1 map = 35%, reduce = 3%, Cumulative CPU 288.59 sec
>
> 2014-10-01 07:16:40,806 Stage-1 map = 37%, reduce = 3%, Cumulative CPU 298.89 sec
>
> 2014-10-01 07:16:45,278 Stage-1 map = 38%, reduce = 3%, Cumulative CPU 306.72 sec
>
> 2014-10-01 07:16:58,531 Stage-1 map = 38%, reduce = 4%, Cumulative CPU 312.76 sec
>
> 2014-10-01 07:17:02,945 Stage-1 map = 40%, reduce = 4%, Cumulative CPU 321.87 sec
>
> 2014-10-01 07:17:22,828 Stage-1 map = 42%, reduce = 4%, Cumulative CPU 336.71 sec
>
> 2014-10-01 07:17:28,341 Stage-1 map = 43%, reduce = 5%, Cumulative CPU 343.66 sec
>
> 2014-10-01 07:17:39,370 Stage-1 map = 44%, reduce = 5%, Cumulative CPU 350.7 sec
>
> 2014-10-01 07:17:51,541 Stage-1 map = 47%, reduce = 5%, Cumulative CPU 365.91 sec
>
> 2014-10-01 07:18:08,097 Stage-1 map = 49%, reduce = 5%, Cumulative CPU 375.35 sec
>
> 2014-10-01 07:18:22,452 Stage-1 map = 49%, reduce = 7%, Cumulative CPU 383.34 sec
>
> 2014-10-01 07:18:25,756 Stage-1 map = 50%, reduce = 7%, Cumulative CPU 383.69 sec
>
> 2014-10-01 07:19:26,488 Stage-1 map = 50%, reduce = 7%, Cumulative CPU 389.38 sec
>
> 2014-10-01 07:19:50,793 Stage-1 map = 52%, reduce = 7%, Cumulative CPU 391.28 sec
>
> 2014-10-01 07:19:58,551 Stage-1 map = 50%, reduce = 7%, Cumulative CPU 386.39 sec
>
> 2014-10-01 07:20:50,430 Stage-1 map = 52%, reduce = 7%, Cumulative CPU 393.79 sec
>
> 2014-10-01 07:21:51,114 Stage-1 map = 52%, reduce = 7%, Cumulative CPU 411.94 sec
>
> 2014-10-01 07:22:19,773 Stage-1 map = 54%, reduce = 7%, Cumulative CPU 417.31 sec
>
> 2014-10-01 07:23:20,493 Stage-1 map = 54%, reduce = 7%, Cumulative CPU 423.21 sec
>
> 2014-10-01 07:24:21,168 Stage-1 map = 54%, reduce = 7%, Cumulative CPU 425.94 sec
>
> 2014-10-01 07:25:21,814 Stage-1 map = 54%, reduce = 7%, Cumulative CPU 441.2 sec
>
> 2014-10-01 07:25:36,145 Stage-1 map = 55%, reduce = 7%, Cumulative CPU 442.7 sec
>
> 2014-10-01 07:26:36,837 Stage-1 map = 55%, reduce = 7%, Cumulative CPU 448.75 sec
>
> 2014-10-01 07:26:40,140 Stage-1 map = 55%, reduce = 8%, Cumulative CPU 449.35 sec
>
> 2014-10-01 07:27:11,064 Stage-1 map = 56%, reduce = 8%, Cumulative CPU 455.24 sec
>
> 2014-10-01 07:27:19,871 Stage-1 map = 57%, reduce = 8%, Cumulative CPU 460.35 sec
>
> 2014-10-01 07:27:56,277 Stage-1 map = 58%, reduce = 8%, Cumulative CPU 481.34 sec
>
> 2014-10-01 07:28:40,332 Stage-1 map = 59%, reduce = 8%, Cumulative CPU 504.81 sec
>
> 2014-10-01 07:29:40,957 Stage-1 map = 59%, reduce = 8%, Cumulative CPU 510.71 sec
>
> 2014-10-01 07:30:41,638 Stage-1 map = 59%, reduce = 8%, Cumulative CPU 513.74 sec
>
> 2014-10-01 07:31:00,419 Stage-1 map = 60%, reduce = 8%, Cumulative CPU 514.6 sec
>
> 2014-10-01 07:32:01,081 Stage-1 map = 60%, reduce = 8%, Cumulative CPU 518.18 sec
>
> 2014-10-01 07:32:50,724 Stage-1 map = 62%, reduce = 8%, Cumulative CPU 520.43 sec
>
> 2014-10-01 07:33:00,648 Stage-1 map = 60%, reduce = 8%, Cumulative CPU 521.71 sec
>
> 2014-10-01 07:33:14,974 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 459.27 sec
>
> MapReduce Total cumulative CPU time: 7 minutes 39 seconds 270 msec
>
> Ended Job = job_1411750973876_0099 with errors
>
> FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
>
> MapReduce Jobs Launched:
>
> Job 0: Map: 42 Reduce: 12 Cumulative CPU: 459.27 sec HDFS Read: 1989875843 HDFS Write: 0 FAIL
>
> Total MapReduce CPU Time Spent: 7 minutes 39 seconds 270 msec
>
>
>
> Any idea?
>
> Tks,
> Thiago Henrique S. Bento
> B2W Digital / B2W Labs | BIT SP
> thiago.bento@b2wdigital.com
> Tel.: +55 (11) 3454-3024
> .............................................................................................................................................................
> B2W DIGITAL #Americanas.com #Shoptime.com #Submarino.com #SouBarato.com
>
>
> Esta mensagem pode conter informações confidenciais e somente o indivíduo ou entidade a quem foi destinada pode utilizá-la. A transmissão incorreta da mensagem não acarreta a perda de sua confidencialidade. Caso esta mensagem tenha sido recebida por engano, solicitamos que o fato seja comunicado ao remetente e que a mensagem seja eliminada de seu sistema imediatamente. É vedado a qualquer pessoa que não seja o destinatário usar, revelar, distribuir ou copiar qualquer parte desta mensagem. Ambiente de comunicação sujeito a monitoramento.
>
> This message may include confidential information and only the intended addresses have the right to use it as is, or any part of it. A wrong transmission does not break its confidentiality. If you've received it because of a mistake or erroneous transmission, please notify the sender and delete it from your system immediately. This communication environment is controlled and monitored.
>
> B2W Digital
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
RES: Join error with ORC Hive tables
Posted by Thiago Henrique dos Santos Bento <th...@b2wdigital.com>.
Erro:
2014-10-02 15:59:08,654 FATAL [IPC Server handler 7 on 46438] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1411750973876_0130_m_000023_1000 - exited : java.io.IOException: java.lang.reflect.InvocationTargetException
at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:302)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.<init>(HadoopShimsSecure.java:249)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(HadoopShimsSecure.java:363)
at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:591)
at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:168)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:409)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:288)
... 11 more
Caused by: java.io.IOException: Malformed ORC file hdfs://arnaldo01:8020/apps/hive/warehouse/mds.db/ds_customer_qg/part-m-00001. Invalid postscript.
at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.ensureOrcFooter(ReaderImpl.java:226)
at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.extractMetaInfoFromFooter(ReaderImpl.java:336)
at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.<init>(ReaderImpl.java:292)
at org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:197)
at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:999)
at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.<init>(CombineHiveRecordReader.java:65)
... 16 more
Hive Version: 0.13
Query:
SELECT t1.ID_TP_CLIENTE
, t1.COD_CLIENTE
, t1.DESC_MARCA
, t1.DOCUMENT_ID_NR
, t1.NM_CLIENTE
, t1.DT_NASCIMENTO
, t1.ID_TP_SEXO
, t1.DT_CRIACAO_CARGA
, t1.DT_LAST_CARGA
--
FROM ds_customer_qg t1
join ds_customer_qg t2
on t1.COD_CLIENTE = t2.COD_CLIENTE
De: Prasanth Jayachandran [mailto:pjayachandran@hortonworks.com]
Enviada em: Wednesday, October 1, 2014 3:24 PM
Para: user@hive.apache.org
Assunto: Re: Join error with ORC Hive tables
Hi
Can you post the exception stacktrace from hadoop execution logs? What version of hive are you using?
Can you provide the join query that you are using?
Thanks
Prasanth Jayachandran
On Oct 1, 2014, at 7:41 AM, Thiago Henrique dos Santos Bento <th...@b2wdigital.com>> wrote:
Hi!
I'm trying to run a simple join with two tables stored as ORC File. (the same query runs normally without ORC File)
I got this error while map-reduce job:
Driver returned: 2. Errors: OK
Query ID = hue_20141001071313_33e31d68-3b4a-49d5-acc0-30c0a42276ab
Total jobs = 1
Stage-1 is selected by condition resolver.
Launching Job 1 out of 1
Number of reduce tasks not specified. Estimated from input data size: 12
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Starting Job = job_1411750973876_0099, Tracking URL = http://arnaldo02:8088/proxy/application_1411750973876_0099/
Kill Command = /usr/lib/hadoop/bin/hadoop job -kill job_1411750973876_0099
Hadoop job information for Stage-1: number of mappers: 42; number of reducers: 12
2014-10-01 07:13:51,965 Stage-1 map = 0%, reduce = 0%
2014-10-01 07:14:18,882 Stage-1 map = 2%, reduce = 0%, Cumulative CPU 37.32 sec
2014-10-01 07:14:19,990 Stage-1 map = 3%, reduce = 0%, Cumulative CPU 40.31 sec
2014-10-01 07:14:21,100 Stage-1 map = 5%, reduce = 0%, Cumulative CPU 43.33 sec
2014-10-01 07:14:25,544 Stage-1 map = 7%, reduce = 0%, Cumulative CPU 57.49 sec
2014-10-01 07:14:39,996 Stage-1 map = 8%, reduce = 0%, Cumulative CPU 73.08 sec
2014-10-01 07:14:43,338 Stage-1 map = 9%, reduce = 0%, Cumulative CPU 76.76 sec
2014-10-01 07:14:48,882 Stage-1 map = 11%, reduce = 0%, Cumulative CPU 92.88 sec
2014-10-01 07:14:51,107 Stage-1 map = 12%, reduce = 0%, Cumulative CPU 100.93 sec
2014-10-01 07:14:52,212 Stage-1 map = 13%, reduce = 0%, Cumulative CPU 106.04 sec
2014-10-01 07:14:54,426 Stage-1 map = 14%, reduce = 0%, Cumulative CPU 111.02 sec
2014-10-01 07:14:57,751 Stage-1 map = 15%, reduce = 0%, Cumulative CPU 126.41 sec
2014-10-01 07:15:05,507 Stage-1 map = 17%, reduce = 0%, Cumulative CPU 135.71 sec
2014-10-01 07:15:10,023 Stage-1 map = 18%, reduce = 0%, Cumulative CPU 146.69 sec
2014-10-01 07:15:19,987 Stage-1 map = 19%, reduce = 0%, Cumulative CPU 156.43 sec
2014-10-01 07:15:29,957 Stage-1 map = 19%, reduce = 1%, Cumulative CPU 168.3 sec
2014-10-01 07:15:37,703 Stage-1 map = 21%, reduce = 1%, Cumulative CPU 179.77 sec
2014-10-01 07:15:45,511 Stage-1 map = 23%, reduce = 1%, Cumulative CPU 192.87 sec
2014-10-01 07:15:49,948 Stage-1 map = 25%, reduce = 1%, Cumulative CPU 204.72 sec
2014-10-01 07:15:53,287 Stage-1 map = 26%, reduce = 1%, Cumulative CPU 211.83 sec
2014-10-01 07:16:05,456 Stage-1 map = 28%, reduce = 1%, Cumulative CPU 231.45 sec
2014-10-01 07:16:07,664 Stage-1 map = 29%, reduce = 1%, Cumulative CPU 233.46 sec
2014-10-01 07:16:09,876 Stage-1 map = 29%, reduce = 2%, Cumulative CPU 233.74 sec
2014-10-01 07:16:15,408 Stage-1 map = 32%, reduce = 2%, Cumulative CPU 255.51 sec
2014-10-01 07:16:18,727 Stage-1 map = 33%, reduce = 2%, Cumulative CPU 258.57 sec
2014-10-01 07:16:19,830 Stage-1 map = 33%, reduce = 3%, Cumulative CPU 261.95 sec
2014-10-01 07:16:35,288 Stage-1 map = 35%, reduce = 3%, Cumulative CPU 288.59 sec
2014-10-01 07:16:40,806 Stage-1 map = 37%, reduce = 3%, Cumulative CPU 298.89 sec
2014-10-01 07:16:45,278 Stage-1 map = 38%, reduce = 3%, Cumulative CPU 306.72 sec
2014-10-01 07:16:58,531 Stage-1 map = 38%, reduce = 4%, Cumulative CPU 312.76 sec
2014-10-01 07:17:02,945 Stage-1 map = 40%, reduce = 4%, Cumulative CPU 321.87 sec
2014-10-01 07:17:22,828 Stage-1 map = 42%, reduce = 4%, Cumulative CPU 336.71 sec
2014-10-01 07:17:28,341 Stage-1 map = 43%, reduce = 5%, Cumulative CPU 343.66 sec
2014-10-01 07:17:39,370 Stage-1 map = 44%, reduce = 5%, Cumulative CPU 350.7 sec
2014-10-01 07:17:51,541 Stage-1 map = 47%, reduce = 5%, Cumulative CPU 365.91 sec
2014-10-01 07:18:08,097 Stage-1 map = 49%, reduce = 5%, Cumulative CPU 375.35 sec
2014-10-01 07:18:22,452 Stage-1 map = 49%, reduce = 7%, Cumulative CPU 383.34 sec
2014-10-01 07:18:25,756 Stage-1 map = 50%, reduce = 7%, Cumulative CPU 383.69 sec
2014-10-01 07:19:26,488 Stage-1 map = 50%, reduce = 7%, Cumulative CPU 389.38 sec
2014-10-01 07:19:50,793 Stage-1 map = 52%, reduce = 7%, Cumulative CPU 391.28 sec
2014-10-01 07:19:58,551 Stage-1 map = 50%, reduce = 7%, Cumulative CPU 386.39 sec
2014-10-01 07:20:50,430 Stage-1 map = 52%, reduce = 7%, Cumulative CPU 393.79 sec
2014-10-01 07:21:51,114 Stage-1 map = 52%, reduce = 7%, Cumulative CPU 411.94 sec
2014-10-01 07:22:19,773 Stage-1 map = 54%, reduce = 7%, Cumulative CPU 417.31 sec
2014-10-01 07:23:20,493 Stage-1 map = 54%, reduce = 7%, Cumulative CPU 423.21 sec
2014-10-01 07:24:21,168 Stage-1 map = 54%, reduce = 7%, Cumulative CPU 425.94 sec
2014-10-01 07:25:21,814 Stage-1 map = 54%, reduce = 7%, Cumulative CPU 441.2 sec
2014-10-01 07:25:36,145 Stage-1 map = 55%, reduce = 7%, Cumulative CPU 442.7 sec
2014-10-01 07:26:36,837 Stage-1 map = 55%, reduce = 7%, Cumulative CPU 448.75 sec
2014-10-01 07:26:40,140 Stage-1 map = 55%, reduce = 8%, Cumulative CPU 449.35 sec
2014-10-01 07:27:11,064 Stage-1 map = 56%, reduce = 8%, Cumulative CPU 455.24 sec
2014-10-01 07:27:19,871 Stage-1 map = 57%, reduce = 8%, Cumulative CPU 460.35 sec
2014-10-01 07:27:56,277 Stage-1 map = 58%, reduce = 8%, Cumulative CPU 481.34 sec
2014-10-01 07:28:40,332 Stage-1 map = 59%, reduce = 8%, Cumulative CPU 504.81 sec
2014-10-01 07:29:40,957 Stage-1 map = 59%, reduce = 8%, Cumulative CPU 510.71 sec
2014-10-01 07:30:41,638 Stage-1 map = 59%, reduce = 8%, Cumulative CPU 513.74 sec
2014-10-01 07:31:00,419 Stage-1 map = 60%, reduce = 8%, Cumulative CPU 514.6 sec
2014-10-01 07:32:01,081 Stage-1 map = 60%, reduce = 8%, Cumulative CPU 518.18 sec
2014-10-01 07:32:50,724 Stage-1 map = 62%, reduce = 8%, Cumulative CPU 520.43 sec
2014-10-01 07:33:00,648 Stage-1 map = 60%, reduce = 8%, Cumulative CPU 521.71 sec
2014-10-01 07:33:14,974 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 459.27 sec
MapReduce Total cumulative CPU time: 7 minutes 39 seconds 270 msec
Ended Job = job_1411750973876_0099 with errors
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:
Job 0: Map: 42 Reduce: 12 Cumulative CPU: 459.27 sec HDFS Read: 1989875843 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 7 minutes 39 seconds 270 msec
Any idea?
Tks,
Thiago Henrique S. Bento
B2W Digital / B2W Labs | BIT SP
thiago.bento@b2wdigital.com<ma...@b2wdigital.com>
Tel.: +55 (11) 3454-3024
.............................................................................................................................................................
B2W DIGITAL #Americanas.com<http://americanas.com/> #Shoptime.com<http://shoptime.com/> #Submarino.com<http://submarino.com/> #SouBarato.com<http://soubarato.com/>
________________________________
Esta mensagem pode conter informações confidenciais e somente o indivíduo ou entidade a quem foi destinada pode utilizá-la. A transmissão incorreta da mensagem não acarreta a perda de sua confidencialidade. Caso esta mensagem tenha sido recebida por engano, solicitamos que o fato seja comunicado ao remetente e que a mensagem seja eliminada de seu sistema imediatamente. É vedado a qualquer pessoa que não seja o destinatário usar, revelar, distribuir ou copiar qualquer parte desta mensagem. Ambiente de comunicação sujeito a monitoramento.
This message may include confidential information and only the intended addresses have the right to use it as is, or any part of it. A wrong transmission does not break its confidentiality. If you've received it because of a mistake or erroneous transmission, please notify the sender and delete it from your system immediately. This communication environment is controlled and monitored.
B2W Digital
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Join error with ORC Hive tables
Posted by Prasanth Jayachandran <pj...@hortonworks.com>.
Hi
Can you post the exception stacktrace from hadoop execution logs? What version of hive are you using?
Can you provide the join query that you are using?
Thanks
Prasanth Jayachandran
On Oct 1, 2014, at 7:41 AM, Thiago Henrique dos Santos Bento <th...@b2wdigital.com> wrote:
> Hi!
>
> I’m trying to run a simple join with two tables stored as ORC File. (the same query runs normally without ORC File)
> I got this error while map-reduce job:
> Driver returned: 2. Errors: OK
>
> Query ID = hue_20141001071313_33e31d68-3b4a-49d5-acc0-30c0a42276ab
>
> Total jobs = 1
>
> Stage-1 is selected by condition resolver.
>
> Launching Job 1 out of 1
>
> Number of reduce tasks not specified. Estimated from input data size: 12
>
> In order to change the average load for a reducer (in bytes):
>
> set hive.exec.reducers.bytes.per.reducer=<number>
>
> In order to limit the maximum number of reducers:
>
> set hive.exec.reducers.max=<number>
>
> In order to set a constant number of reducers:
>
> set mapreduce.job.reduces=<number>
>
> Starting Job = job_1411750973876_0099, Tracking URL = http://arnaldo02:8088/proxy/application_1411750973876_0099/
>
> Kill Command = /usr/lib/hadoop/bin/hadoop job -kill job_1411750973876_0099
>
> Hadoop job information for Stage-1: number of mappers: 42; number of reducers: 12
>
> 2014-10-01 07:13:51,965 Stage-1 map = 0%, reduce = 0%
>
> 2014-10-01 07:14:18,882 Stage-1 map = 2%, reduce = 0%, Cumulative CPU 37.32 sec
>
> 2014-10-01 07:14:19,990 Stage-1 map = 3%, reduce = 0%, Cumulative CPU 40.31 sec
>
> 2014-10-01 07:14:21,100 Stage-1 map = 5%, reduce = 0%, Cumulative CPU 43.33 sec
>
> 2014-10-01 07:14:25,544 Stage-1 map = 7%, reduce = 0%, Cumulative CPU 57.49 sec
>
> 2014-10-01 07:14:39,996 Stage-1 map = 8%, reduce = 0%, Cumulative CPU 73.08 sec
>
> 2014-10-01 07:14:43,338 Stage-1 map = 9%, reduce = 0%, Cumulative CPU 76.76 sec
>
> 2014-10-01 07:14:48,882 Stage-1 map = 11%, reduce = 0%, Cumulative CPU 92.88 sec
>
> 2014-10-01 07:14:51,107 Stage-1 map = 12%, reduce = 0%, Cumulative CPU 100.93 sec
>
> 2014-10-01 07:14:52,212 Stage-1 map = 13%, reduce = 0%, Cumulative CPU 106.04 sec
>
> 2014-10-01 07:14:54,426 Stage-1 map = 14%, reduce = 0%, Cumulative CPU 111.02 sec
>
> 2014-10-01 07:14:57,751 Stage-1 map = 15%, reduce = 0%, Cumulative CPU 126.41 sec
>
> 2014-10-01 07:15:05,507 Stage-1 map = 17%, reduce = 0%, Cumulative CPU 135.71 sec
>
> 2014-10-01 07:15:10,023 Stage-1 map = 18%, reduce = 0%, Cumulative CPU 146.69 sec
>
> 2014-10-01 07:15:19,987 Stage-1 map = 19%, reduce = 0%, Cumulative CPU 156.43 sec
>
> 2014-10-01 07:15:29,957 Stage-1 map = 19%, reduce = 1%, Cumulative CPU 168.3 sec
>
> 2014-10-01 07:15:37,703 Stage-1 map = 21%, reduce = 1%, Cumulative CPU 179.77 sec
>
> 2014-10-01 07:15:45,511 Stage-1 map = 23%, reduce = 1%, Cumulative CPU 192.87 sec
>
> 2014-10-01 07:15:49,948 Stage-1 map = 25%, reduce = 1%, Cumulative CPU 204.72 sec
>
> 2014-10-01 07:15:53,287 Stage-1 map = 26%, reduce = 1%, Cumulative CPU 211.83 sec
>
> 2014-10-01 07:16:05,456 Stage-1 map = 28%, reduce = 1%, Cumulative CPU 231.45 sec
>
> 2014-10-01 07:16:07,664 Stage-1 map = 29%, reduce = 1%, Cumulative CPU 233.46 sec
>
> 2014-10-01 07:16:09,876 Stage-1 map = 29%, reduce = 2%, Cumulative CPU 233.74 sec
>
> 2014-10-01 07:16:15,408 Stage-1 map = 32%, reduce = 2%, Cumulative CPU 255.51 sec
>
> 2014-10-01 07:16:18,727 Stage-1 map = 33%, reduce = 2%, Cumulative CPU 258.57 sec
>
> 2014-10-01 07:16:19,830 Stage-1 map = 33%, reduce = 3%, Cumulative CPU 261.95 sec
>
> 2014-10-01 07:16:35,288 Stage-1 map = 35%, reduce = 3%, Cumulative CPU 288.59 sec
>
> 2014-10-01 07:16:40,806 Stage-1 map = 37%, reduce = 3%, Cumulative CPU 298.89 sec
>
> 2014-10-01 07:16:45,278 Stage-1 map = 38%, reduce = 3%, Cumulative CPU 306.72 sec
>
> 2014-10-01 07:16:58,531 Stage-1 map = 38%, reduce = 4%, Cumulative CPU 312.76 sec
>
> 2014-10-01 07:17:02,945 Stage-1 map = 40%, reduce = 4%, Cumulative CPU 321.87 sec
>
> 2014-10-01 07:17:22,828 Stage-1 map = 42%, reduce = 4%, Cumulative CPU 336.71 sec
>
> 2014-10-01 07:17:28,341 Stage-1 map = 43%, reduce = 5%, Cumulative CPU 343.66 sec
>
> 2014-10-01 07:17:39,370 Stage-1 map = 44%, reduce = 5%, Cumulative CPU 350.7 sec
>
> 2014-10-01 07:17:51,541 Stage-1 map = 47%, reduce = 5%, Cumulative CPU 365.91 sec
>
> 2014-10-01 07:18:08,097 Stage-1 map = 49%, reduce = 5%, Cumulative CPU 375.35 sec
>
> 2014-10-01 07:18:22,452 Stage-1 map = 49%, reduce = 7%, Cumulative CPU 383.34 sec
>
> 2014-10-01 07:18:25,756 Stage-1 map = 50%, reduce = 7%, Cumulative CPU 383.69 sec
>
> 2014-10-01 07:19:26,488 Stage-1 map = 50%, reduce = 7%, Cumulative CPU 389.38 sec
>
> 2014-10-01 07:19:50,793 Stage-1 map = 52%, reduce = 7%, Cumulative CPU 391.28 sec
>
> 2014-10-01 07:19:58,551 Stage-1 map = 50%, reduce = 7%, Cumulative CPU 386.39 sec
>
> 2014-10-01 07:20:50,430 Stage-1 map = 52%, reduce = 7%, Cumulative CPU 393.79 sec
>
> 2014-10-01 07:21:51,114 Stage-1 map = 52%, reduce = 7%, Cumulative CPU 411.94 sec
>
> 2014-10-01 07:22:19,773 Stage-1 map = 54%, reduce = 7%, Cumulative CPU 417.31 sec
>
> 2014-10-01 07:23:20,493 Stage-1 map = 54%, reduce = 7%, Cumulative CPU 423.21 sec
>
> 2014-10-01 07:24:21,168 Stage-1 map = 54%, reduce = 7%, Cumulative CPU 425.94 sec
>
> 2014-10-01 07:25:21,814 Stage-1 map = 54%, reduce = 7%, Cumulative CPU 441.2 sec
>
> 2014-10-01 07:25:36,145 Stage-1 map = 55%, reduce = 7%, Cumulative CPU 442.7 sec
>
> 2014-10-01 07:26:36,837 Stage-1 map = 55%, reduce = 7%, Cumulative CPU 448.75 sec
>
> 2014-10-01 07:26:40,140 Stage-1 map = 55%, reduce = 8%, Cumulative CPU 449.35 sec
>
> 2014-10-01 07:27:11,064 Stage-1 map = 56%, reduce = 8%, Cumulative CPU 455.24 sec
>
> 2014-10-01 07:27:19,871 Stage-1 map = 57%, reduce = 8%, Cumulative CPU 460.35 sec
>
> 2014-10-01 07:27:56,277 Stage-1 map = 58%, reduce = 8%, Cumulative CPU 481.34 sec
>
> 2014-10-01 07:28:40,332 Stage-1 map = 59%, reduce = 8%, Cumulative CPU 504.81 sec
>
> 2014-10-01 07:29:40,957 Stage-1 map = 59%, reduce = 8%, Cumulative CPU 510.71 sec
>
> 2014-10-01 07:30:41,638 Stage-1 map = 59%, reduce = 8%, Cumulative CPU 513.74 sec
>
> 2014-10-01 07:31:00,419 Stage-1 map = 60%, reduce = 8%, Cumulative CPU 514.6 sec
>
> 2014-10-01 07:32:01,081 Stage-1 map = 60%, reduce = 8%, Cumulative CPU 518.18 sec
>
> 2014-10-01 07:32:50,724 Stage-1 map = 62%, reduce = 8%, Cumulative CPU 520.43 sec
>
> 2014-10-01 07:33:00,648 Stage-1 map = 60%, reduce = 8%, Cumulative CPU 521.71 sec
>
> 2014-10-01 07:33:14,974 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 459.27 sec
>
> MapReduce Total cumulative CPU time: 7 minutes 39 seconds 270 msec
>
> Ended Job = job_1411750973876_0099 with errors
>
> FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
>
> MapReduce Jobs Launched:
>
> Job 0: Map: 42 Reduce: 12 Cumulative CPU: 459.27 sec HDFS Read: 1989875843 HDFS Write: 0 FAIL
>
> Total MapReduce CPU Time Spent: 7 minutes 39 seconds 270 msec
>
>
>
> Any idea?
>
> Tks,
> Thiago Henrique S. Bento
> B2W Digital / B2W Labs | BIT SP
> thiago.bento@b2wdigital.com
> Tel.: +55 (11) 3454-3024
> .............................................................................................................................................................
> B2W DIGITAL #Americanas.com #Shoptime.com #Submarino.com #SouBarato.com
>
>
> Esta mensagem pode conter informações confidenciais e somente o indivíduo ou entidade a quem foi destinada pode utilizá-la. A transmissão incorreta da mensagem não acarreta a perda de sua confidencialidade. Caso esta mensagem tenha sido recebida por engano, solicitamos que o fato seja comunicado ao remetente e que a mensagem seja eliminada de seu sistema imediatamente. É vedado a qualquer pessoa que não seja o destinatário usar, revelar, distribuir ou copiar qualquer parte desta mensagem. Ambiente de comunicação sujeito a monitoramento.
>
> This message may include confidential information and only the intended addresses have the right to use it as is, or any part of it. A wrong transmission does not break its confidentiality. If you've received it because of a mistake or erroneous transmission, please notify the sender and delete it from your system immediately. This communication environment is controlled and monitored.
>
> B2W Digital
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.