You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by "Arthur.hk.chan@gmail.com" <ar...@gmail.com> on 2014/10/12 00:49:52 UTC
Smoke Test after 1 days 7 hours 5 minutes 19 seconds 70 msec, Failed with Error: GC overhead limit exceeded
Hi,
My Hive version is 0.13.1, I tried a smoke test, after 1 days 7 hours 5 minutes 19 seconds 70 msec, the job failed with error: Error: GC overhead limit exceeded
LOG:
2014-10-12 06:16:07,288 Stage-6 map = 100%, reduce = 50%, Cumulative CPU 425.35 sec
2014-10-12 06:16:12,431 Stage-6 map = 100%, reduce = 67%, Cumulative CPU 433.01 sec
2014-10-12 06:16:15,515 Stage-6 map = 100%, reduce = 100%, Cumulative CPU 447.59 sec
…...
Hadoop job information for Stage-19: number of mappers: 3; number of reducers: 0
2014-10-12 06:16:30,643 Stage-19 map = 0%, reduce = 0%
2014-10-12 06:16:55,494 Stage-19 map = 33%, reduce = 0%, Cumulative CPU 153.83 sec
2014-10-12 06:16:56,520 Stage-19 map = 0%, reduce = 0%
2014-10-12 06:17:57,037 Stage-19 map = 0%, reduce = 0%
2014-10-12 06:18:27,720 Stage-19 map = 100%, reduce = 0%
MapReduce Total cumulative CPU time: 2 minutes 33 seconds 830 msec
Ended Job = job_1413024651684_0033 with errors
Error during job, obtaining debugging information...
Examining task ID: task_1413024651684_0033_m_000001 (and more) from job job_1413024651684_0033
Task with the most failures(4):
-----
Task ID:
task_1413024651684_0033_m_000002
URL:
http://m1:8088/taskdetails.jsp?jobid=job_1413024651684_0033&tipid=task_1413024651684_0033_m_000002
-----
Diagnostic Messages for this Task:
Error: GC overhead limit exceeded
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:
Job 0: Map: 5 Reduce: 1 Cumulative CPU: 10705.42 sec HDFS Read: 829911667 HDFS Write: 693918010684 SUCCESS
Job 1: Map: 2684 Reduce: 721 Cumulative CPU: 100612.23 sec HDFS Read: 720031197955 HDFS Write: 56301916 SUCCESS
Job 2: Map: 25 Reduce: 6 Cumulative CPU: 447.59 sec HDFS Read: 5785850462 HDFS Write: 22244710 SUCCESS
Job 3: Map: 3 Cumulative CPU: 153.83 sec HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 1 days 7 hours 5 minutes 19 seconds 70 msec
my smoke test SQL :
SELECT O_YEAR,
SUM(CASE
WHEN NATION = 'BRAZIL' THEN VOLUME
ELSE 0
END) / SUM(VOLUME) AS MKT_SHARE
FROM (SELECT YEAR(cast(O_ORDERDATE as date)) AS O_YEAR,
L_EXTENDEDPRICE * ( 1 - L_DISCOUNT ) AS VOLUME,
N2.N_NAME AS NATION
FROM PART,
SUPPLIER,
LINEITEM,
ORDERS,
CUSTOMER,
NATION N1,
NATION N2,
REGION
WHERE P_PARTKEY = L_PARTKEY
AND S_SUPPKEY = L_SUPPKEY
AND L_ORDERKEY = O_ORDERKEY
AND O_CUSTKEY = C_CUSTKEY
AND C_NATIONKEY = N1.N_NATIONKEY
AND N1.N_REGIONKEY = R_REGIONKEY
AND R_NAME = 'AMERICA'
AND S_NATIONKEY = N2.N_NATIONKEY
AND cast(O_ORDERDATE as date) >= cast('1995-01-01' as date)
AND cast(O_ORDERDATE as date) <= cast('1996-12-31' as date)
AND P_TYPE = 'ECONOMY ANODIZED STEEL') AS ALL_NATIONS
GROUP BY O_YEAR
ORDER BY O_YEAR;
Please help.
Regards
Arthur
Re: Smoke Test after 1 days 7 hours 5 minutes 19 seconds 70 msec,
Failed with Error: GC overhead limit exceeded
Posted by Suhas Gogate <vg...@pivotal.io>.
Arthur, just curious.. what tuning you did? It seems, your last failing
map-only job (w/ 3 map tasks) either need to have more mappers or more heap
allocated to each of them.
--Suhas
On Mon, Oct 13, 2014 at 3:12 AM, Arthur.hk.chan@gmail.com <
arthur.hk.chan@gmail.com> wrote:
> Hi,
>
> I have managed to resolve the issue by turning the SQL.
>
> Regards
> Arthur
>
> On 12 Oct, 2014, at 6:49 am, Arthur.hk.chan@gmail.com <
> arthur.hk.chan@gmail.com> wrote:
>
> Hi,
>
> My Hive version is 0.13.1, I tried a smoke test, after 1 days 7 hours 5
> minutes 19 seconds 70 msec, the job failed with error: Error: GC overhead
> limit exceeded
>
>
> LOG:
> 2014-10-12 06:16:07,288 Stage-6 map = 100%, reduce = 50%, Cumulative CPU
> 425.35 sec
> 2014-10-12 06:16:12,431 Stage-6 map = 100%, reduce = 67%, Cumulative CPU
> 433.01 sec
> 2014-10-12 06:16:15,515 Stage-6 map = 100%, reduce = 100%, Cumulative CPU
> 447.59 sec
> …...
> Hadoop job information for Stage-19: number of mappers: 3; number of
> reducers: 0
> 2014-10-12 06:16:30,643 Stage-19 map = 0%, reduce = 0%
> 2014-10-12 06:16:55,494 Stage-19 map = 33%, reduce = 0%, Cumulative CPU
> 153.83 sec
> 2014-10-12 06:16:56,520 Stage-19 map = 0%, reduce = 0%
> 2014-10-12 06:17:57,037 Stage-19 map = 0%, reduce = 0%
> 2014-10-12 06:18:27,720 Stage-19 map = 100%, reduce = 0%
> MapReduce Total cumulative CPU time: 2 minutes 33 seconds 830 msec
> Ended Job = job_1413024651684_0033 with errors
> Error during job, obtaining debugging information...
> Examining task ID: task_1413024651684_0033_m_000001 (and more) from job
> job_1413024651684_0033
>
> Task with the most failures(4):
> -----
> Task ID:
> task_1413024651684_0033_m_000002
>
> URL:
>
> http://m1:8088/taskdetails.jsp?jobid=job_1413024651684_0033&tipid=task_1413024651684_0033_m_000002
> -----
> Diagnostic Messages for this Task:
> Error: GC overhead limit exceeded
>
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> MapReduce Jobs Launched:
> Job 0: Map: 5 Reduce: 1 Cumulative CPU: 10705.42 sec HDFS Read:
> 829911667 HDFS Write: 693918010684 SUCCESS
> Job 1: Map: 2684 Reduce: 721 Cumulative CPU: 100612.23 sec HDFS Read:
> 720031197955 HDFS Write: 56301916 SUCCESS
> Job 2: Map: 25 Reduce: 6 Cumulative CPU: 447.59 sec HDFS Read:
> 5785850462 HDFS Write: 22244710 SUCCESS
> Job 3: Map: 3 Cumulative CPU: 153.83 sec HDFS Read: 0 HDFS Write: 0
> FAIL
> Total MapReduce CPU Time Spent: 1 days 7 hours 5 minutes 19 seconds 70 msec
>
> my smoke test SQL :
> SELECT O_YEAR,
> SUM(CASE
> WHEN NATION = 'BRAZIL' THEN VOLUME
> ELSE 0
> END) / SUM(VOLUME) AS MKT_SHARE
> FROM (SELECT YEAR(cast(O_ORDERDATE as date)) AS O_YEAR,
> L_EXTENDEDPRICE * ( 1 - L_DISCOUNT ) AS VOLUME,
> N2.N_NAME AS NATION
> FROM PART,
> SUPPLIER,
> LINEITEM,
> ORDERS,
> CUSTOMER,
> NATION N1,
> NATION N2,
> REGION
> WHERE P_PARTKEY = L_PARTKEY
> AND S_SUPPKEY = L_SUPPKEY
> AND L_ORDERKEY = O_ORDERKEY
> AND O_CUSTKEY = C_CUSTKEY
> AND C_NATIONKEY = N1.N_NATIONKEY
> AND N1.N_REGIONKEY = R_REGIONKEY
> AND R_NAME = 'AMERICA'
> AND S_NATIONKEY = N2.N_NATIONKEY
> AND cast(O_ORDERDATE as date) >= cast('1995-01-01' as
> date)
> AND cast(O_ORDERDATE as date) <= cast('1996-12-31' as
> date)
> AND P_TYPE = 'ECONOMY ANODIZED STEEL') AS ALL_NATIONS
> GROUP BY O_YEAR
> ORDER BY O_YEAR;
>
>
> Please help.
> Regards
> Arthur
>
>
>
>
>
Re: Smoke Test after 1 days 7 hours 5 minutes 19 seconds 70 msec, Failed with Error: GC overhead limit exceeded
Posted by "Arthur.hk.chan@gmail.com" <ar...@gmail.com>.
Hi,
I have managed to resolve the issue by turning the SQL.
Regards
Arthur
On 12 Oct, 2014, at 6:49 am, Arthur.hk.chan@gmail.com <ar...@gmail.com> wrote:
> Hi,
>
> My Hive version is 0.13.1, I tried a smoke test, after 1 days 7 hours 5 minutes 19 seconds 70 msec, the job failed with error: Error: GC overhead limit exceeded
>
>
> LOG:
> 2014-10-12 06:16:07,288 Stage-6 map = 100%, reduce = 50%, Cumulative CPU 425.35 sec
> 2014-10-12 06:16:12,431 Stage-6 map = 100%, reduce = 67%, Cumulative CPU 433.01 sec
> 2014-10-12 06:16:15,515 Stage-6 map = 100%, reduce = 100%, Cumulative CPU 447.59 sec
> …...
> Hadoop job information for Stage-19: number of mappers: 3; number of reducers: 0
> 2014-10-12 06:16:30,643 Stage-19 map = 0%, reduce = 0%
> 2014-10-12 06:16:55,494 Stage-19 map = 33%, reduce = 0%, Cumulative CPU 153.83 sec
> 2014-10-12 06:16:56,520 Stage-19 map = 0%, reduce = 0%
> 2014-10-12 06:17:57,037 Stage-19 map = 0%, reduce = 0%
> 2014-10-12 06:18:27,720 Stage-19 map = 100%, reduce = 0%
> MapReduce Total cumulative CPU time: 2 minutes 33 seconds 830 msec
> Ended Job = job_1413024651684_0033 with errors
> Error during job, obtaining debugging information...
> Examining task ID: task_1413024651684_0033_m_000001 (and more) from job job_1413024651684_0033
>
> Task with the most failures(4):
> -----
> Task ID:
> task_1413024651684_0033_m_000002
>
> URL:
> http://m1:8088/taskdetails.jsp?jobid=job_1413024651684_0033&tipid=task_1413024651684_0033_m_000002
> -----
> Diagnostic Messages for this Task:
> Error: GC overhead limit exceeded
>
> FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> MapReduce Jobs Launched:
> Job 0: Map: 5 Reduce: 1 Cumulative CPU: 10705.42 sec HDFS Read: 829911667 HDFS Write: 693918010684 SUCCESS
> Job 1: Map: 2684 Reduce: 721 Cumulative CPU: 100612.23 sec HDFS Read: 720031197955 HDFS Write: 56301916 SUCCESS
> Job 2: Map: 25 Reduce: 6 Cumulative CPU: 447.59 sec HDFS Read: 5785850462 HDFS Write: 22244710 SUCCESS
> Job 3: Map: 3 Cumulative CPU: 153.83 sec HDFS Read: 0 HDFS Write: 0 FAIL
> Total MapReduce CPU Time Spent: 1 days 7 hours 5 minutes 19 seconds 70 msec
>
> my smoke test SQL :
> SELECT O_YEAR,
> SUM(CASE
> WHEN NATION = 'BRAZIL' THEN VOLUME
> ELSE 0
> END) / SUM(VOLUME) AS MKT_SHARE
> FROM (SELECT YEAR(cast(O_ORDERDATE as date)) AS O_YEAR,
> L_EXTENDEDPRICE * ( 1 - L_DISCOUNT ) AS VOLUME,
> N2.N_NAME AS NATION
> FROM PART,
> SUPPLIER,
> LINEITEM,
> ORDERS,
> CUSTOMER,
> NATION N1,
> NATION N2,
> REGION
> WHERE P_PARTKEY = L_PARTKEY
> AND S_SUPPKEY = L_SUPPKEY
> AND L_ORDERKEY = O_ORDERKEY
> AND O_CUSTKEY = C_CUSTKEY
> AND C_NATIONKEY = N1.N_NATIONKEY
> AND N1.N_REGIONKEY = R_REGIONKEY
> AND R_NAME = 'AMERICA'
> AND S_NATIONKEY = N2.N_NATIONKEY
> AND cast(O_ORDERDATE as date) >= cast('1995-01-01' as date)
> AND cast(O_ORDERDATE as date) <= cast('1996-12-31' as date)
> AND P_TYPE = 'ECONOMY ANODIZED STEEL') AS ALL_NATIONS
> GROUP BY O_YEAR
> ORDER BY O_YEAR;
>
>
> Please help.
> Regards
> Arthur
>
>
>