You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hawq.apache.org by Leon Zhang <le...@gmail.com> on 2015/11/27 08:23:59 UTC
Problem about Hawq on YARN
Hi, HAWQ Developers:
Thanks for the friendly hawq community, they help me to setup the
hawq-2.0 on YARN. After I execute the TPC-DS queries, I encounter some yarn
error for part of queries.
For instance, the query1:
-- start query 12 in stream 0 using template query1.tpl and seed 1711160884
with customer_total_return as
(select sr_customer_sk as ctr_customer_sk
,sr_store_sk as ctr_store_sk
,sum(SR_REVERSED_CHARGE) as ctr_total_return
from store_returns
,date_dim
where sr_returned_date_sk = d_date_sk
and d_year =1998
group by sr_customer_sk
,sr_store_sk)
select c_customer_id
from customer_total_return ctr1
,store
,customer
where ctr1.ctr_total_return > (select avg(ctr_total_return)*1.2
from customer_total_return ctr2
where ctr1.ctr_store_sk = ctr2.ctr_store_sk)
and s_store_sk = ctr1.ctr_store_sk
and s_state = 'TN'
and ctr1.ctr_customer_sk = c_customer_sk
order by c_customer_id
limit 100;
psql:/mnt/xiaolin/query/scale_1/query_0.sql:502: WARNING: Sync RPC
framework (inet) finds exception raised.
psql:/mnt/xiaolin/query/scale_1/query_0.sql:502: ERROR: failed to return
resource to HAWQ resource manager because of failing to receive content.
(pquery.c:983)
And many other errors like this:
$ grep psql:/mnt/xiaolin/query/scale_1/query_0.sql
hawq_yarn/log_1/log_1.log
psql:/mnt/xiaolin/query/scale_1/query_0.sql:502: WARNING: Sync RPC
framework (inet) finds exception raised.
psql:/mnt/xiaolin/query/scale_1/query_0.sql:502: ERROR: failed to return
resource to HAWQ resource manager because of failing to receive content.
(pquery.c:983)
psql:/mnt/xiaolin/query/scale_1/query_0.sql:1103: WARNING: Sync RPC
framework (inet) finds exception raised.
psql:/mnt/xiaolin/query/scale_1/query_0.sql:1103: ERROR: failed to return
resource to HAWQ resource manager because of failing to receive content.
(pquery.c:983)
psql:/mnt/xiaolin/query/scale_1/query_0.sql:1132: ERROR: failed to acquire
resource because of too many unavailable segments. (pquery.c:798)
psql:/mnt/xiaolin/query/scale_1/query_0.sql:1997: WARNING: Sync RPC
framework (inet) finds exception raised.
psql:/mnt/xiaolin/query/scale_1/query_0.sql:1997: ERROR: failed to return
resource to HAWQ resource manager because of failing to receive content.
(pquery.c:983)
psql:/mnt/xiaolin/query/scale_1/query_0.sql:2150: WARNING: Sync RPC
framework (inet) finds exception raised.
psql:/mnt/xiaolin/query/scale_1/query_0.sql:2150: ERROR: failed to return
resource to HAWQ resource manager because of failing to receive content.
(pquery.c:983)
psql:/mnt/xiaolin/query/scale_1/query_0.sql:2455: ERROR: insufficient
memory reserved for statement (execHHashagg.c:732) (seg12 dserver2:40404
pid=331172) (dispatcher.c:1701)
psql:/mnt/xiaolin/query/scale_1/query_0.sql:2543: WARNING: Sync RPC
framework (inet) finds exception raised.
psql:/mnt/xiaolin/query/scale_1/query_0.sql:2543: ERROR: failed to return
resource to HAWQ resource manager because of failing to receive content.
(pquery.c:983)
psql:/mnt/xiaolin/query/scale_1/query_0.sql:2639: WARNING: Sync RPC
framework (inet) finds exception raised.
psql:/mnt/xiaolin/query/scale_1/query_0.sql:2639: ERROR: failed to return
resource to HAWQ resource manager because of failing to receive content.
(pquery.c:983)
psql:/mnt/xiaolin/query/scale_1/query_0.sql:2661: ERROR: failed to acquire
resource because of too many unavailable segments. (pquery.c:798)
psql:/mnt/xiaolin/query/scale_1/query_0.sql:2720: ERROR: failed to acquire
resource because of too many unavailable segments. (pquery.c:798)
psql:/mnt/xiaolin/query/scale_1/query_0.sql:2742: ERROR: failed to acquire
resource because of too many unavailable segments. (pquery.c:798)
psql:/mnt/xiaolin/query/scale_1/query_0.sql:2794: ERROR: failed to acquire
resource because of too many unavailable segments. (pquery.c:798)
psql:/mnt/xiaolin/query/scale_1/query_0.sql:2902: ERROR: failed to acquire
resource because of too many unavailable segments. (pquery.c:798)
psql:/mnt/xiaolin/query/scale_1/query_0.sql:2935: ERROR: failed to acquire
resource because of too many unavailable segments. (pquery.c:798)
psql:/mnt/xiaolin/query/scale_1/query_0.sql:3182: WARNING: Sync RPC
framework (inet) finds exception raised.
psql:/mnt/xiaolin/query/scale_1/query_0.sql:3182: ERROR: failed to return
resource to HAWQ resource manager because of failing to receive content.
(pquery.c:983)
psql:/mnt/xiaolin/query/scale_1/query_0.sql:3286: ERROR: failed to acquire
resource because of too many unavailable segments. (pquery.c:798)
psql:/mnt/xiaolin/query/scale_1/query_0.sql:3380: ERROR: failed to acquire
resource because of too many unavailable segments. (pquery.c:798)
psql:/mnt/xiaolin/query/scale_1/query_0.sql:3428: ERROR: failed to acquire
resource because of too many unavailable segments. (pquery.c:798)
psql:/mnt/xiaolin/query/scale_1/query_0.sql:3457: ERROR: failed to acquire
resource because of too many unavailable segments. (pquery.c:798)
psql:/mnt/xiaolin/query/scale_1/query_0.sql:3497: ERROR: failed to acquire
resource because of too many unavailable segments. (pquery.c:798)
psql:/mnt/xiaolin/query/scale_1/query_0.sql:3528: ERROR: failed to acquire
resource because of too many unavailable segments. (pquery.c:798)
psql:/mnt/xiaolin/query/scale_1/query_0.sql:3595: ERROR: failed to acquire
resource because of too many unavailable segments. (pquery.c:798)
psql:/mnt/xiaolin/query/scale_1/query_0.sql:3626: ERROR: failed to acquire
resource because of too many unavailable segments. (pquery.c:798)
psql:/mnt/xiaolin/query/scale_1/query_0.sql:3687: ERROR: failed to acquire
resource because of too many unavailable segments. (pquery.c:798)
psql:/mnt/xiaolin/query/scale_1/query_0.sql:3710: ERROR: failed to acquire
resource because of too many unavailable segments. (pquery.c:798)
psql:/mnt/xiaolin/query/scale_1/query_0.sql:3817: ERROR: failed to acquire
resource because of too many unavailable segments. (pquery.c:798)
psql:/mnt/xiaolin/query/scale_1/query_0.sql:3845: ERROR: failed to acquire
resource because of too many unavailable segments. (pquery.c:798)
psql:/mnt/xiaolin/query/scale_1/query_0.sql:3866: ERROR: failed to acquire
resource because of too many unavailable segments. (pquery.c:798)
psql:/mnt/xiaolin/query/scale_1/query_0.sql:3922: ERROR: failed to acquire
resource because of too many unavailable segments. (pquery.c:798)
psql:/mnt/xiaolin/query/scale_1/query_0.sql:3936: ERROR: failed to acquire
resource because of too many unavailable segments. (pquery.c:798)
psql:/mnt/xiaolin/query/scale_1/query_0.sql:4004: ERROR: failed to acquire
resource because of too many unavailable segments. (pquery.c:798)
psql:/mnt/xiaolin/query/scale_1/query_0.sql:4064: ERROR: failed to acquire
resource because of too many unavailable segments. (pquery.c:798)
psql:/mnt/xiaolin/query/scale_1/query_0.sql:4085: ERROR: failed to acquire
resource because of too many unavailable segments. (pquery.c:798)
psql:/mnt/xiaolin/query/scale_1/query_0.sql:4113: ERROR: failed to acquire
resource because of too many unavailable segments. (pquery.c:798)
psql:/mnt/xiaolin/query/scale_1/query_0.sql:4489: ERROR: insufficient
memory reserved for statement
psql:/mnt/xiaolin/query/scale_1/query_0.sql:4633: WARNING: Sync RPC
framework (inet) finds exception raised.
psql:/mnt/xiaolin/query/scale_1/query_0.sql:4633: ERROR: failed to return
resource to HAWQ resource manager because of failing to receive content.
(pquery.c:983)
psql:/mnt/xiaolin/query/scale_1/query_0.sql:4677: ERROR: failed to acquire
resource because of too many unavailable segments. (pquery.c:798)
psql:/mnt/xiaolin/query/scale_1/query_0.sql:4805: ERROR: failed to acquire
resource because of too many unavailable segments. (pquery.c:798)
I read through the log of YARN, and I can not find any error about
this. I am not sure if any parameters can be tuned to fix issue? Or if it
is a bug?
Any help will be appreciated, thanks in advance.
Re: Problem about Hawq on YARN
Posted by Wen Lin <wl...@pivotal.io>.
Hi, Leon,
The log you attached maybe known bug in hawq dev team or an unknown one.
It's hard to conclude based on it.
Would you please send a full log? Then we can know what happened before the
error message was printed.
Thank you!
On Fri, Nov 27, 2015 at 3:23 PM, Leon Zhang <le...@gmail.com> wrote:
> Hi, HAWQ Developers:
>
> Thanks for the friendly hawq community, they help me to setup the
> hawq-2.0 on YARN. After I execute the TPC-DS queries, I encounter some yarn
> error for part of queries.
>
> For instance, the query1:
> -- start query 12 in stream 0 using template query1.tpl and seed 1711160884
> with customer_total_return as
> (select sr_customer_sk as ctr_customer_sk
> ,sr_store_sk as ctr_store_sk
> ,sum(SR_REVERSED_CHARGE) as ctr_total_return
> from store_returns
> ,date_dim
> where sr_returned_date_sk = d_date_sk
> and d_year =1998
> group by sr_customer_sk
> ,sr_store_sk)
> select c_customer_id
> from customer_total_return ctr1
> ,store
> ,customer
> where ctr1.ctr_total_return > (select avg(ctr_total_return)*1.2
> from customer_total_return ctr2
> where ctr1.ctr_store_sk = ctr2.ctr_store_sk)
> and s_store_sk = ctr1.ctr_store_sk
> and s_state = 'TN'
> and ctr1.ctr_customer_sk = c_customer_sk
> order by c_customer_id
> limit 100;
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:502: WARNING: Sync RPC
> framework (inet) finds exception raised.
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:502: ERROR: failed to return
> resource to HAWQ resource manager because of failing to receive content.
> (pquery.c:983)
>
> And many other errors like this:
> $ grep psql:/mnt/xiaolin/query/scale_1/query_0.sql
> hawq_yarn/log_1/log_1.log
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:502: WARNING: Sync RPC
> framework (inet) finds exception raised.
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:502: ERROR: failed to return
> resource to HAWQ resource manager because of failing to receive content.
> (pquery.c:983)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:1103: WARNING: Sync RPC
> framework (inet) finds exception raised.
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:1103: ERROR: failed to return
> resource to HAWQ resource manager because of failing to receive content.
> (pquery.c:983)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:1132: ERROR: failed to acquire
> resource because of too many unavailable segments. (pquery.c:798)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:1997: WARNING: Sync RPC
> framework (inet) finds exception raised.
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:1997: ERROR: failed to return
> resource to HAWQ resource manager because of failing to receive content.
> (pquery.c:983)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:2150: WARNING: Sync RPC
> framework (inet) finds exception raised.
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:2150: ERROR: failed to return
> resource to HAWQ resource manager because of failing to receive content.
> (pquery.c:983)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:2455: ERROR: insufficient
> memory reserved for statement (execHHashagg.c:732) (seg12 dserver2:40404
> pid=331172) (dispatcher.c:1701)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:2543: WARNING: Sync RPC
> framework (inet) finds exception raised.
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:2543: ERROR: failed to return
> resource to HAWQ resource manager because of failing to receive content.
> (pquery.c:983)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:2639: WARNING: Sync RPC
> framework (inet) finds exception raised.
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:2639: ERROR: failed to return
> resource to HAWQ resource manager because of failing to receive content.
> (pquery.c:983)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:2661: ERROR: failed to acquire
> resource because of too many unavailable segments. (pquery.c:798)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:2720: ERROR: failed to acquire
> resource because of too many unavailable segments. (pquery.c:798)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:2742: ERROR: failed to acquire
> resource because of too many unavailable segments. (pquery.c:798)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:2794: ERROR: failed to acquire
> resource because of too many unavailable segments. (pquery.c:798)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:2902: ERROR: failed to acquire
> resource because of too many unavailable segments. (pquery.c:798)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:2935: ERROR: failed to acquire
> resource because of too many unavailable segments. (pquery.c:798)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:3182: WARNING: Sync RPC
> framework (inet) finds exception raised.
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:3182: ERROR: failed to return
> resource to HAWQ resource manager because of failing to receive content.
> (pquery.c:983)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:3286: ERROR: failed to acquire
> resource because of too many unavailable segments. (pquery.c:798)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:3380: ERROR: failed to acquire
> resource because of too many unavailable segments. (pquery.c:798)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:3428: ERROR: failed to acquire
> resource because of too many unavailable segments. (pquery.c:798)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:3457: ERROR: failed to acquire
> resource because of too many unavailable segments. (pquery.c:798)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:3497: ERROR: failed to acquire
> resource because of too many unavailable segments. (pquery.c:798)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:3528: ERROR: failed to acquire
> resource because of too many unavailable segments. (pquery.c:798)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:3595: ERROR: failed to acquire
> resource because of too many unavailable segments. (pquery.c:798)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:3626: ERROR: failed to acquire
> resource because of too many unavailable segments. (pquery.c:798)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:3687: ERROR: failed to acquire
> resource because of too many unavailable segments. (pquery.c:798)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:3710: ERROR: failed to acquire
> resource because of too many unavailable segments. (pquery.c:798)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:3817: ERROR: failed to acquire
> resource because of too many unavailable segments. (pquery.c:798)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:3845: ERROR: failed to acquire
> resource because of too many unavailable segments. (pquery.c:798)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:3866: ERROR: failed to acquire
> resource because of too many unavailable segments. (pquery.c:798)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:3922: ERROR: failed to acquire
> resource because of too many unavailable segments. (pquery.c:798)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:3936: ERROR: failed to acquire
> resource because of too many unavailable segments. (pquery.c:798)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:4004: ERROR: failed to acquire
> resource because of too many unavailable segments. (pquery.c:798)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:4064: ERROR: failed to acquire
> resource because of too many unavailable segments. (pquery.c:798)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:4085: ERROR: failed to acquire
> resource because of too many unavailable segments. (pquery.c:798)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:4113: ERROR: failed to acquire
> resource because of too many unavailable segments. (pquery.c:798)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:4489: ERROR: insufficient
> memory reserved for statement
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:4633: WARNING: Sync RPC
> framework (inet) finds exception raised.
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:4633: ERROR: failed to return
> resource to HAWQ resource manager because of failing to receive content.
> (pquery.c:983)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:4677: ERROR: failed to acquire
> resource because of too many unavailable segments. (pquery.c:798)
> psql:/mnt/xiaolin/query/scale_1/query_0.sql:4805: ERROR: failed to acquire
> resource because of too many unavailable segments. (pquery.c:798)
>
> I read through the log of YARN, and I can not find any error about
> this. I am not sure if any parameters can be tuned to fix issue? Or if it
> is a bug?
>
> Any help will be appreciated, thanks in advance.
>