You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hawq.apache.org by Shailesh Doshi <sd...@pivotal.io> on 2016/05/19 15:31:16 UTC
Re: user Digest 17 May 2016 02:59:27 -0000 Issue 67
test
On Mon, May 16, 2016 at 10:59 PM, <
user-digest-help@hawq.incubator.apache.org> wrote:
>
> user Digest 17 May 2016 02:59:27 -0000 Issue 67
>
> Topics (messages 212 through 215)
>
> Re: no partition for partitioning key make gpload failed.
> 212 by: Lei Chang
> 213 by: Gagan Brahmi
> 214 by: yin.zhb.163.com
>
> Re: HAWQ YARN RPC Errors
> 215 by: Wen Lin
>
> Administrivia:
>
> ---------------------------------------------------------------------
> To post to the list, e-mail: user@hawq.incubator.apache.org
> To unsubscribe, e-mail: user-digest-unsubscribe@hawq.incubator.apache.org
> For additional commands, e-mail:
> user-digest-help@hawq.incubator.apache.org
>
> ----------------------------------------------------------------------
>
>
>
> ---------- Forwarded message ----------
> From: Lei Chang <le...@apache.org>
> To: user <us...@hawq.incubator.apache.org>
> Cc: dev <de...@hawq.incubator.apache.org>
> Date: Sun, 15 May 2016 09:16:11 +0800
> Subject: Re: no partition for partitioning key make gpload failed.
>
> Hi, i think it is because your data inserted has a partition key not in
> the range. for example, i can reproduce this on my laptop by the follow
> 'insert'. You can change you partition table definition accordingly.
>
> postgres=# insert into stocks
> values(1,1,'1','1',1,1,'1','20170101'::date);
>
>
> ERROR: no partition for partitioning key (seg1 localhost:40000 pid=35447)
> Thanks
> Lei
>
>
> On Sat, May 14, 2016 at 11:34 PM, yin.zhb@163.com <yi...@163.com> wrote:
>
>> Hi,all:
>> I am using hawq 1.1.0,When I using gpload there is a problem:
>>
>> [gpadmin@stars71 ~]$ psql -d postgres
>> postgres=# create table stocks
>> postgres-# (
>> postgres(# storeno bigint,
>> postgres(# itemno bigint,
>> postgres(# itemname text,
>> postgres(# spec text,
>> postgres(# itemseq bigint,
>> postgres(# stocks bigint,
>> postgres(# unit text,
>> postgres(# ymd date
>> postgres(# )
>> postgres-# distributed by (itemno)
>> postgres-# partition by range(ymd)
>> postgres-# (
>>
>> postgres(# partition p2015 start ('20150101'::date) end ('20160101'::date),
>> postgres(# partition p2016 start ('20160101'::date) end ('20170101'::date)
>> postgres(# );
>>
>> NOTICE: CREATE TABLE will create partition "stocks_1_prt_p2015" for table "stocks"
>>
>> NOTICE: CREATE TABLE will create partition "stocks_1_prt_p2016" for table "stocks"
>> CREATE TABLE
>> postgres=# \q
>>
>> [gpadmin@stars71
>> ctl]$ gpfdist -d /data/data/ -p 8081 -l /home/gpadmin/log &
>> [gpadmin@stars71 ctl]$ cat stocks.ctl
>> ---
>> VERSION: 1.0.0.1
>> DATABASE: postgres
>> USER: gpadmin
>> HOST: stars71
>> PORT: 5432
>> GPLOAD:
>> INPUT:
>> - SOURCE:
>> LOCAL_HOSTNAME:
>> - stars71
>> PORT: 8081
>> FILE:
>> - /data/data/*
>> - COLUMNS:
>> - storeno: bigint
>> - itemno: bigint
>> - itemname: text
>> - spec: text
>> - itemseq: bigint
>> - stocks: bigint
>> - unit: text
>> - ymd: date
>> - FORMAT: text
>> - DELIMITER: E'\t'
>> - ERROR_LIMIT: 25
>> - ERROR_TABLE: stocks_err
>> OUTPUT:
>> - TABLE: stocks
>> - MODE: INSERT
>> [gpadmin@stars71 ctl]$ gpload -f stocks.ctl
>> 2016-05-14 23:24:19|INFO|gpload session started 2016-05-14 23:24:19
>> 2016-05-14 23:24:19|INFO|setting schema 'public' for table 'stocks'
>>
>> 2016-05-14 23:24:20|INFO|started gpfdist -p 8081 -P 8082 -f "/data/data/*" -t 30
>>
>> 2016-05-14 23:24:20|WARN|A gpload control file processing warning occurred. A delimiter must have a length of one. Special characters must be quoted. gpload will assume this is a sql escape character sequence.
>> 2016-05-14 23:24:20|ERROR|ERROR: *no partition for partitioning key*
>> (seg5 localhost:40000 pid=14591)
>>
>> encountered while running INSERT INTO public."stocks" ("storeno","itemno","itemname","spec","itemseq","stocks","unit","ymd") SELECT "storeno","itemno","itemname","spec","itemseq","stocks","unit","ymd" FROM ext_gpload20160514_232419_14569
>> 2016-05-14 23:24:20|INFO|rows Inserted = 0
>> 2016-05-14 23:24:20|INFO|rows Updated = 0
>> 2016-05-14 23:24:20|INFO|data formatting errors = 0
>> 2016-05-14 23:24:20|INFO|gpload failed
>>
>> It was failed,even if there is only one line.
>>
>> ------------------------------
>> yin.zhb@163.com
>>
>
>
>
> ---------- Forwarded message ----------
> From: Gagan Brahmi <ga...@gmail.com>
> To: user@hawq.incubator.apache.org
> Cc: dev <de...@hawq.incubator.apache.org>
> Date: Sat, 14 May 2016 20:19:14 -0700
> Subject: Re: no partition for partitioning key make gpload failed.
>
> Looks like you've a date in ymd column which falls outside the defined
> range for the two partitions.
>
> You can define a partition which can hold all the data which falls before 20150101
> with something like below:
>
> ...partition pre2015 end ('20150101'::date)...
>
>
> Regards,
>
> Gagan Brahmi
> On May 14, 2016 08:34, "yin.zhb@163.com" <yi...@163.com> wrote:
>
>> Hi,all:
>> I am using hawq 1.1.0,When I using gpload there is a problem:
>>
>> [gpadmin@stars71 ~]$ psql -d postgres
>> postgres=# create table stocks
>> postgres-# (
>> postgres(# storeno bigint,
>> postgres(# itemno bigint,
>> postgres(# itemname text,
>> postgres(# spec text,
>> postgres(# itemseq bigint,
>> postgres(# stocks bigint,
>> postgres(# unit text,
>> postgres(# ymd date
>> postgres(# )
>> postgres-# distributed by (itemno)
>> postgres-# partition by range(ymd)
>> postgres-# (
>>
>> postgres(# partition p2015 start ('20150101'::date) end ('20160101'::date),
>> postgres(# partition p2016 start ('20160101'::date) end ('20170101'::date)
>> postgres(# );
>>
>> NOTICE: CREATE TABLE will create partition "stocks_1_prt_p2015" for table "stocks"
>>
>> NOTICE: CREATE TABLE will create partition "stocks_1_prt_p2016" for table "stocks"
>> CREATE TABLE
>> postgres=# \q
>>
>> [gpadmin@stars71
>> ctl]$ gpfdist -d /data/data/ -p 8081 -l /home/gpadmin/log &
>> [gpadmin@stars71 ctl]$ cat stocks.ctl
>> ---
>> VERSION: 1.0.0.1
>> DATABASE: postgres
>> USER: gpadmin
>> HOST: stars71
>> PORT: 5432
>> GPLOAD:
>> INPUT:
>> - SOURCE:
>> LOCAL_HOSTNAME:
>> - stars71
>> PORT: 8081
>> FILE:
>> - /data/data/*
>> - COLUMNS:
>> - storeno: bigint
>> - itemno: bigint
>> - itemname: text
>> - spec: text
>> - itemseq: bigint
>> - stocks: bigint
>> - unit: text
>> - ymd: date
>> - FORMAT: text
>> - DELIMITER: E'\t'
>> - ERROR_LIMIT: 25
>> - ERROR_TABLE: stocks_err
>> OUTPUT:
>> - TABLE: stocks
>> - MODE: INSERT
>> [gpadmin@stars71 ctl]$ gpload -f stocks.ctl
>> 2016-05-14 23:24:19|INFO|gpload session started 2016-05-14 23:24:19
>> 2016-05-14 23:24:19|INFO|setting schema 'public' for table 'stocks'
>>
>> 2016-05-14 23:24:20|INFO|started gpfdist -p 8081 -P 8082 -f "/data/data/*" -t 30
>>
>> 2016-05-14 23:24:20|WARN|A gpload control file processing warning occurred. A delimiter must have a length of one. Special characters must be quoted. gpload will assume this is a sql escape character sequence.
>> 2016-05-14 23:24:20|ERROR|ERROR: *no partition for partitioning key*
>> (seg5 localhost:40000 pid=14591)
>>
>> encountered while running INSERT INTO public."stocks" ("storeno","itemno","itemname","spec","itemseq","stocks","unit","ymd") SELECT "storeno","itemno","itemname","spec","itemseq","stocks","unit","ymd" FROM ext_gpload20160514_232419_14569
>> 2016-05-14 23:24:20|INFO|rows Inserted = 0
>> 2016-05-14 23:24:20|INFO|rows Updated = 0
>> 2016-05-14 23:24:20|INFO|data formatting errors = 0
>> 2016-05-14 23:24:20|INFO|gpload failed
>>
>> It was failed,even if there is only one line.
>>
>> ------------------------------
>> yin.zhb@163.com
>>
>
>
> ---------- Forwarded message ----------
> From: "yin.zhb@163.com" <yi...@163.com>
> To: user <us...@hawq.incubator.apache.org>
> Cc: dev <de...@hawq.incubator.apache.org>
> Date: Sun, 15 May 2016 12:10:32 +0800
> Subject: Re: Re: no partition for partitioning key make gpload failed.
> The point is I set error limits, It should be skip the bad lines,Let the
> others go.
>
> ------------------------------
> yin.zhb@163.com
>
>
> *From:* Gagan Brahmi <ga...@gmail.com>
> *Date:* 2016-05-15 11:19
> *To:* user <us...@hawq.incubator.apache.org>
> *CC:* dev <de...@hawq.incubator.apache.org>
> *Subject:* Re: no partition for partitioning key make gpload failed.
>
> Looks like you've a date in ymd column which falls outside the defined
> range for the two partitions.
>
> You can define a partition which can hold all the data which falls before 20150101
> with something like below:
>
> ...partition pre2015 end ('20150101'::date)...
>
>
> Regards,
>
> Gagan Brahmi
> On May 14, 2016 08:34, "yin.zhb@163.com" <yi...@163.com> wrote:
>
>> Hi,all:
>> I am using hawq 1.1.0,When I using gpload there is a problem:
>>
>> [gpadmin@stars71 ~]$ psql -d postgres
>> postgres=# create table stocks
>> postgres-# (
>> postgres(# storeno bigint,
>> postgres(# itemno bigint,
>> postgres(# itemname text,
>> postgres(# spec text,
>> postgres(# itemseq bigint,
>> postgres(# stocks bigint,
>> postgres(# unit text,
>> postgres(# ymd date
>> postgres(# )
>> postgres-# distributed by (itemno)
>> postgres-# partition by range(ymd)
>> postgres-# (
>>
>> postgres(# partition p2015 start ('20150101'::date) end ('20160101'::date),
>> postgres(# partition p2016 start ('20160101'::date) end ('20170101'::date)
>> postgres(# );
>>
>> NOTICE: CREATE TABLE will create partition "stocks_1_prt_p2015" for table "stocks"
>>
>> NOTICE: CREATE TABLE will create partition "stocks_1_prt_p2016" for table "stocks"
>> CREATE TABLE
>> postgres=# \q
>>
>> [gpadmin@stars71
>> ctl]$ gpfdist -d /data/data/ -p 8081 -l /home/gpadmin/log &
>> [gpadmin@stars71 ctl]$ cat stocks.ctl
>> ---
>> VERSION: 1.0.0.1
>> DATABASE: postgres
>> USER: gpadmin
>> HOST: stars71
>> PORT: 5432
>> GPLOAD:
>> INPUT:
>> - SOURCE:
>> LOCAL_HOSTNAME:
>> - stars71
>> PORT: 8081
>> FILE:
>> - /data/data/*
>> - COLUMNS:
>> - storeno: bigint
>> - itemno: bigint
>> - itemname: text
>> - spec: text
>> - itemseq: bigint
>> - stocks: bigint
>> - unit: text
>> - ymd: date
>> - FORMAT: text
>> - DELIMITER: E'\t'
>> - ERROR_LIMIT: 25
>> - ERROR_TABLE: stocks_err
>> OUTPUT:
>> - TABLE: stocks
>> - MODE: INSERT
>> [gpadmin@stars71 ctl]$ gpload -f stocks.ctl
>> 2016-05-14 23:24:19|INFO|gpload session started 2016-05-14 23:24:19
>> 2016-05-14 23:24:19|INFO|setting schema 'public' for table 'stocks'
>>
>> 2016-05-14 23:24:20|INFO|started gpfdist -p 8081 -P 8082 -f "/data/data/*" -t 30
>>
>> 2016-05-14 23:24:20|WARN|A gpload control file processing warning occurred. A delimiter must have a length of one. Special characters must be quoted. gpload will assume this is a sql escape character sequence.
>> 2016-05-14 23:24:20|ERROR|ERROR: *no partition for partitioning key*
>> (seg5 localhost:40000 pid=14591)
>>
>> encountered while running INSERT INTO public."stocks" ("storeno","itemno","itemname","spec","itemseq","stocks","unit","ymd") SELECT "storeno","itemno","itemname","spec","itemseq","stocks","unit","ymd" FROM ext_gpload20160514_232419_14569
>> 2016-05-14 23:24:20|INFO|rows Inserted = 0
>> 2016-05-14 23:24:20|INFO|rows Updated = 0
>> 2016-05-14 23:24:20|INFO|data formatting errors = 0
>> 2016-05-14 23:24:20|INFO|gpload failed
>>
>> It was failed,even if there is only one line.
>>
>> ------------------------------
>> yin.zhb@163.com
>>
>
>
> ---------- Forwarded message ----------
> From: Wen Lin <wl...@pivotal.io>
> To: dev@hawq.incubator.apache.org
> Cc: user@hawq.incubator.apache.org
> Date: Tue, 17 May 2016 10:59:16 +0800
> Subject: Re: HAWQ YARN RPC Errors
> Hi, Gagan,
>
> Where is the log? There is no attachment in your email.
>
> Thanks!
>
> On Sun, May 15, 2016 at 1:24 AM, Gagan Brahmi <ga...@gmail.com>
> wrote:
>
>> Hi Wen,
>>
>> Please find attached logs which has a few instances of the occurrence
>> of the error.
>>
>>
>> Regards,
>> Gagan Brahmi
>>
>> On Thu, May 12, 2016 at 7:33 PM, Wen Lin <wl...@pivotal.io> wrote:
>> > Hi, Gagan,
>> >
>> > It seems a sync failure between QD and Resource Manager. Not related to
>> > libyarn 's RPC.
>> > Would you like to attach the master's log file? Thanks!
>> >
>> > On Fri, May 13, 2016 at 12:58 AM, Gagan Brahmi <ga...@gmail.com>
>> > wrote:
>> >
>> >> Hi Team,
>> >>
>> >> Do we have some recommended tuning for the RPC warning/errors
>> >> encountered intermittently?
>> >>
>> >> The error which is seen is the following:
>> >>
>> >> WARNING: Sync RPC framework (inet) finds exception raised.
>> >> ERROR: failed to return resource to resource manager, failed to
>> >> receive content (pquery.c:991)
>> >>
>> >> This error however, disappears when we retry the query. There are
>> >> cases when the query is to be retried more than once.
>> >>
>> >> The error looks to be invoked when COMM2RM_CLIENT_FAIL_RECV is
>> encountered.
>> >>
>> >> The setup is using YARN resource manager. And the following is the
>> >> yarn-client configuration used:
>> >>
>> >> <configuration>
>> >>
>> >> <property>
>> >> <name>hadoop.security.authentication</name>
>> >> <value>kerberos</value>
>> >> </property>
>> >>
>> >> <property>
>> >> <name>rpc.client.connect.retry</name>
>> >> <value>10</value>
>> >> </property>
>> >>
>> >> <property>
>> >> <name>rpc.client.connect.tcpnodelay</name>
>> >> <value>true</value>
>> >> </property>
>> >>
>> >> <property>
>> >> <name>rpc.client.connect.timeout</name>
>> >> <value>600000</value>
>> >> </property>
>> >>
>> >> <property>
>> >> <name>rpc.client.max.idle</name>
>> >> <value>10000</value>
>> >> </property>
>> >>
>> >> <property>
>> >> <name>rpc.client.ping.interval</name>
>> >> <value>10000</value>
>> >> </property>
>> >>
>> >> <property>
>> >> <name>rpc.client.read.timeout</name>
>> >> <value>3600000</value>
>> >> </property>
>> >>
>> >> <property>
>> >> <name>rpc.client.socket.linger.timeout</name>
>> >> <value>-1</value>
>> >> </property>
>> >>
>> >> <property>
>> >> <name>rpc.client.timeout</name>
>> >> <value>3600000</value>
>> >> </property>
>> >>
>> >> <property>
>> >> <name>rpc.client.write.timeout</name>
>> >> <value>3600000</value>
>> >> </property>
>> >>
>> >> <property>
>> >> <name>yarn.client.failover.max.attempts</name>
>> >> <value>15</value>
>> >> </property>
>> >>
>> >> </configuration>
>> >>
>> >> I would appreciate some recommendations.
>> >>
>> >>
>> >> Regards,
>> >> Gagan Brahmi
>> >>
>>
>
>
>
--
*Shailesh Doshi | Data Engineering | Pivotal*
*Mobile: 513-604-9746 **| Email <sd...@pivotal.io>**: **sdoshi@pivotal.io*
<sd...@pivotal.io>
Big Data - Scale out & Open Source
<http://pivotal.io/big-data/pivotal-big-data-suite>