You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hawq.apache.org by Jon Roberts <jr...@pivotal.io> on 2016/12/14 14:36:23 UTC
unrecognized configuration parameter "ONETARY"
I'm getting the error message: unrecognized configuration parameter
"ONETARY" with a few of the TPC-DS queries where I'm increasing the number
of vsegs to get better performance. The error message alone is confusing
so even if I am doing something wrong, the message should be improved.
My environment:
AWS d2.8xlarge nodes
- 24 2 TB disks
- 252 GB RAM
- 36 cores
- Centos 6
- 10 nodes
- 1 admin node
- 10GB network
- 7 TB of data
- Standard TPC-DS Queries
- hawq_rm_memory_limit_perseg = 200gb
- hawq_rm_stmt_vseg_memory = 16gb
- Random distribution on all tables
I'm tried reducing the statement memory but that doesn't change anything.
Query 88 is a good example of this because it fails quickly.
set hawq_rm_nvseg_perquery_perseg_limit=12;
time psql -f 188.tpcds.88.sql
SET
Timing is on.
SET
Time: 0.157 ms
h8_30_to_9 | h9_to_9_30 | h9_30_to_10 | h10_to_10_30 | h10_30_to_11 |
h11_to_11_30 | h11_30_to_12 | h12_to_12_30
------------+------------+-------------+--------------+--------------+--------------+--------------+--------------
16279055 | 32496701 | 32493080 | 48732586 | 48782652 |
28460584 | 28453299 | 32518016
(1 row)
Time: 259695.969 ms
real 4m19.706s
user 0m0.001s
sys 0m0.003s
Next:
set hawq_rm_nvseg_perquery_perseg_limit=14;
time psql -f 188.tpcds.88.sql
SET
Timing is on.
SET
Time: 0.171 ms
psql:188.tpcds.88.sql:95: ERROR: Error dispatching to seg25
ip-172-21-13-189.ec2.internal:40000: connection pointer is NULL
DETAIL: Master unable to connect to seg25
ip-172-21-13-189.ec2.internal:40000: FATAL: unrecognized configuration
parameter "ONETARY"
real 0m8.787s
user 0m0.003s
sys 0m0.002s
Jon Roberts
Re: unrecognized configuration parameter "ONETARY"
Posted by Yi Jin <yj...@pivotal.io>.
Jon,
Thank you, yes, I am looking into this bug now.
Best,
Yi
On Thu, Dec 29, 2016 at 2:33 AM, Jon Roberts <jr...@pivotal.io> wrote:
> I created a Jira for this bug:
> https://issues.apache.org/jira/browse/HAWQ-1244
>
> Jon Roberts
> Principal Engineer | jroberts@pivotal.io | 615-426-8661
>
> On Wed, Dec 21, 2016 at 12:58 AM, Ming Li <ml...@pivotal.io> wrote:
>
> > Thanks Radar,
> >
> > It is strange that 'unrecognized configuration' will issue FATAL error, I
> > config in hawq-site.xml, and grep pg_log on master and segment date
> > directory, no FATAL error of 'unrecognized configuration'. Any ideas?
> >
> > On Wed, Dec 21, 2016 at 2:45 PM, Radar Da lei <rl...@pivotal.io> wrote:
> >
> > > Hi Ming,
> > >
> > > 'hawq_lc_monetary' is only used to store user defined '--lc-monetary'
> > value
> > > then pass below option to initdb:
> > > '--lc-monetary=${hawq_lc_monetary} '
> > >
> > > So finally we give '--lc-monetary=en_US.utf8 ' to initdb. Base on this
> I
> > > don't think the GUC 'ONETARY' have any dependencies with hawq admin
> > > tools. Thanks.
> > >
> > > Regards,
> > > Radar
> > >
> > > On Tue, Dec 20, 2016 at 10:18 AM, Ming Li <ml...@pivotal.io> wrote:
> > >
> > >> Hi Radar,
> > >>
> > >> It seems that the config param 'hawq_lc_monetary' can only be
> recognized
> > >> by hawq admin tools in python, and set hawq guc 'lc-monetary' finally.
> > >>
> > >> My question is: Why not directly using 'lc-monetary' instead of '
> > >> hawq_lc_monetary' in the hawq admin tools? Is there any possible to
> > >> conflict?
> > >>
> > >> Thanks.
> > >>
> > >> On Tue, Dec 20, 2016 at 6:45 AM, Yi Jin <yj...@pivotal.io> wrote:
> > >>
> > >>> Thank you jon, I will specially recheck the guc population logic. Yi
> > >>>
> > >>> On Tue, Dec 20, 2016 at 4:07 AM, Jon Roberts <jr...@pivotal.io>
> > >>> wrote:
> > >>>
> > >>> > The segment actually has this error:
> > >>> > 2016-12-14 13:47:34.760839
> > >>> > UTC,"gpadmin","gpadmin",p737499,th542214432,"172.21.
> > >>> > 13.196","40327",2016-12-14
> > >>> > 13:47:34 UTC,0,con23798,,seg-10000,,,,,
> "FATAL","42704","unrecognized
> > >>> > configuration parameter ""ONETARY""",,,,,,,0,,"guc.c",10006,
> > >>> >
> > >>> > This made me check out the configs because the error was raised
> from
> > >>> > guc.c. I found that /usr/local/hawq/etc/_mgmt_config has
> "onetary"
> > >>> text:
> > >>> > hawq_lc_monetary=en_US.utf8
> > >>> >
> > >>> > I'm using nodes with 24 drives so I have 24 temp directories set
> for
> > >>> both
> > >>> > the master and the segments. Thinking that possibly the problem is
> > >>> related
> > >>> > to the number of vSegs * the size of the config file exceeding a
> > >>> variable
> > >>> > size, I decided to reduce the temp directories down to just two for
> > >>> both
> > >>> > the master and segments.
> > >>> >
> > >>> > After restarting HAWQ, I could get the query to use 16 and even 24
> > >>> vSegs
> > >>> > without a problem.
> > >>> >
> > >>> > So maybe the guc logic needs revisiting to make sure there isn't an
> > >>> issue
> > >>> > with parsing the GUCs when there are many vSegs and many temp
> > >>> directories.
> > >>> > I would think this would be a problem even with the default number
> of
> > >>> vSegs
> > >>> > per host on really large clusters.
> > >>> >
> > >>> >
> > >>> > Jon Roberts
> > >>> > Principal Engineer | jroberts@pivotal.io | 615-426-8661
> > >>> <(615)%20426-8661>
> > >>> >
> > >>> > On Mon, Dec 19, 2016 at 4:44 AM, Yi Jin <yj...@pivotal.io> wrote:
> > >>> >
> > >>> > > Hi Jon,
> > >>> > >
> > >>> > > I think there is no ONTARY configuration item, so maybe I need
> the
> > >>> full
> > >>> > log
> > >>> > > containing that error to check the error routine.
> > >>> > >
> > >>> > > Best,
> > >>> > > Yi
> > >>> > >
> > >>> > > On Mon, Dec 19, 2016 at 5:39 PM, Paul Guo <pa...@gmail.com>
> > wrote:
> > >>> > >
> > >>> > > > You could grep the log to see whether there is "ONETARY"
> setting
> > >>> before
> > >>> > > > this error occurs, and also grep configuration files and
> related
> > >>> test
> > >>> > > files
> > >>> > > > to find who tried to set this.
> > >>> > > >
> > >>> > > > 2016-12-14 22:36 GMT+08:00 Jon Roberts <jr...@pivotal.io>:
> > >>> > > >
> > >>> > > > > I'm getting the error message: unrecognized configuration
> > >>> parameter
> > >>> > > > > "ONETARY" with a few of the TPC-DS queries where I'm
> increasing
> > >>> the
> > >>> > > > number
> > >>> > > > > of vsegs to get better performance. The error message alone
> is
> > >>> > > confusing
> > >>> > > > > so even if I am doing something wrong, the message should be
> > >>> > improved.
> > >>> > > > >
> > >>> > > > > My environment:
> > >>> > > > >
> > >>> > > > > AWS d2.8xlarge nodes
> > >>> > > > > - 24 2 TB disks
> > >>> > > > > - 252 GB RAM
> > >>> > > > > - 36 cores
> > >>> > > > > - Centos 6
> > >>> > > > > - 10 nodes
> > >>> > > > > - 1 admin node
> > >>> > > > > - 10GB network
> > >>> > > > > - 7 TB of data
> > >>> > > > > - Standard TPC-DS Queries
> > >>> > > > > - hawq_rm_memory_limit_perseg = 200gb
> > >>> > > > > - hawq_rm_stmt_vseg_memory = 16gb
> > >>> > > > > - Random distribution on all tables
> > >>> > > > >
> > >>> > > > > I'm tried reducing the statement memory but that doesn't
> change
> > >>> > > anything.
> > >>> > > > >
> > >>> > > > > Query 88 is a good example of this because it fails quickly.
> > >>> > > > >
> > >>> > > > > set hawq_rm_nvseg_perquery_perseg_limit=12;
> > >>> > > > >
> > >>> > > > > time psql -f 188.tpcds.88.sql
> > >>> > > > > SET
> > >>> > > > > Timing is on.
> > >>> > > > > SET
> > >>> > > > > Time: 0.157 ms
> > >>> > > > > h8_30_to_9 | h9_to_9_30 | h9_30_to_10 | h10_to_10_30 |
> > >>> h10_30_to_11
> > >>> > |
> > >>> > > > > h11_to_11_30 | h11_30_to_12 | h12_to_12_30
> > >>> > > > > ------------+------------+-------------+--------------+-----
> > >>> > > > > ---------+--------------+--------------+--------------
> > >>> > > > > 16279055 | 32496701 | 32493080 | 48732586 |
> > >>> 48782652
> > >>> > |
> > >>> > > > > 28460584 | 28453299 | 32518016
> > >>> > > > > (1 row)
> > >>> > > > >
> > >>> > > > > Time: 259695.969 ms
> > >>> > > > >
> > >>> > > > > real 4m19.706s
> > >>> > > > > user 0m0.001s
> > >>> > > > > sys 0m0.003s
> > >>> > > > >
> > >>> > > > > Next:
> > >>> > > > > set hawq_rm_nvseg_perquery_perseg_limit=14;
> > >>> > > > >
> > >>> > > > > time psql -f 188.tpcds.88.sql
> > >>> > > > > SET
> > >>> > > > > Timing is on.
> > >>> > > > > SET
> > >>> > > > > Time: 0.171 ms
> > >>> > > > > psql:188.tpcds.88.sql:95: ERROR: Error dispatching to seg25
> > >>> > > > > ip-172-21-13-189.ec2.internal:40000: connection pointer is
> > NULL
> > >>> > > > > DETAIL: Master unable to connect to seg25
> > >>> > > > > ip-172-21-13-189.ec2.internal:40000: FATAL: unrecognized
> > >>> > > configuration
> > >>> > > > > parameter "ONETARY"
> > >>> > > > >
> > >>> > > > > real 0m8.787s
> > >>> > > > > user 0m0.003s
> > >>> > > > > sys 0m0.002s
> > >>> > > > >
> > >>> > > > >
> > >>> > > > > Jon Roberts
> > >>> > > > >
> > >>> > > >
> > >>> > >
> > >>> >
> > >>>
> > >>
> > >>
> > >
> >
>
Re: unrecognized configuration parameter "ONETARY"
Posted by Jon Roberts <jr...@pivotal.io>.
I created a Jira for this bug:
https://issues.apache.org/jira/browse/HAWQ-1244
Jon Roberts
Principal Engineer | jroberts@pivotal.io | 615-426-8661
On Wed, Dec 21, 2016 at 12:58 AM, Ming Li <ml...@pivotal.io> wrote:
> Thanks Radar,
>
> It is strange that 'unrecognized configuration' will issue FATAL error, I
> config in hawq-site.xml, and grep pg_log on master and segment date
> directory, no FATAL error of 'unrecognized configuration'. Any ideas?
>
> On Wed, Dec 21, 2016 at 2:45 PM, Radar Da lei <rl...@pivotal.io> wrote:
>
> > Hi Ming,
> >
> > 'hawq_lc_monetary' is only used to store user defined '--lc-monetary'
> value
> > then pass below option to initdb:
> > '--lc-monetary=${hawq_lc_monetary} '
> >
> > So finally we give '--lc-monetary=en_US.utf8 ' to initdb. Base on this I
> > don't think the GUC 'ONETARY' have any dependencies with hawq admin
> > tools. Thanks.
> >
> > Regards,
> > Radar
> >
> > On Tue, Dec 20, 2016 at 10:18 AM, Ming Li <ml...@pivotal.io> wrote:
> >
> >> Hi Radar,
> >>
> >> It seems that the config param 'hawq_lc_monetary' can only be recognized
> >> by hawq admin tools in python, and set hawq guc 'lc-monetary' finally.
> >>
> >> My question is: Why not directly using 'lc-monetary' instead of '
> >> hawq_lc_monetary' in the hawq admin tools? Is there any possible to
> >> conflict?
> >>
> >> Thanks.
> >>
> >> On Tue, Dec 20, 2016 at 6:45 AM, Yi Jin <yj...@pivotal.io> wrote:
> >>
> >>> Thank you jon, I will specially recheck the guc population logic. Yi
> >>>
> >>> On Tue, Dec 20, 2016 at 4:07 AM, Jon Roberts <jr...@pivotal.io>
> >>> wrote:
> >>>
> >>> > The segment actually has this error:
> >>> > 2016-12-14 13:47:34.760839
> >>> > UTC,"gpadmin","gpadmin",p737499,th542214432,"172.21.
> >>> > 13.196","40327",2016-12-14
> >>> > 13:47:34 UTC,0,con23798,,seg-10000,,,,,"FATAL","42704","unrecognized
> >>> > configuration parameter ""ONETARY""",,,,,,,0,,"guc.c",10006,
> >>> >
> >>> > This made me check out the configs because the error was raised from
> >>> > guc.c. I found that /usr/local/hawq/etc/_mgmt_config has "onetary"
> >>> text:
> >>> > hawq_lc_monetary=en_US.utf8
> >>> >
> >>> > I'm using nodes with 24 drives so I have 24 temp directories set for
> >>> both
> >>> > the master and the segments. Thinking that possibly the problem is
> >>> related
> >>> > to the number of vSegs * the size of the config file exceeding a
> >>> variable
> >>> > size, I decided to reduce the temp directories down to just two for
> >>> both
> >>> > the master and segments.
> >>> >
> >>> > After restarting HAWQ, I could get the query to use 16 and even 24
> >>> vSegs
> >>> > without a problem.
> >>> >
> >>> > So maybe the guc logic needs revisiting to make sure there isn't an
> >>> issue
> >>> > with parsing the GUCs when there are many vSegs and many temp
> >>> directories.
> >>> > I would think this would be a problem even with the default number of
> >>> vSegs
> >>> > per host on really large clusters.
> >>> >
> >>> >
> >>> > Jon Roberts
> >>> > Principal Engineer | jroberts@pivotal.io | 615-426-8661
> >>> <(615)%20426-8661>
> >>> >
> >>> > On Mon, Dec 19, 2016 at 4:44 AM, Yi Jin <yj...@pivotal.io> wrote:
> >>> >
> >>> > > Hi Jon,
> >>> > >
> >>> > > I think there is no ONTARY configuration item, so maybe I need the
> >>> full
> >>> > log
> >>> > > containing that error to check the error routine.
> >>> > >
> >>> > > Best,
> >>> > > Yi
> >>> > >
> >>> > > On Mon, Dec 19, 2016 at 5:39 PM, Paul Guo <pa...@gmail.com>
> wrote:
> >>> > >
> >>> > > > You could grep the log to see whether there is "ONETARY" setting
> >>> before
> >>> > > > this error occurs, and also grep configuration files and related
> >>> test
> >>> > > files
> >>> > > > to find who tried to set this.
> >>> > > >
> >>> > > > 2016-12-14 22:36 GMT+08:00 Jon Roberts <jr...@pivotal.io>:
> >>> > > >
> >>> > > > > I'm getting the error message: unrecognized configuration
> >>> parameter
> >>> > > > > "ONETARY" with a few of the TPC-DS queries where I'm increasing
> >>> the
> >>> > > > number
> >>> > > > > of vsegs to get better performance. The error message alone is
> >>> > > confusing
> >>> > > > > so even if I am doing something wrong, the message should be
> >>> > improved.
> >>> > > > >
> >>> > > > > My environment:
> >>> > > > >
> >>> > > > > AWS d2.8xlarge nodes
> >>> > > > > - 24 2 TB disks
> >>> > > > > - 252 GB RAM
> >>> > > > > - 36 cores
> >>> > > > > - Centos 6
> >>> > > > > - 10 nodes
> >>> > > > > - 1 admin node
> >>> > > > > - 10GB network
> >>> > > > > - 7 TB of data
> >>> > > > > - Standard TPC-DS Queries
> >>> > > > > - hawq_rm_memory_limit_perseg = 200gb
> >>> > > > > - hawq_rm_stmt_vseg_memory = 16gb
> >>> > > > > - Random distribution on all tables
> >>> > > > >
> >>> > > > > I'm tried reducing the statement memory but that doesn't change
> >>> > > anything.
> >>> > > > >
> >>> > > > > Query 88 is a good example of this because it fails quickly.
> >>> > > > >
> >>> > > > > set hawq_rm_nvseg_perquery_perseg_limit=12;
> >>> > > > >
> >>> > > > > time psql -f 188.tpcds.88.sql
> >>> > > > > SET
> >>> > > > > Timing is on.
> >>> > > > > SET
> >>> > > > > Time: 0.157 ms
> >>> > > > > h8_30_to_9 | h9_to_9_30 | h9_30_to_10 | h10_to_10_30 |
> >>> h10_30_to_11
> >>> > |
> >>> > > > > h11_to_11_30 | h11_30_to_12 | h12_to_12_30
> >>> > > > > ------------+------------+-------------+--------------+-----
> >>> > > > > ---------+--------------+--------------+--------------
> >>> > > > > 16279055 | 32496701 | 32493080 | 48732586 |
> >>> 48782652
> >>> > |
> >>> > > > > 28460584 | 28453299 | 32518016
> >>> > > > > (1 row)
> >>> > > > >
> >>> > > > > Time: 259695.969 ms
> >>> > > > >
> >>> > > > > real 4m19.706s
> >>> > > > > user 0m0.001s
> >>> > > > > sys 0m0.003s
> >>> > > > >
> >>> > > > > Next:
> >>> > > > > set hawq_rm_nvseg_perquery_perseg_limit=14;
> >>> > > > >
> >>> > > > > time psql -f 188.tpcds.88.sql
> >>> > > > > SET
> >>> > > > > Timing is on.
> >>> > > > > SET
> >>> > > > > Time: 0.171 ms
> >>> > > > > psql:188.tpcds.88.sql:95: ERROR: Error dispatching to seg25
> >>> > > > > ip-172-21-13-189.ec2.internal:40000: connection pointer is
> NULL
> >>> > > > > DETAIL: Master unable to connect to seg25
> >>> > > > > ip-172-21-13-189.ec2.internal:40000: FATAL: unrecognized
> >>> > > configuration
> >>> > > > > parameter "ONETARY"
> >>> > > > >
> >>> > > > > real 0m8.787s
> >>> > > > > user 0m0.003s
> >>> > > > > sys 0m0.002s
> >>> > > > >
> >>> > > > >
> >>> > > > > Jon Roberts
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> >>
> >>
> >
>
Re: unrecognized configuration parameter "ONETARY"
Posted by Ming Li <ml...@pivotal.io>.
Thanks Radar,
It is strange that 'unrecognized configuration' will issue FATAL error, I
config in hawq-site.xml, and grep pg_log on master and segment date
directory, no FATAL error of 'unrecognized configuration'. Any ideas?
On Wed, Dec 21, 2016 at 2:45 PM, Radar Da lei <rl...@pivotal.io> wrote:
> Hi Ming,
>
> 'hawq_lc_monetary' is only used to store user defined '--lc-monetary' value
> then pass below option to initdb:
> '--lc-monetary=${hawq_lc_monetary} '
>
> So finally we give '--lc-monetary=en_US.utf8 ' to initdb. Base on this I
> don't think the GUC 'ONETARY' have any dependencies with hawq admin
> tools. Thanks.
>
> Regards,
> Radar
>
> On Tue, Dec 20, 2016 at 10:18 AM, Ming Li <ml...@pivotal.io> wrote:
>
>> Hi Radar,
>>
>> It seems that the config param 'hawq_lc_monetary' can only be recognized
>> by hawq admin tools in python, and set hawq guc 'lc-monetary' finally.
>>
>> My question is: Why not directly using 'lc-monetary' instead of '
>> hawq_lc_monetary' in the hawq admin tools? Is there any possible to
>> conflict?
>>
>> Thanks.
>>
>> On Tue, Dec 20, 2016 at 6:45 AM, Yi Jin <yj...@pivotal.io> wrote:
>>
>>> Thank you jon, I will specially recheck the guc population logic. Yi
>>>
>>> On Tue, Dec 20, 2016 at 4:07 AM, Jon Roberts <jr...@pivotal.io>
>>> wrote:
>>>
>>> > The segment actually has this error:
>>> > 2016-12-14 13:47:34.760839
>>> > UTC,"gpadmin","gpadmin",p737499,th542214432,"172.21.
>>> > 13.196","40327",2016-12-14
>>> > 13:47:34 UTC,0,con23798,,seg-10000,,,,,"FATAL","42704","unrecognized
>>> > configuration parameter ""ONETARY""",,,,,,,0,,"guc.c",10006,
>>> >
>>> > This made me check out the configs because the error was raised from
>>> > guc.c. I found that /usr/local/hawq/etc/_mgmt_config has "onetary"
>>> text:
>>> > hawq_lc_monetary=en_US.utf8
>>> >
>>> > I'm using nodes with 24 drives so I have 24 temp directories set for
>>> both
>>> > the master and the segments. Thinking that possibly the problem is
>>> related
>>> > to the number of vSegs * the size of the config file exceeding a
>>> variable
>>> > size, I decided to reduce the temp directories down to just two for
>>> both
>>> > the master and segments.
>>> >
>>> > After restarting HAWQ, I could get the query to use 16 and even 24
>>> vSegs
>>> > without a problem.
>>> >
>>> > So maybe the guc logic needs revisiting to make sure there isn't an
>>> issue
>>> > with parsing the GUCs when there are many vSegs and many temp
>>> directories.
>>> > I would think this would be a problem even with the default number of
>>> vSegs
>>> > per host on really large clusters.
>>> >
>>> >
>>> > Jon Roberts
>>> > Principal Engineer | jroberts@pivotal.io | 615-426-8661
>>> <(615)%20426-8661>
>>> >
>>> > On Mon, Dec 19, 2016 at 4:44 AM, Yi Jin <yj...@pivotal.io> wrote:
>>> >
>>> > > Hi Jon,
>>> > >
>>> > > I think there is no ONTARY configuration item, so maybe I need the
>>> full
>>> > log
>>> > > containing that error to check the error routine.
>>> > >
>>> > > Best,
>>> > > Yi
>>> > >
>>> > > On Mon, Dec 19, 2016 at 5:39 PM, Paul Guo <pa...@gmail.com> wrote:
>>> > >
>>> > > > You could grep the log to see whether there is "ONETARY" setting
>>> before
>>> > > > this error occurs, and also grep configuration files and related
>>> test
>>> > > files
>>> > > > to find who tried to set this.
>>> > > >
>>> > > > 2016-12-14 22:36 GMT+08:00 Jon Roberts <jr...@pivotal.io>:
>>> > > >
>>> > > > > I'm getting the error message: unrecognized configuration
>>> parameter
>>> > > > > "ONETARY" with a few of the TPC-DS queries where I'm increasing
>>> the
>>> > > > number
>>> > > > > of vsegs to get better performance. The error message alone is
>>> > > confusing
>>> > > > > so even if I am doing something wrong, the message should be
>>> > improved.
>>> > > > >
>>> > > > > My environment:
>>> > > > >
>>> > > > > AWS d2.8xlarge nodes
>>> > > > > - 24 2 TB disks
>>> > > > > - 252 GB RAM
>>> > > > > - 36 cores
>>> > > > > - Centos 6
>>> > > > > - 10 nodes
>>> > > > > - 1 admin node
>>> > > > > - 10GB network
>>> > > > > - 7 TB of data
>>> > > > > - Standard TPC-DS Queries
>>> > > > > - hawq_rm_memory_limit_perseg = 200gb
>>> > > > > - hawq_rm_stmt_vseg_memory = 16gb
>>> > > > > - Random distribution on all tables
>>> > > > >
>>> > > > > I'm tried reducing the statement memory but that doesn't change
>>> > > anything.
>>> > > > >
>>> > > > > Query 88 is a good example of this because it fails quickly.
>>> > > > >
>>> > > > > set hawq_rm_nvseg_perquery_perseg_limit=12;
>>> > > > >
>>> > > > > time psql -f 188.tpcds.88.sql
>>> > > > > SET
>>> > > > > Timing is on.
>>> > > > > SET
>>> > > > > Time: 0.157 ms
>>> > > > > h8_30_to_9 | h9_to_9_30 | h9_30_to_10 | h10_to_10_30 |
>>> h10_30_to_11
>>> > |
>>> > > > > h11_to_11_30 | h11_30_to_12 | h12_to_12_30
>>> > > > > ------------+------------+-------------+--------------+-----
>>> > > > > ---------+--------------+--------------+--------------
>>> > > > > 16279055 | 32496701 | 32493080 | 48732586 |
>>> 48782652
>>> > |
>>> > > > > 28460584 | 28453299 | 32518016
>>> > > > > (1 row)
>>> > > > >
>>> > > > > Time: 259695.969 ms
>>> > > > >
>>> > > > > real 4m19.706s
>>> > > > > user 0m0.001s
>>> > > > > sys 0m0.003s
>>> > > > >
>>> > > > > Next:
>>> > > > > set hawq_rm_nvseg_perquery_perseg_limit=14;
>>> > > > >
>>> > > > > time psql -f 188.tpcds.88.sql
>>> > > > > SET
>>> > > > > Timing is on.
>>> > > > > SET
>>> > > > > Time: 0.171 ms
>>> > > > > psql:188.tpcds.88.sql:95: ERROR: Error dispatching to seg25
>>> > > > > ip-172-21-13-189.ec2.internal:40000: connection pointer is NULL
>>> > > > > DETAIL: Master unable to connect to seg25
>>> > > > > ip-172-21-13-189.ec2.internal:40000: FATAL: unrecognized
>>> > > configuration
>>> > > > > parameter "ONETARY"
>>> > > > >
>>> > > > > real 0m8.787s
>>> > > > > user 0m0.003s
>>> > > > > sys 0m0.002s
>>> > > > >
>>> > > > >
>>> > > > > Jon Roberts
>>> > > > >
>>> > > >
>>> > >
>>> >
>>>
>>
>>
>
Re: unrecognized configuration parameter "ONETARY"
Posted by Radar Da lei <rl...@pivotal.io>.
Hi Ming,
'hawq_lc_monetary' is only used to store user defined '--lc-monetary' value
then pass below option to initdb:
'--lc-monetary=${hawq_lc_monetary} '
So finally we give '--lc-monetary=en_US.utf8 ' to initdb. Base on this I
don't think the GUC 'ONETARY' have any dependencies with hawq admin tools.
Thanks.
Regards,
Radar
On Tue, Dec 20, 2016 at 10:18 AM, Ming Li <ml...@pivotal.io> wrote:
> Hi Radar,
>
> It seems that the config param 'hawq_lc_monetary' can only be recognized
> by hawq admin tools in python, and set hawq guc 'lc-monetary' finally.
>
> My question is: Why not directly using 'lc-monetary' instead of '
> hawq_lc_monetary' in the hawq admin tools? Is there any possible to
> conflict?
>
> Thanks.
>
> On Tue, Dec 20, 2016 at 6:45 AM, Yi Jin <yj...@pivotal.io> wrote:
>
>> Thank you jon, I will specially recheck the guc population logic. Yi
>>
>> On Tue, Dec 20, 2016 at 4:07 AM, Jon Roberts <jr...@pivotal.io> wrote:
>>
>> > The segment actually has this error:
>> > 2016-12-14 13:47:34.760839
>> > UTC,"gpadmin","gpadmin",p737499,th542214432,"172.21.
>> > 13.196","40327",2016-12-14
>> > 13:47:34 UTC,0,con23798,,seg-10000,,,,,"FATAL","42704","unrecognized
>> > configuration parameter ""ONETARY""",,,,,,,0,,"guc.c",10006,
>> >
>> > This made me check out the configs because the error was raised from
>> > guc.c. I found that /usr/local/hawq/etc/_mgmt_config has "onetary"
>> text:
>> > hawq_lc_monetary=en_US.utf8
>> >
>> > I'm using nodes with 24 drives so I have 24 temp directories set for
>> both
>> > the master and the segments. Thinking that possibly the problem is
>> related
>> > to the number of vSegs * the size of the config file exceeding a
>> variable
>> > size, I decided to reduce the temp directories down to just two for both
>> > the master and segments.
>> >
>> > After restarting HAWQ, I could get the query to use 16 and even 24 vSegs
>> > without a problem.
>> >
>> > So maybe the guc logic needs revisiting to make sure there isn't an
>> issue
>> > with parsing the GUCs when there are many vSegs and many temp
>> directories.
>> > I would think this would be a problem even with the default number of
>> vSegs
>> > per host on really large clusters.
>> >
>> >
>> > Jon Roberts
>> > Principal Engineer | jroberts@pivotal.io | 615-426-8661
>> <(615)%20426-8661>
>> >
>> > On Mon, Dec 19, 2016 at 4:44 AM, Yi Jin <yj...@pivotal.io> wrote:
>> >
>> > > Hi Jon,
>> > >
>> > > I think there is no ONTARY configuration item, so maybe I need the
>> full
>> > log
>> > > containing that error to check the error routine.
>> > >
>> > > Best,
>> > > Yi
>> > >
>> > > On Mon, Dec 19, 2016 at 5:39 PM, Paul Guo <pa...@gmail.com> wrote:
>> > >
>> > > > You could grep the log to see whether there is "ONETARY" setting
>> before
>> > > > this error occurs, and also grep configuration files and related
>> test
>> > > files
>> > > > to find who tried to set this.
>> > > >
>> > > > 2016-12-14 22:36 GMT+08:00 Jon Roberts <jr...@pivotal.io>:
>> > > >
>> > > > > I'm getting the error message: unrecognized configuration
>> parameter
>> > > > > "ONETARY" with a few of the TPC-DS queries where I'm increasing
>> the
>> > > > number
>> > > > > of vsegs to get better performance. The error message alone is
>> > > confusing
>> > > > > so even if I am doing something wrong, the message should be
>> > improved.
>> > > > >
>> > > > > My environment:
>> > > > >
>> > > > > AWS d2.8xlarge nodes
>> > > > > - 24 2 TB disks
>> > > > > - 252 GB RAM
>> > > > > - 36 cores
>> > > > > - Centos 6
>> > > > > - 10 nodes
>> > > > > - 1 admin node
>> > > > > - 10GB network
>> > > > > - 7 TB of data
>> > > > > - Standard TPC-DS Queries
>> > > > > - hawq_rm_memory_limit_perseg = 200gb
>> > > > > - hawq_rm_stmt_vseg_memory = 16gb
>> > > > > - Random distribution on all tables
>> > > > >
>> > > > > I'm tried reducing the statement memory but that doesn't change
>> > > anything.
>> > > > >
>> > > > > Query 88 is a good example of this because it fails quickly.
>> > > > >
>> > > > > set hawq_rm_nvseg_perquery_perseg_limit=12;
>> > > > >
>> > > > > time psql -f 188.tpcds.88.sql
>> > > > > SET
>> > > > > Timing is on.
>> > > > > SET
>> > > > > Time: 0.157 ms
>> > > > > h8_30_to_9 | h9_to_9_30 | h9_30_to_10 | h10_to_10_30 |
>> h10_30_to_11
>> > |
>> > > > > h11_to_11_30 | h11_30_to_12 | h12_to_12_30
>> > > > > ------------+------------+-------------+--------------+-----
>> > > > > ---------+--------------+--------------+--------------
>> > > > > 16279055 | 32496701 | 32493080 | 48732586 |
>> 48782652
>> > |
>> > > > > 28460584 | 28453299 | 32518016
>> > > > > (1 row)
>> > > > >
>> > > > > Time: 259695.969 ms
>> > > > >
>> > > > > real 4m19.706s
>> > > > > user 0m0.001s
>> > > > > sys 0m0.003s
>> > > > >
>> > > > > Next:
>> > > > > set hawq_rm_nvseg_perquery_perseg_limit=14;
>> > > > >
>> > > > > time psql -f 188.tpcds.88.sql
>> > > > > SET
>> > > > > Timing is on.
>> > > > > SET
>> > > > > Time: 0.171 ms
>> > > > > psql:188.tpcds.88.sql:95: ERROR: Error dispatching to seg25
>> > > > > ip-172-21-13-189.ec2.internal:40000: connection pointer is NULL
>> > > > > DETAIL: Master unable to connect to seg25
>> > > > > ip-172-21-13-189.ec2.internal:40000: FATAL: unrecognized
>> > > configuration
>> > > > > parameter "ONETARY"
>> > > > >
>> > > > > real 0m8.787s
>> > > > > user 0m0.003s
>> > > > > sys 0m0.002s
>> > > > >
>> > > > >
>> > > > > Jon Roberts
>> > > > >
>> > > >
>> > >
>> >
>>
>
>
Re: unrecognized configuration parameter "ONETARY"
Posted by Ming Li <ml...@pivotal.io>.
Hi Radar,
It seems that the config param 'hawq_lc_monetary' can only be recognized by
hawq admin tools in python, and set hawq guc 'lc-monetary' finally.
My question is: Why not directly using 'lc-monetary' instead of '
hawq_lc_monetary' in the hawq admin tools? Is there any possible to
conflict?
Thanks.
On Tue, Dec 20, 2016 at 6:45 AM, Yi Jin <yj...@pivotal.io> wrote:
> Thank you jon, I will specially recheck the guc population logic. Yi
>
> On Tue, Dec 20, 2016 at 4:07 AM, Jon Roberts <jr...@pivotal.io> wrote:
>
> > The segment actually has this error:
> > 2016-12-14 13:47:34.760839
> > UTC,"gpadmin","gpadmin",p737499,th542214432,"172.21.
> > 13.196","40327",2016-12-14
> > 13:47:34 UTC,0,con23798,,seg-10000,,,,,"FATAL","42704","unrecognized
> > configuration parameter ""ONETARY""",,,,,,,0,,"guc.c",10006,
> >
> > This made me check out the configs because the error was raised from
> > guc.c. I found that /usr/local/hawq/etc/_mgmt_config has "onetary"
> text:
> > hawq_lc_monetary=en_US.utf8
> >
> > I'm using nodes with 24 drives so I have 24 temp directories set for both
> > the master and the segments. Thinking that possibly the problem is
> related
> > to the number of vSegs * the size of the config file exceeding a variable
> > size, I decided to reduce the temp directories down to just two for both
> > the master and segments.
> >
> > After restarting HAWQ, I could get the query to use 16 and even 24 vSegs
> > without a problem.
> >
> > So maybe the guc logic needs revisiting to make sure there isn't an issue
> > with parsing the GUCs when there are many vSegs and many temp
> directories.
> > I would think this would be a problem even with the default number of
> vSegs
> > per host on really large clusters.
> >
> >
> > Jon Roberts
> > Principal Engineer | jroberts@pivotal.io | 615-426-8661
> >
> > On Mon, Dec 19, 2016 at 4:44 AM, Yi Jin <yj...@pivotal.io> wrote:
> >
> > > Hi Jon,
> > >
> > > I think there is no ONTARY configuration item, so maybe I need the full
> > log
> > > containing that error to check the error routine.
> > >
> > > Best,
> > > Yi
> > >
> > > On Mon, Dec 19, 2016 at 5:39 PM, Paul Guo <pa...@gmail.com> wrote:
> > >
> > > > You could grep the log to see whether there is "ONETARY" setting
> before
> > > > this error occurs, and also grep configuration files and related test
> > > files
> > > > to find who tried to set this.
> > > >
> > > > 2016-12-14 22:36 GMT+08:00 Jon Roberts <jr...@pivotal.io>:
> > > >
> > > > > I'm getting the error message: unrecognized configuration parameter
> > > > > "ONETARY" with a few of the TPC-DS queries where I'm increasing the
> > > > number
> > > > > of vsegs to get better performance. The error message alone is
> > > confusing
> > > > > so even if I am doing something wrong, the message should be
> > improved.
> > > > >
> > > > > My environment:
> > > > >
> > > > > AWS d2.8xlarge nodes
> > > > > - 24 2 TB disks
> > > > > - 252 GB RAM
> > > > > - 36 cores
> > > > > - Centos 6
> > > > > - 10 nodes
> > > > > - 1 admin node
> > > > > - 10GB network
> > > > > - 7 TB of data
> > > > > - Standard TPC-DS Queries
> > > > > - hawq_rm_memory_limit_perseg = 200gb
> > > > > - hawq_rm_stmt_vseg_memory = 16gb
> > > > > - Random distribution on all tables
> > > > >
> > > > > I'm tried reducing the statement memory but that doesn't change
> > > anything.
> > > > >
> > > > > Query 88 is a good example of this because it fails quickly.
> > > > >
> > > > > set hawq_rm_nvseg_perquery_perseg_limit=12;
> > > > >
> > > > > time psql -f 188.tpcds.88.sql
> > > > > SET
> > > > > Timing is on.
> > > > > SET
> > > > > Time: 0.157 ms
> > > > > h8_30_to_9 | h9_to_9_30 | h9_30_to_10 | h10_to_10_30 |
> h10_30_to_11
> > |
> > > > > h11_to_11_30 | h11_30_to_12 | h12_to_12_30
> > > > > ------------+------------+-------------+--------------+-----
> > > > > ---------+--------------+--------------+--------------
> > > > > 16279055 | 32496701 | 32493080 | 48732586 |
> 48782652
> > |
> > > > > 28460584 | 28453299 | 32518016
> > > > > (1 row)
> > > > >
> > > > > Time: 259695.969 ms
> > > > >
> > > > > real 4m19.706s
> > > > > user 0m0.001s
> > > > > sys 0m0.003s
> > > > >
> > > > > Next:
> > > > > set hawq_rm_nvseg_perquery_perseg_limit=14;
> > > > >
> > > > > time psql -f 188.tpcds.88.sql
> > > > > SET
> > > > > Timing is on.
> > > > > SET
> > > > > Time: 0.171 ms
> > > > > psql:188.tpcds.88.sql:95: ERROR: Error dispatching to seg25
> > > > > ip-172-21-13-189.ec2.internal:40000: connection pointer is NULL
> > > > > DETAIL: Master unable to connect to seg25
> > > > > ip-172-21-13-189.ec2.internal:40000: FATAL: unrecognized
> > > configuration
> > > > > parameter "ONETARY"
> > > > >
> > > > > real 0m8.787s
> > > > > user 0m0.003s
> > > > > sys 0m0.002s
> > > > >
> > > > >
> > > > > Jon Roberts
> > > > >
> > > >
> > >
> >
>
Re: unrecognized configuration parameter "ONETARY"
Posted by Yi Jin <yj...@pivotal.io>.
Thank you jon, I will specially recheck the guc population logic. Yi
On Tue, Dec 20, 2016 at 4:07 AM, Jon Roberts <jr...@pivotal.io> wrote:
> The segment actually has this error:
> 2016-12-14 13:47:34.760839
> UTC,"gpadmin","gpadmin",p737499,th542214432,"172.21.
> 13.196","40327",2016-12-14
> 13:47:34 UTC,0,con23798,,seg-10000,,,,,"FATAL","42704","unrecognized
> configuration parameter ""ONETARY""",,,,,,,0,,"guc.c",10006,
>
> This made me check out the configs because the error was raised from
> guc.c. I found that /usr/local/hawq/etc/_mgmt_config has "onetary" text:
> hawq_lc_monetary=en_US.utf8
>
> I'm using nodes with 24 drives so I have 24 temp directories set for both
> the master and the segments. Thinking that possibly the problem is related
> to the number of vSegs * the size of the config file exceeding a variable
> size, I decided to reduce the temp directories down to just two for both
> the master and segments.
>
> After restarting HAWQ, I could get the query to use 16 and even 24 vSegs
> without a problem.
>
> So maybe the guc logic needs revisiting to make sure there isn't an issue
> with parsing the GUCs when there are many vSegs and many temp directories.
> I would think this would be a problem even with the default number of vSegs
> per host on really large clusters.
>
>
> Jon Roberts
> Principal Engineer | jroberts@pivotal.io | 615-426-8661
>
> On Mon, Dec 19, 2016 at 4:44 AM, Yi Jin <yj...@pivotal.io> wrote:
>
> > Hi Jon,
> >
> > I think there is no ONTARY configuration item, so maybe I need the full
> log
> > containing that error to check the error routine.
> >
> > Best,
> > Yi
> >
> > On Mon, Dec 19, 2016 at 5:39 PM, Paul Guo <pa...@gmail.com> wrote:
> >
> > > You could grep the log to see whether there is "ONETARY" setting before
> > > this error occurs, and also grep configuration files and related test
> > files
> > > to find who tried to set this.
> > >
> > > 2016-12-14 22:36 GMT+08:00 Jon Roberts <jr...@pivotal.io>:
> > >
> > > > I'm getting the error message: unrecognized configuration parameter
> > > > "ONETARY" with a few of the TPC-DS queries where I'm increasing the
> > > number
> > > > of vsegs to get better performance. The error message alone is
> > confusing
> > > > so even if I am doing something wrong, the message should be
> improved.
> > > >
> > > > My environment:
> > > >
> > > > AWS d2.8xlarge nodes
> > > > - 24 2 TB disks
> > > > - 252 GB RAM
> > > > - 36 cores
> > > > - Centos 6
> > > > - 10 nodes
> > > > - 1 admin node
> > > > - 10GB network
> > > > - 7 TB of data
> > > > - Standard TPC-DS Queries
> > > > - hawq_rm_memory_limit_perseg = 200gb
> > > > - hawq_rm_stmt_vseg_memory = 16gb
> > > > - Random distribution on all tables
> > > >
> > > > I'm tried reducing the statement memory but that doesn't change
> > anything.
> > > >
> > > > Query 88 is a good example of this because it fails quickly.
> > > >
> > > > set hawq_rm_nvseg_perquery_perseg_limit=12;
> > > >
> > > > time psql -f 188.tpcds.88.sql
> > > > SET
> > > > Timing is on.
> > > > SET
> > > > Time: 0.157 ms
> > > > h8_30_to_9 | h9_to_9_30 | h9_30_to_10 | h10_to_10_30 | h10_30_to_11
> |
> > > > h11_to_11_30 | h11_30_to_12 | h12_to_12_30
> > > > ------------+------------+-------------+--------------+-----
> > > > ---------+--------------+--------------+--------------
> > > > 16279055 | 32496701 | 32493080 | 48732586 | 48782652
> |
> > > > 28460584 | 28453299 | 32518016
> > > > (1 row)
> > > >
> > > > Time: 259695.969 ms
> > > >
> > > > real 4m19.706s
> > > > user 0m0.001s
> > > > sys 0m0.003s
> > > >
> > > > Next:
> > > > set hawq_rm_nvseg_perquery_perseg_limit=14;
> > > >
> > > > time psql -f 188.tpcds.88.sql
> > > > SET
> > > > Timing is on.
> > > > SET
> > > > Time: 0.171 ms
> > > > psql:188.tpcds.88.sql:95: ERROR: Error dispatching to seg25
> > > > ip-172-21-13-189.ec2.internal:40000: connection pointer is NULL
> > > > DETAIL: Master unable to connect to seg25
> > > > ip-172-21-13-189.ec2.internal:40000: FATAL: unrecognized
> > configuration
> > > > parameter "ONETARY"
> > > >
> > > > real 0m8.787s
> > > > user 0m0.003s
> > > > sys 0m0.002s
> > > >
> > > >
> > > > Jon Roberts
> > > >
> > >
> >
>
Re: unrecognized configuration parameter "ONETARY"
Posted by Jon Roberts <jr...@pivotal.io>.
The segment actually has this error:
2016-12-14 13:47:34.760839
UTC,"gpadmin","gpadmin",p737499,th542214432,"172.21.13.196","40327",2016-12-14
13:47:34 UTC,0,con23798,,seg-10000,,,,,"FATAL","42704","unrecognized
configuration parameter ""ONETARY""",,,,,,,0,,"guc.c",10006,
This made me check out the configs because the error was raised from
guc.c. I found that /usr/local/hawq/etc/_mgmt_config has "onetary" text:
hawq_lc_monetary=en_US.utf8
I'm using nodes with 24 drives so I have 24 temp directories set for both
the master and the segments. Thinking that possibly the problem is related
to the number of vSegs * the size of the config file exceeding a variable
size, I decided to reduce the temp directories down to just two for both
the master and segments.
After restarting HAWQ, I could get the query to use 16 and even 24 vSegs
without a problem.
So maybe the guc logic needs revisiting to make sure there isn't an issue
with parsing the GUCs when there are many vSegs and many temp directories.
I would think this would be a problem even with the default number of vSegs
per host on really large clusters.
Jon Roberts
Principal Engineer | jroberts@pivotal.io | 615-426-8661
On Mon, Dec 19, 2016 at 4:44 AM, Yi Jin <yj...@pivotal.io> wrote:
> Hi Jon,
>
> I think there is no ONTARY configuration item, so maybe I need the full log
> containing that error to check the error routine.
>
> Best,
> Yi
>
> On Mon, Dec 19, 2016 at 5:39 PM, Paul Guo <pa...@gmail.com> wrote:
>
> > You could grep the log to see whether there is "ONETARY" setting before
> > this error occurs, and also grep configuration files and related test
> files
> > to find who tried to set this.
> >
> > 2016-12-14 22:36 GMT+08:00 Jon Roberts <jr...@pivotal.io>:
> >
> > > I'm getting the error message: unrecognized configuration parameter
> > > "ONETARY" with a few of the TPC-DS queries where I'm increasing the
> > number
> > > of vsegs to get better performance. The error message alone is
> confusing
> > > so even if I am doing something wrong, the message should be improved.
> > >
> > > My environment:
> > >
> > > AWS d2.8xlarge nodes
> > > - 24 2 TB disks
> > > - 252 GB RAM
> > > - 36 cores
> > > - Centos 6
> > > - 10 nodes
> > > - 1 admin node
> > > - 10GB network
> > > - 7 TB of data
> > > - Standard TPC-DS Queries
> > > - hawq_rm_memory_limit_perseg = 200gb
> > > - hawq_rm_stmt_vseg_memory = 16gb
> > > - Random distribution on all tables
> > >
> > > I'm tried reducing the statement memory but that doesn't change
> anything.
> > >
> > > Query 88 is a good example of this because it fails quickly.
> > >
> > > set hawq_rm_nvseg_perquery_perseg_limit=12;
> > >
> > > time psql -f 188.tpcds.88.sql
> > > SET
> > > Timing is on.
> > > SET
> > > Time: 0.157 ms
> > > h8_30_to_9 | h9_to_9_30 | h9_30_to_10 | h10_to_10_30 | h10_30_to_11 |
> > > h11_to_11_30 | h11_30_to_12 | h12_to_12_30
> > > ------------+------------+-------------+--------------+-----
> > > ---------+--------------+--------------+--------------
> > > 16279055 | 32496701 | 32493080 | 48732586 | 48782652 |
> > > 28460584 | 28453299 | 32518016
> > > (1 row)
> > >
> > > Time: 259695.969 ms
> > >
> > > real 4m19.706s
> > > user 0m0.001s
> > > sys 0m0.003s
> > >
> > > Next:
> > > set hawq_rm_nvseg_perquery_perseg_limit=14;
> > >
> > > time psql -f 188.tpcds.88.sql
> > > SET
> > > Timing is on.
> > > SET
> > > Time: 0.171 ms
> > > psql:188.tpcds.88.sql:95: ERROR: Error dispatching to seg25
> > > ip-172-21-13-189.ec2.internal:40000: connection pointer is NULL
> > > DETAIL: Master unable to connect to seg25
> > > ip-172-21-13-189.ec2.internal:40000: FATAL: unrecognized
> configuration
> > > parameter "ONETARY"
> > >
> > > real 0m8.787s
> > > user 0m0.003s
> > > sys 0m0.002s
> > >
> > >
> > > Jon Roberts
> > >
> >
>
Re: unrecognized configuration parameter "ONETARY"
Posted by Yi Jin <yj...@pivotal.io>.
Hi Jon,
I think there is no ONTARY configuration item, so maybe I need the full log
containing that error to check the error routine.
Best,
Yi
On Mon, Dec 19, 2016 at 5:39 PM, Paul Guo <pa...@gmail.com> wrote:
> You could grep the log to see whether there is "ONETARY" setting before
> this error occurs, and also grep configuration files and related test files
> to find who tried to set this.
>
> 2016-12-14 22:36 GMT+08:00 Jon Roberts <jr...@pivotal.io>:
>
> > I'm getting the error message: unrecognized configuration parameter
> > "ONETARY" with a few of the TPC-DS queries where I'm increasing the
> number
> > of vsegs to get better performance. The error message alone is confusing
> > so even if I am doing something wrong, the message should be improved.
> >
> > My environment:
> >
> > AWS d2.8xlarge nodes
> > - 24 2 TB disks
> > - 252 GB RAM
> > - 36 cores
> > - Centos 6
> > - 10 nodes
> > - 1 admin node
> > - 10GB network
> > - 7 TB of data
> > - Standard TPC-DS Queries
> > - hawq_rm_memory_limit_perseg = 200gb
> > - hawq_rm_stmt_vseg_memory = 16gb
> > - Random distribution on all tables
> >
> > I'm tried reducing the statement memory but that doesn't change anything.
> >
> > Query 88 is a good example of this because it fails quickly.
> >
> > set hawq_rm_nvseg_perquery_perseg_limit=12;
> >
> > time psql -f 188.tpcds.88.sql
> > SET
> > Timing is on.
> > SET
> > Time: 0.157 ms
> > h8_30_to_9 | h9_to_9_30 | h9_30_to_10 | h10_to_10_30 | h10_30_to_11 |
> > h11_to_11_30 | h11_30_to_12 | h12_to_12_30
> > ------------+------------+-------------+--------------+-----
> > ---------+--------------+--------------+--------------
> > 16279055 | 32496701 | 32493080 | 48732586 | 48782652 |
> > 28460584 | 28453299 | 32518016
> > (1 row)
> >
> > Time: 259695.969 ms
> >
> > real 4m19.706s
> > user 0m0.001s
> > sys 0m0.003s
> >
> > Next:
> > set hawq_rm_nvseg_perquery_perseg_limit=14;
> >
> > time psql -f 188.tpcds.88.sql
> > SET
> > Timing is on.
> > SET
> > Time: 0.171 ms
> > psql:188.tpcds.88.sql:95: ERROR: Error dispatching to seg25
> > ip-172-21-13-189.ec2.internal:40000: connection pointer is NULL
> > DETAIL: Master unable to connect to seg25
> > ip-172-21-13-189.ec2.internal:40000: FATAL: unrecognized configuration
> > parameter "ONETARY"
> >
> > real 0m8.787s
> > user 0m0.003s
> > sys 0m0.002s
> >
> >
> > Jon Roberts
> >
>
Re: unrecognized configuration parameter "ONETARY"
Posted by Paul Guo <pa...@gmail.com>.
You could grep the log to see whether there is "ONETARY" setting before
this error occurs, and also grep configuration files and related test files
to find who tried to set this.
2016-12-14 22:36 GMT+08:00 Jon Roberts <jr...@pivotal.io>:
> I'm getting the error message: unrecognized configuration parameter
> "ONETARY" with a few of the TPC-DS queries where I'm increasing the number
> of vsegs to get better performance. The error message alone is confusing
> so even if I am doing something wrong, the message should be improved.
>
> My environment:
>
> AWS d2.8xlarge nodes
> - 24 2 TB disks
> - 252 GB RAM
> - 36 cores
> - Centos 6
> - 10 nodes
> - 1 admin node
> - 10GB network
> - 7 TB of data
> - Standard TPC-DS Queries
> - hawq_rm_memory_limit_perseg = 200gb
> - hawq_rm_stmt_vseg_memory = 16gb
> - Random distribution on all tables
>
> I'm tried reducing the statement memory but that doesn't change anything.
>
> Query 88 is a good example of this because it fails quickly.
>
> set hawq_rm_nvseg_perquery_perseg_limit=12;
>
> time psql -f 188.tpcds.88.sql
> SET
> Timing is on.
> SET
> Time: 0.157 ms
> h8_30_to_9 | h9_to_9_30 | h9_30_to_10 | h10_to_10_30 | h10_30_to_11 |
> h11_to_11_30 | h11_30_to_12 | h12_to_12_30
> ------------+------------+-------------+--------------+-----
> ---------+--------------+--------------+--------------
> 16279055 | 32496701 | 32493080 | 48732586 | 48782652 |
> 28460584 | 28453299 | 32518016
> (1 row)
>
> Time: 259695.969 ms
>
> real 4m19.706s
> user 0m0.001s
> sys 0m0.003s
>
> Next:
> set hawq_rm_nvseg_perquery_perseg_limit=14;
>
> time psql -f 188.tpcds.88.sql
> SET
> Timing is on.
> SET
> Time: 0.171 ms
> psql:188.tpcds.88.sql:95: ERROR: Error dispatching to seg25
> ip-172-21-13-189.ec2.internal:40000: connection pointer is NULL
> DETAIL: Master unable to connect to seg25
> ip-172-21-13-189.ec2.internal:40000: FATAL: unrecognized configuration
> parameter "ONETARY"
>
> real 0m8.787s
> user 0m0.003s
> sys 0m0.002s
>
>
> Jon Roberts
>