[jira] [Commented] (HAWQ-984) hawq config is too slow.

["Paul Guo (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HAWQ-984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15438486#comment-15438486 ] 

Paul Guo commented on HAWQ-984:
-------------------------------

1) gpscp
It still uses popen2.Popen4(). This function introduces millions of useless close() sys call finally in above test command. Using subprocess.Popen()  without close_fds as an alternative easily resolves this.

2) gppylib/commands/base.py

    def __init__(self,name,pool,timeout=5):

The worker thread will block at most 5 seconds in each loop (Queue.get()) to fetch potential commands even we have known that there will be no more commands to run for some threads. This really does not make sense since some idle threads will block for 5 seconds also before exiting.

Setting timeout to zero will make python code spin. I tested a small timeout value e.g. 0.1s and it works fine. It seems that  0.1 is a good timeout candidate.

> hawq config is too slow.
> ------------------------
>
>                 Key: HAWQ-984
>                 URL: https://issues.apache.org/jira/browse/HAWQ-984
>             Project: Apache HAWQ
>          Issue Type: Bug
>          Components: Command Line Tools
>            Reporter: Paul Guo
>            Assignee: Lei Chang
>             Fix For: 2.0.1.0-incubating
>
>
> I tried to set a simple guc value via "hawq config" on my centos
> virtual system, but it spends >6 seconds. I know "hawq config" just
> simple scp-update xml files to various nodes. This should have
> been very fast if network situation is fine (My test hawq system is all-in-one
> test environment so network latency is not an issue.)
> I expect this is done on my test system (just one seg node) in < 1 second.
> But actually the time is:
> $ time hawq config -c lc_messages -v en_US.UTF-8
> GUC lc_messages already exist in hawq-site.xml
> Update it with value: en_US.UTF-8
> GUC		: lc_messages
> Value		: en_US.UTF-8
> real	0m6.363s
> user	0m0.943s
> sys	0m1.258s
> I quickly monitored the processes when running "hawq config",
> it seems that the scp command finishes very early and adding
> print debug code in python file gpscp shows that the code in it
> finishes also very early.
> I'm not sure if anyone sees that long like my case (my system
> is probably quite mangled after dev and test). I roughly traced
> the python stack. I suspect there is something wrong in python
> or we are not using related module well or we should replace modules.
> I just checked "hawq config" but it is possible other CLI suffers from this also.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)