You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hawq.apache.org by "Chunling Wang (JIRA)" <ji...@apache.org> on 2016/03/21 08:49:25 UTC

[jira] [Commented] (HAWQ-564) QD hangs when connecting to resource manager

    [ https://issues.apache.org/jira/browse/HAWQ-564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15203864#comment-15203864 ] 

Chunling Wang commented on HAWQ-564:
------------------------------------

There is another way to cause this bug without fault injection.
1. First run query and get some QEs.
{code}
dispatch=# select count(*) from test_dispatch as t1, test_dispatch as t2, test_dispatch as t3 where t1.id *2 = t2.id and t1.id < t3.id;
 count
-------
  3725
(1 row)
{code}

{code}
$ ps -ef|grep postgres
  501 30190     1   0  2:34下午 ??         0:00.31 /usr/local/hawq/bin/postgres -D /Users/wangchunling/hawq-data-directory/masterdd -i -M master -p 5432 --silent-mode=true
  501 30191 30190   0  2:34下午 ??         0:00.01 postgres: port  5432, master logger process
  501 30194 30190   0  2:34下午 ??         0:00.00 postgres: port  5432, stats collector process
  501 30195 30190   0  2:34下午 ??         0:00.01 postgres: port  5432, writer process
  501 30196 30190   0  2:34下午 ??         0:00.00 postgres: port  5432, checkpoint process
  501 30197 30190   0  2:34下午 ??         0:00.00 postgres: port  5432, seqserver process
  501 30198 30190   0  2:34下午 ??         0:00.00 postgres: port  5432, WAL Send Server process
  501 30199 30190   0  2:34下午 ??         0:00.00 postgres: port  5432, DFS Metadata Cache process
  501 30200 30190   0  2:34下午 ??         0:00.07 postgres: port  5432, master resource manager
  501 30216     1   0  2:34下午 ??         0:00.37 /usr/local/hawq/bin/postgres -D /Users/wangchunling/hawq-data-directory/segmentdd -i -M segment -p 40000 --silent-mode=true
  501 30217 30216   0  2:34下午 ??         0:00.02 postgres: port 40000, logger process
  501 30220 30216   0  2:34下午 ??         0:00.00 postgres: port 40000, stats collector process
  501 30221 30216   0  2:34下午 ??         0:00.01 postgres: port 40000, writer process
  501 30222 30216   0  2:34下午 ??         0:00.00 postgres: port 40000, checkpoint process
  501 30223 30216   0  2:34下午 ??         0:00.03 postgres: port 40000, segment resource manager
  501 30231 30190   0  2:35下午 ??         0:00.03 postgres: port  5432, wangchunling dispatch [local] con12 cmd6 idle [local]
  501 30235 30216   0  2:35下午 ??         0:00.13 postgres: port 40000, wangchunling dispatch 127.0.0.1(65051) con12 seg0 idle
  501 30239 30216   0  2:35下午 ??         0:00.06 postgres: port 40000, wangchunling dispatch 127.0.0.1(65061) con12 seg0 idle
  501 30240 30216   0  2:35下午 ??         0:00.06 postgres: port 40000, wangchunling dispatch 127.0.0.1(65063) con12 seg0 idle
  501 30242 99560   0  2:36下午 ttys000    0:00.00 grep postgres
{code}

2. Kill some QE and there is no QE.
{code}
$ kill -9 30235
$ ps -ef|grep postgres
  501 30190     1   0  2:34下午 ??         0:00.32 /usr/local/hawq/bin/postgres -D /Users/wangchunling/hawq-data-directory/masterdd -i -M master -p 5432 --silent-mode=true
  501 30191 30190   0  2:34下午 ??         0:00.01 postgres: port  5432, master logger process
  501 30194 30190   0  2:34下午 ??         0:00.00 postgres: port  5432, stats collector process
  501 30195 30190   0  2:34下午 ??         0:00.01 postgres: port  5432, writer process
  501 30196 30190   0  2:34下午 ??         0:00.00 postgres: port  5432, checkpoint process
  501 30197 30190   0  2:34下午 ??         0:00.00 postgres: port  5432, seqserver process
  501 30198 30190   0  2:34下午 ??         0:00.00 postgres: port  5432, WAL Send Server process
  501 30199 30190   0  2:34下午 ??         0:00.00 postgres: port  5432, DFS Metadata Cache process
  501 30200 30190   0  2:34下午 ??         0:00.08 postgres: port  5432, master resource manager
  501 30216     1   0  2:34下午 ??         0:00.58 /usr/local/hawq/bin/postgres -D /Users/wangchunling/hawq-data-directory/segmentdd -i -M segment -p 40000 --silent-mode=true
  501 30217 30216   0  2:34下午 ??         0:00.03 postgres: port 40000, logger process
  501 30231 30190   0  2:35下午 ??         0:00.04 postgres: port  5432, wangchunling dispatch [local] con12 cmd6 idle [local]
  501 30248 30216   0  2:36下午 ??         0:00.00 postgres: port 40000, stats collector process
  501 30249 30216   0  2:36下午 ??         0:00.00 postgres: port 40000, writer process
  501 30250 30216   0  2:36下午 ??         0:00.00 postgres: port 40000, checkpoint process
  501 30251 30216   0  2:36下午 ??         0:00.00 postgres: port 40000, segment resource manager
  501 30255 99560   0  2:36下午 ttys000    0:00.00 grep postgres
{code}
3. Run query again and get some new QEs.
{code}
dispatch=# select count(*) from test_dispatch as t1, test_dispatch as t2, test_dispatch as t3 where t1.id *2 = t2.id and t1.id < t3.id;
 count
-------
  3725
(1 row)
{code}

{code}
$ ps -ef|grep postgres
  501 30190     1   0  2:34下午 ??         0:00.33 /usr/local/hawq/bin/postgres -D /Users/wangchunling/hawq-data-directory/masterdd -i -M master -p 5432 --silent-mode=true
  501 30191 30190   0  2:34下午 ??         0:00.01 postgres: port  5432, master logger process
  501 30194 30190   0  2:34下午 ??         0:00.00 postgres: port  5432, stats collector process
  501 30195 30190   0  2:34下午 ??         0:00.02 postgres: port  5432, writer process
  501 30196 30190   0  2:34下午 ??         0:00.00 postgres: port  5432, checkpoint process
  501 30197 30190   0  2:34下午 ??         0:00.00 postgres: port  5432, seqserver process
  501 30198 30190   0  2:34下午 ??         0:00.00 postgres: port  5432, WAL Send Server process
  501 30199 30190   0  2:34下午 ??         0:00.00 postgres: port  5432, DFS Metadata Cache process
  501 30200 30190   0  2:34下午 ??         0:00.11 postgres: port  5432, master resource manager
  501 30216     1   0  2:34下午 ??         0:00.61 /usr/local/hawq/bin/postgres -D /Users/wangchunling/hawq-data-directory/segmentdd -i -M segment -p 40000 --silent-mode=true
  501 30217 30216   0  2:34下午 ??         0:00.05 postgres: port 40000, logger process
  501 30231 30190   0  2:35下午 ??         0:00.05 postgres: port  5432, wangchunling dispatch [local] con12 cmd9 idle [local]
  501 30248 30216   0  2:36下午 ??         0:00.00 postgres: port 40000, stats collector process
  501 30249 30216   0  2:36下午 ??         0:00.01 postgres: port 40000, writer process
  501 30250 30216   0  2:36下午 ??         0:00.00 postgres: port 40000, checkpoint process
  501 30251 30216   0  2:36下午 ??         0:00.01 postgres: port 40000, segment resource manager
  501 30262 30216   0  2:36下午 ??         0:00.08 postgres: port 40000, wangchunling dispatch 127.0.0.1(65092) con12 seg0 idle
  501 30263 30216   0  2:36下午 ??         0:00.06 postgres: port 40000, wangchunling dispatch 127.0.0.1(65094) con12 seg0 idle
  501 30264 30216   0  2:36下午 ??         0:00.06 postgres: port 40000, wangchunling dispatch 127.0.0.1(65096) con12 seg0 idle
  501 30267 99560   0  2:36下午 ttys000    0:00.00 grep postgres
{code}
4. Kill some QE again.
{code}
$ kill -9 30264
$ ps -ef|grep postgres
  501 30190     1   0  2:34下午 ??         0:00.35 /usr/local/hawq/bin/postgres -D /Users/wangchunling/hawq-data-directory/masterdd -i -M master -p 5432 --silent-mode=true
  501 30191 30190   0  2:34下午 ??         0:00.02 postgres: port  5432, master logger process
  501 30194 30190   0  2:34下午 ??         0:00.00 postgres: port  5432, stats collector process
  501 30195 30190   0  2:34下午 ??         0:00.03 postgres: port  5432, writer process
  501 30196 30190   0  2:34下午 ??         0:00.00 postgres: port  5432, checkpoint process
  501 30197 30190   0  2:34下午 ??         0:00.00 postgres: port  5432, seqserver process
  501 30198 30190   0  2:34下午 ??         0:00.01 postgres: port  5432, WAL Send Server process
  501 30199 30190   0  2:34下午 ??         0:00.00 postgres: port  5432, DFS Metadata Cache process
  501 30200 30190   0  2:34下午 ??         0:00.20 postgres: port  5432, master resource manager
  501 30216     1   0  2:34下午 ??         0:00.87 /usr/local/hawq/bin/postgres -D /Users/wangchunling/hawq-data-directory/segmentdd -i -M segment -p 40000 --silent-mode=true
  501 30217 30216   0  2:34下午 ??         0:00.07 postgres: port 40000, logger process
  501 30231 30190   0  2:35下午 ??         0:00.06 postgres: port  5432, wangchunling dispatch [local] con12 cmd11 idle [local]
  501 30280 30216   0  2:38下午 ??         0:00.00 postgres: port 40000, stats collector process
  501 30281 30216   0  2:38下午 ??         0:00.00 postgres: port 40000, writer process
  501 30282 30216   0  2:38下午 ??         0:00.00 postgres: port 40000, checkpoint process
  501 30283 30216   0  2:38下午 ??         0:00.00 postgres: port 40000, segment resource manager
  501 30287 99560   0  2:38下午 ttys000    0:00.00 grep postgres
{code}
5. Run query and find QD hangs.
{code}
dispatch=# select count(*) from test_dispatch as t1, test_dispatch as t2, test_dispatch as t3 where t1.id *2 = t2.id and t1.id < t3.id;

{code}
6. And there is still no QEs.
{code}
$ ps -ef|grep postgres
  501 30190     1   0  2:34下午 ??         0:00.39 /usr/local/hawq/bin/postgres -D /Users/wangchunling/hawq-data-directory/masterdd -i -M master -p 5432 --silent-mode=true
  501 30191 30190   0  2:34下午 ??         0:00.04 postgres: port  5432, master logger process
  501 30194 30190   0  2:34下午 ??         0:00.01 postgres: port  5432, stats collector process
  501 30195 30190   0  2:34下午 ??         0:00.05 postgres: port  5432, writer process
  501 30196 30190   0  2:34下午 ??         0:00.01 postgres: port  5432, checkpoint process
  501 30197 30190   0  2:34下午 ??         0:00.00 postgres: port  5432, seqserver process
  501 30198 30190   0  2:34下午 ??         0:00.01 postgres: port  5432, WAL Send Server process
  501 30199 30190   0  2:34下午 ??         0:00.00 postgres: port  5432, DFS Metadata Cache process
  501 30200 30190   0  2:34下午 ??         0:00.32 postgres: port  5432, master resource manager
  501 30216     1   0  2:34下午 ??         0:01.01 /usr/local/hawq/bin/postgres -D /Users/wangchunling/hawq-data-directory/segmentdd -i -M segment -p 40000 --silent-mode=true
  501 30217 30216   0  2:34下午 ??         0:00.07 postgres: port 40000, logger process
  501 30231 30190   0  2:35下午 ??         0:00.10 postgres: port  5432, wangchunling dispatch [local] con12 cmd14 SELECT [local]
  501 30280 30216   0  2:38下午 ??         0:00.00 postgres: port 40000, stats collector process
  501 30281 30216   0  2:38下午 ??         0:00.02 postgres: port 40000, writer process
  501 30282 30216   0  2:38下午 ??         0:00.00 postgres: port 40000, checkpoint process
  501 30283 30216   0  2:38下午 ??         0:00.04 postgres: port 40000, segment resource manager
  501 30307 99560   0  2:40下午 ttys000    0:00.00 grep postgres
{code}

> QD hangs when connecting to resource manager
> --------------------------------------------
>
>                 Key: HAWQ-564
>                 URL: https://issues.apache.org/jira/browse/HAWQ-564
>             Project: Apache HAWQ
>          Issue Type: Bug
>          Components: Resource Manager
>    Affects Versions: 2.0.0
>            Reporter: Chunling Wang
>            Assignee: Lei Chang
>
> When first inject panic in QE process, we run a query and segment is down. After the segment is up, we run another query and get correct answer. Then we inject the same panic second time. After the segment is down and then up again, we run a query and find QD process hangs when connecting to resource manager. Here is the backtrace when QD hangs:
> {code}
> * thread #1: tid = 0x21d8be, 0x00007fff890355be libsystem_kernel.dylib`poll + 10, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
>   * frame #0: 0x00007fff890355be libsystem_kernel.dylib`poll + 10
>     frame #1: 0x0000000101daeafe postgres`processAllCommFileDescs + 158 at rmcomm_AsyncComm.c:156
>     frame #2: 0x0000000101db85f5 postgres`callSyncRPCRemote(hostname=0x00007f9c19e00cd0, port=5437, sendbuff=0x00007f9c1b918f50, sendbuffsize=80, sendmsgid=259, exprecvmsgid=2307, recvsmb=<unavailable>, errorbuf=0x000000010230c1a0, errorbufsize=<unavailable>) + 645 at rmcomm_SyncComm.c:122
>     frame #3: 0x0000000101db2d85 postgres`acquireResourceFromRM [inlined] callSyncRPCToRM(sendbuff=0x00007f9c1b918f50, sendbuffsize=<unavailable>, sendmsgid=259, exprecvmsgid=2307, recvsmb=0x00007f9c1b918e70, errorbuf=<unavailable>, errorbufsize=1024) + 73 at rmcomm_QD2RM.c:2780
>     frame #4: 0x0000000101db2d3c postgres`acquireResourceFromRM(index=<unavailable>, sessionid=12, slice_size=462524016, iobytes=134217728, preferred_nodes=0x00007f9c1a02d398, preferred_nodes_size=<unavailable>, max_seg_count_fix=<unavailable>, min_seg_count_fix=<unavailable>, errorbuf=<unavailable>, errorbufsize=<unavailable>) + 572 at rmcomm_QD2RM.c:742
>     frame #5: 0x0000000101c979e7 postgres`AllocateResource(life=QRL_ONCE, slice_size=5, iobytes=134217728, max_target_segment_num=1, min_target_segment_num=1, vol_info=0x00007f9c1a02d398, vol_info_size=1) + 631 at pquery.c:796
>     frame #6: 0x0000000101e8c60f postgres`calculate_planner_segment_num(query=<unavailable>, resourceLife=QRL_ONCE, fullRangeTable=<unavailable>, intoPolicy=<unavailable>, sliceNum=5) + 14287 at cdbdatalocality.c:4207
>     frame #7: 0x0000000101c0f671 postgres`planner + 106 at planner.c:496
>     frame #8: 0x0000000101c0f607 postgres`planner(parse=0x00007f9c1a02a140, cursorOptions=<unavailable>, boundParams=0x0000000000000000, resourceLife=QRL_ONCE) + 311 at planner.c:310
>     frame #9: 0x0000000101c8eb33 postgres`pg_plan_query(querytree=0x00007f9c1a02a140, boundParams=0x0000000000000000, resource_life=QRL_ONCE) + 99 at postgres.c:837
>     frame #10: 0x0000000101c956ae postgres`exec_simple_query + 21 at postgres.c:911
>     frame #11: 0x0000000101c95699 postgres`exec_simple_query(query_string=0x00007f9c1a028a30, seqServerHost=0x0000000000000000, seqServerPort=-1) + 1577 at postgres.c:1671
>     frame #12: 0x0000000101c91a4c postgres`PostgresMain(argc=<unavailable>, argv=<unavailable>, username=0x00007f9c1b808cf0) + 9404 at postgres.c:4754
>     frame #13: 0x0000000101c4ae02 postgres`ServerLoop [inlined] BackendRun + 105 at postmaster.c:5889
>     frame #14: 0x0000000101c4ad99 postgres`ServerLoop at postmaster.c:5484
>     frame #15: 0x0000000101c4ad99 postgres`ServerLoop + 9593 at postmaster.c:2163
>     frame #16: 0x0000000101c47d3b postgres`PostmasterMain(argc=<unavailable>, argv=<unavailable>) + 5019 at postmaster.c:1454
>     frame #17: 0x0000000101bb1aa9 postgres`main(argc=9, argv=0x00007f9c19c1eef0) + 1433 at main.c:209
>     frame #18: 0x00007fff95e8c5c9 libdyld.dylib`start + 1
>   thread #2: tid = 0x21d8bf, 0x00007fff890355be libsystem_kernel.dylib`poll + 10
>     frame #0: 0x00007fff890355be libsystem_kernel.dylib`poll + 10
>     frame #1: 0x0000000101dfe723 postgres`rxThreadFunc(arg=<unavailable>) + 2163 at ic_udp.c:6251
>     frame #2: 0x00007fff95e822fc libsystem_pthread.dylib`_pthread_body + 131
>     frame #3: 0x00007fff95e82279 libsystem_pthread.dylib`_pthread_start + 176
>     frame #4: 0x00007fff95e804b1 libsystem_pthread.dylib`thread_start + 13
>   thread #3: tid = 0x21d9c2, 0x00007fff890343f6 libsystem_kernel.dylib`__select + 10
>     frame #0: 0x00007fff890343f6 libsystem_kernel.dylib`__select + 10
>     frame #1: 0x0000000101e9d42e postgres`pg_usleep(microsec=<unavailable>) + 78 at pgsleep.c:43
>     frame #2: 0x0000000101db1a66 postgres`generateResourceRefreshHeartBeat(arg=0x00007f9c19f02480) + 166 at rmcomm_QD2RM.c:1519
>     frame #3: 0x00007fff95e822fc libsystem_pthread.dylib`_pthread_body + 131
>     frame #4: 0x00007fff95e82279 libsystem_pthread.dylib`_pthread_start + 176
>     frame #5: 0x00007fff95e804b1 libsystem_pthread.dylib`thread_start + 13
> {code}
> And here is the operations:
> 1. Before injection, get query answer correctly.
> {code}
> dispatch=# select count(*) from test_dispatch as t1, test_dispatch as t2, test_dispatch as t3 where t1.id *2 = t2.id and t1.id < t3.id;
>  count
> -------
>   3725
> (1 row)
> {code}
> 2. Inject panic, fault triggered, and segment is down.
> {code}
> dispatch=# select count(*) from test_dispatch as t1, test_dispatch as t2, test_dispatch as t3 where t1.id *2 = t2.id and t1.id < t3.id;
> ERROR:  fault triggered, fault name:'fail_qe_when_do_query' fault type:'panic' (faultinjector.c:656)  (seg0 localhost:40000 pid=26936)
> dispatch=# select count(*) from test_dispatch as t1, test_dispatch as t2, test_dispatch as t3 where t1.id *2 = t2.id and t1.id < t3.id;
> ERROR:  failed to acquire resource from resource manager, 1 of 1 segments is unavailable (pquery.c:807)
> {code}
> 3. After a while and when segment is up, get correct answer.
> {code}
> dispatch=# select count(*) from test_dispatch as t1, test_dispatch as t2, test_dispatch as t3 where t1.id *2 = t2.id and t1.id < t3.id;
>  count
> -------
>   3725
> (1 row)
> {code}
> 4. Inject again, fault triggered, and segment is down.
> {code}
> dispatch=# select count(*) from test_dispatch as t1, test_dispatch as t2, test_dispatch as t3 where t1.id *2 = t2.id and t1.id < t3.id;
> ERROR:  fault triggered, fault name:'fail_qe_when_do_query' fault type:'panic' (faultinjector.c:656)  (seg0 localhost:40000 pid=26994)
> dispatch=# select count(*) from test_dispatch as t1, test_dispatch as t2, test_dispatch as t3 where t1.id *2 = t2.id and t1.id < t3.id;
> ERROR:  failed to acquire resource from resource manager, 1 of 1 segments is unavailable (pquery.c:807)
> {code}
> 5. After a while, run query and find QD hangs.
> {code}
> dispatch=# select count(*) from test_dispatch as t1, test_dispatch as t2, test_dispatch as t3 where t1.id *2 = t2.id and t1.id < t3.id;
> {code}
> 6. Open another terminal, find segment is already up.
> {code}
> dispatch=# select * from gp_segment_configuration;
>  registration_order | role | status | port  |          hostname           |           address           | description
> --------------------+------+--------+-------+-----------------------------+-----------------------------+-------------
>                   0 | m    | u      |  5432 | ChunlingdeMacBook-Pro.local | ChunlingdeMacBook-Pro.local |
>                   1 | p    | u      | 40000 | localhost                   | 127.0.0.1                   |
> (2 rows)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)