You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Enis Soztutar (JIRA)" <ji...@apache.org> on 2017/09/07 00:56:00 UTC
[jira] [Commented] (HBASE-18610) Provide capability to activate
chaos monkey
[ https://issues.apache.org/jira/browse/HBASE-18610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16156265#comment-16156265 ]
Enis Soztutar commented on HBASE-18610:
---------------------------------------
[~ted_yu] I was testing this, but again could not get it working. Two problems when I run these in the docker:
If CM is not launched, the job failed at the end:
{code}
root@securecluster:/usr/src/hbase/hbase-native-client# ./buck-out/gen/core/load-client -num_rows=100000 -appends=false -report_num_rows=1000 -num_cols=10 -conf ../conf/
W0907 00:30:47.859833 8590 hbase-configuration-loader.cc:123] Unable to open file[../conf/hbase-default.xml]
I0907 00:30:47.864672 8590 location-cache.cc:77] Connecting to ZooKeeper. Quorum:localhost:2181
2017-09-07 00:30:47,864:8590(0x7fc36a141840):ZOO_INFO@log_env@726: Client environment:zookeeper.version=zookeeper C client 3.4.8
2017-09-07 00:30:47,864:8590(0x7fc36a141840):ZOO_INFO@log_env@730: Client environment:host.name=securecluster
2017-09-07 00:30:47,864:8590(0x7fc36a141840):ZOO_INFO@log_env@737: Client environment:os.name=Linux
2017-09-07 00:30:47,864:8590(0x7fc36a141840):ZOO_INFO@log_env@738: Client environment:os.arch=3.10.0-514.2.2.el7.x86_64
2017-09-07 00:30:47,864:8590(0x7fc36a141840):ZOO_INFO@log_env@739: Client environment:os.version=#1 SMP Tue Dec 6 23:06:41 UTC 2016
2017-09-07 00:30:47,865:8590(0x7fc36a141840):ZOO_INFO@log_env@747: Client environment:user.name=(null)
2017-09-07 00:30:47,865:8590(0x7fc36a141840):ZOO_INFO@log_env@755: Client environment:user.home=/root
2017-09-07 00:30:47,865:8590(0x7fc36a141840):ZOO_INFO@log_env@767: Client environment:user.dir=/usr/src/hbase/hbase-native-client
2017-09-07 00:30:47,865:8590(0x7fc36a141840):ZOO_INFO@zookeeper_init@800: Initiating client connection, host=localhost:2181 sessionTimeout=90000 watcher=(nil) sessionId=0 sessionPasswd=<null> context=(nil) flags=0
I0907 00:30:47.866176 8590 load-client.cc:330] Sending put requests
2017-09-07 00:30:47,872:8590(0x7fc3555e8700):ZOO_INFO@check_events@1728: initiated connection to server [::1:2181]
2017-09-07 00:30:47,877:8590(0x7fc3555e8700):ZOO_INFO@check_events@1775: session establishment complete on server [::1:2181], sessionId=0x15e59be46e30006, negotiated timeout=90000
I0907 00:30:51.234663 8615 load-client.cc:231] (Thread 9) Written 1000 rows in 3361 ms.
I0907 00:30:51.250079 8608 load-client.cc:231] (Thread 2) Written 1000 rows in 3381 ms.
I0907 00:30:51.254568 8613 load-client.cc:231] (Thread 7) Written 1000 rows in 3381 ms.
I0907 00:30:51.291641 8610 load-client.cc:231] (Thread 4) Written 1000 rows in 3420 ms.
....
I0907 00:31:17.682425 8630 load-client.cc:209] (Thread 1) Sent 10000 gets in 2886 ms.
I0907 00:31:17.683099 8590 load-client.cc:396] Successful. sent multi-get requests for 100000 rows in 2888 ms.
*** Aborted at 1504744277 (unix time) try "date -d @1504744277" if you are using GNU date ***
PC: @ 0x422e85 hbase::MonkeyRunner::~MonkeyRunner()
*** SIGSEGV (@0x0) received by PID 8590 (TID 0x7fc36a141840) from PID 0; stack trace: ***
@ 0x7fc3681033d0 (unknown)
@ 0x422e85 hbase::MonkeyRunner::~MonkeyRunner()
@ 0x41fb04 main
@ 0x7fc366fc3830 __libc_start_main
@ 0x418e29 _start
@ 0x0 (unknown)
Segmentation fault (core dumped)
{code}
If I launch it with the CM tool, then it fails with:
{code}
root@securecluster:/usr/src/hbase/hbase-native-client# ./buck-out/gen/core/load-client -num_rows=100000 -appends=false -report_num_rows=1000 -num_cols=10 -conf /usr/src/hbase/conf/ -monkey serverKilling
W0907 00:42:58.259927 8838 hbase-configuration-loader.cc:123] Unable to open file[/usr/src/hbase/conf/hbase-default.xml]
I0907 00:42:58.260288 8839 mini-cluster.cc:42] got empty classpath
I0907 00:42:58.260357 8839 mini-cluster.cc:51] set clspath 0
I0907 00:42:58.271355 8838 location-cache.cc:77] Connecting to ZooKeeper. Quorum:localhost:2181
2017-09-07 00:42:58,271:8838(0x7fcc53f86840):ZOO_INFO@log_env@726: Client environment:zookeeper.version=zookeeper C client 3.4.8
2017-09-07 00:42:58,271:8838(0x7fcc53f86840):ZOO_INFO@log_env@730: Client environment:host.name=securecluster
2017-09-07 00:42:58,271:8838(0x7fcc53f86840):ZOO_INFO@log_env@737: Client environment:os.name=Linux
2017-09-07 00:42:58,271:8838(0x7fcc53f86840):ZOO_INFO@log_env@738: Client environment:os.arch=3.10.0-514.2.2.el7.x86_64
2017-09-07 00:42:58,271:8838(0x7fcc53f86840):ZOO_INFO@log_env@739: Client environment:os.version=#1 SMP Tue Dec 6 23:06:41 UTC 2016
2017-09-07 00:42:58,271:8838(0x7fcc53f86840):ZOO_INFO@log_env@747: Client environment:user.name=(null)
2017-09-07 00:42:58,271:8838(0x7fcc53f86840):ZOO_INFO@log_env@755: Client environment:user.home=/root
2017-09-07 00:42:58,271:8838(0x7fcc53f86840):ZOO_INFO@log_env@767: Client environment:user.dir=/usr/src/hbase/hbase-native-client
2017-09-07 00:42:58,271:8838(0x7fcc53f86840):ZOO_INFO@zookeeper_init@800: Initiating client connection, host=localhost:2181 sessionTimeout=90000 watcher=(nil) sessionId=0 sessionPasswd=<null> context=(nil) flags=0
I0907 00:42:58.273984 8838 load-client.cc:330] Sending put requests
2017-09-07 00:42:58,273:8838(0x7fcbf37fe700):ZOO_INFO@check_events@1728: initiated connection to server [::1:2181]
2017-09-07 00:42:58,279:8838(0x7fcbf37fe700):ZOO_INFO@check_events@1775: session establishment complete on server [::1:2181], sessionId=0x15e59be46e3000a, negotiated timeout=90000
2017-09-07 00:42:59,110 ERROR [main] util.AbstractHBaseTool (AbstractHBaseTool.java:run(114)) - Error running command-line tool
java.lang.RuntimeException: ChaosMonkeyRunner must run againt a distributed cluster, please check and point to the right configuration dir
at org.apache.hadoop.hbase.chaos.util.ChaosMonkeyRunner.setUpCluster(ChaosMonkeyRunner.java:115)
at org.apache.hadoop.hbase.chaos.util.ChaosMonkeyRunner.doWork(ChaosMonkeyRunner.java:96)
at org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:112)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.hbase.chaos.util.ChaosMonkeyRunner.main(ChaosMonkeyRunner.java:182)
{code}
This succeeds for me:
{code}
../bin/hbase org.apache.hadoop.hbase.chaos.util.ChaosMonkeyRunner -c ../conf/hbase-site.xml -m serverKilling --tableName load_test_table
{code}
> Provide capability to activate chaos monkey
> -------------------------------------------
>
> Key: HBASE-18610
> URL: https://issues.apache.org/jira/browse/HBASE-18610
> Project: HBase
> Issue Type: Sub-task
> Reporter: Ted Yu
> Attachments: 18610.v1.txt, 18610.v4.txt, 18610.v5.txt, 18610.v6.txt, 18610.v7.txt
>
>
> Currently load-client runs against a cluster where region servers are stable.
> We need to introduce chaos monkey so that wider coverage for read path is exercised.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)