You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "George Huang (Jira)" <ji...@apache.org> on 2021/05/11 15:26:00 UTC
[jira] [Created] (HDDS-5216) ozone freon randomkeys failed after
leader SCM node is down
George Huang created HDDS-5216:
----------------------------------
Summary: ozone freon randomkeys failed after leader SCM node is down
Key: HDDS-5216
URL: https://issues.apache.org/jira/browse/HDDS-5216
Project: Apache Ozone
Issue Type: Bug
Components: SCM HA
Affects Versions: 1.2.0
Reporter: George Huang
under .../compose/ozone-ha, create a HA cluster:
docker-compose up -d --scale datanode=3
Initial SCM roles as following:
bash-4.2$ ozone admin scm roles
[scm1:9865:FOLLOWER, scm2:9865:FOLLOWER, scm3:9865:LEADER]
Running freon random key generator as following:
ozone freon randomkeys --numOfVolumes=10 --numOfBuckets 50 --numOfKeys 50 --replicationType=RATIS --factor=THREE
While freon randomkeys was running, put all SCM nodes under blockade and stop leader SCM node:
blockade status:
NODE CONTAINER ID STATUS IP NETWORK PARTITION
18f9c1e2d52f UP 172.31.0.9 NORMAL
ozone-ha_scm1_1
25c74f0a9271 UP 172.31.0.6 NORMAL
ozone-ha_scm2_1
8808d10ccb3a DOWN UNKNOWN
freon randomkeys failed with following error message:
Some test result msg as following:
6:00:30,131 [pool-2-thread-3] ERROR freon.RandomKeyGenerator: Exception while adding key: key-21-80493 in bucket: bucket-44-63818 of volume: vol-1-95998.
INTERNAL_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: No Route to Host from om1/172.31.0.11 to scm3:9863 failed on socket timeout exception: java.net.NoRouteToHostException: No route to host; For more details see: [http://wiki.apache.org/hadoop/NoRouteToHost]
at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:604)
at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.openKey(OzoneManagerProtocolClientSideTranslatorPB.java:595)
at org.apache.hadoop.ozone.client.rpc.RpcClient.createKey(RpcClient.java:756)
at org.apache.hadoop.ozone.client.OzoneBucket.createKey(OzoneBucket.java:502)
at org.apache.hadoop.ozone.freon.RandomKeyGenerator.createKey(RandomKeyGenerator.java:703)
at org.apache.hadoop.ozone.freon.RandomKeyGenerator.access$1100(RandomKeyGenerator.java:86)
at org.apache.hadoop.ozone.freon.RandomKeyGenerator$ObjectCreator.run(RandomKeyGenerator.java:621)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
44.10% |????????????????????????????????????????????? | 11024/25000 Time: 0:09:002021-05-11 06:00:37,231 [pool-2-thread-7] INFO metrics.RatisMetrics: Creating Metrics Registry : ratis.client_message_metrics.client-EA7B54107DBD->4c51bca8-cc0a-4c20-84dd-b5a7cb18c4ac
2021-05-11 06:00:37,231 [pool-2-thread-7] WARN impl.MetricRegistriesImpl: First MetricRegistry has been created without registering reporters. You may need to call MetricRegistries.global().addReporterRegistration(...) before.
100.00% |?????????????????????????????????????????????????????????????????????????????????????????????????????| 25000/25000 Time: 0:15:39
INTERNAL_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: No Route to Host from om1/172.31.0.11 to scm3:9863 failed on socket timeout exception: java.net.NoRouteToHostException: No route to host; For more details see: [http://wiki.apache.org/hadoop/NoRouteToHost]
***************************************************
Status: Failed
Git Base Revision: 7a3bc90b05f257c8ace2f76d74264906f0f7a932
Number of Volumes created: 10
Number of Buckets created: 500
Number of Keys added: 24991
Ratis replication factor: THREE
Ratis replication type: RATIS
Average Time spent in volume creation: 00:00:00,114
Average Time spent in bucket creation: 00:00:01,263
Average Time spent in key creation: 00:02:48,698
Average Time spent in key write: 00:00:04,216
Total bytes written: 255907840
Total Execution time: 00:15:39,968
***************************************************
In this case, I'd expect the freon test would still finish successfully.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org