You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "George Huang (Jira)" <ji...@apache.org> on 2021/05/11 17:22:00 UTC
[jira] [Commented] (HDDS-5217) freon.RandomKeyGenerator: Exception
while adding key: key-7-86181 in bucket: bucket-27-73522 of volume:
vol-1-21313. KEY_NOT_FOUND
org.apache.hadoop.ozone.om.exceptions.OMException: Failed to commit key,
... entry is not found in the OpenKey table
[ https://issues.apache.org/jira/browse/HDDS-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17342765#comment-17342765 ]
George Huang commented on HDDS-5217:
------------------------------------
scm and om logs attached.
> freon.RandomKeyGenerator: Exception while adding key: key-7-86181 in bucket: bucket-27-73522 of volume: vol-1-21313. KEY_NOT_FOUND org.apache.hadoop.ozone.om.exceptions.OMException: Failed to commit key, ... entry is not found in the OpenKey table
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-5217
> URL: https://issues.apache.org/jira/browse/HDDS-5217
> Project: Apache Ozone
> Issue Type: Bug
> Components: OM HA, SCM HA
> Affects Versions: 1.2.0
> Reporter: George Huang
> Priority: Major
> Attachments: hdds-5217.zip
>
>
> Create a cluster under ../ozone-ha
> SCM and OM roles were as following:
> bash-4.2$ ozone admin scm roles
> [scm3:9865:FOLLOWER, scm2:9865:FOLLOWER, scm1:9865:LEADER]
>
> bash-4.2$ ozone admin om roles --service-id=omservice
> om1 : FOLLOWER (om1)
> om3 : FOLLOWER (om3)
> om2 : LEADER (om2)
> Executing freon randomkeys as following:
> bash-4.2$ ozone freon randomkeys --numOfVolumes=10 --numOfBuckets 50 --numOfKeys 50 --replicationType=RATIS --factor=THREE
> Place all SCM and OM nodes under blockade. When test executed at about 10%, executed blockade command as following:
> blockade partition ozone-ha_scm1_1,ozone-ha_om2_1 ozone-ha_scm2_1,ozone-ha_scm3_1,ozone-ha_om1_1,ozone-ha_om3_1
> blockade status
> NODE CONTAINER ID STATUS IP NETWORK PARTITION
> ozone-ha_om1_1 0d38eb51245b UP 192.168.0.12 NORMAL 2
> ozone-ha_om2_1 55935df5a89a UP 192.168.0.9 NORMAL 1
> ozone-ha_om3_1 a41473b9bba0 UP 192.168.0.5 NORMAL 2
> e40e35d6a744 UP 192.168.0.2 NORMAL 1
> ozone-ha_scm1_1
> e034f30ec3d5 UP 192.168.0.4 NORMAL 2
> ozone-ha_scm2_1
> afd4a99f917c UP 192.168.0.8 NORMAL 2
> ozone-ha_scm3_1
>
> freon randomkeys failed with following output:
> 2021-05-11 16:35:41,055 [main] INFO impl.MetricsConfig: Loaded properties from hadoop-metrics2.properties
> 2021-05-11 16:35:41,142 [main] INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
> 2021-05-11 16:35:41,142 [main] INFO impl.MetricsSystemImpl: ozone-freon metrics system started
> 2021-05-11 16:35:41,813 [main] INFO freon.RandomKeyGenerator: Number of Threads: 10
> 2021-05-11 16:35:41,814 [main] INFO freon.RandomKeyGenerator: Number of Volumes: 10.
> 2021-05-11 16:35:41,814 [main] INFO freon.RandomKeyGenerator: Number of Buckets per Volume: 50.
> 2021-05-11 16:35:41,814 [main] INFO freon.RandomKeyGenerator: Number of Keys per Bucket: 50.
> 2021-05-11 16:35:41,814 [main] INFO freon.RandomKeyGenerator: Key size: 10240 bytes
> 2021-05-11 16:35:41,814 [main] INFO freon.RandomKeyGenerator: Buffer size: 4096 bytes
> 2021-05-11 16:35:41,815 [main] INFO freon.RandomKeyGenerator: validateWrites : false
> 2021-05-11 16:35:41,832 [main] INFO freon.RandomKeyGenerator: Starting progress bar Thread.
>
> 0.00% |? | 0/25000 Time: 0:00:002021-05-11 16:35:41,870 [pool-2-thread-3] INFO rpc.RpcClient: Creating Volume: vol-2-31542, with hadoop as owner and space quota set to -1 bytes, counts quota set to -1
> :
> :
> 2021-05-11 16:35:44,612 [pool-2-thread-3] WARN impl.MetricRegistriesImpl: First MetricRegistry has been created without registering reporters. You may need to call MetricRegistries.global().addReporterRegistration(...) before.
> 15.12% |???????????????? | 3779/25000 Time: 0:15:462021-05-11 16:51:28,498 [pool-2-thread-5] INFO retry.RetryInvocationHandler: com.google.protobuf.ServiceException: java.net.SocketTimeoutException: Call From afd4a99f917c/192.168.0.8 to om2:9862 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/192.168.0.8:33724 remote=om2/192.168.0.9:9862]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking $Proxy21.submitRequest over nodeId=om2,nodeAddress=om2:9862. Trying to failover immediately.
> 2021-05-11 16:51:28,499 [pool-2-thread-2] INFO retry.RetryInvocationHandler: com.google.protobuf.ServiceException: java.net.SocketTimeoutException: Call From afd4a99f917c/192.168.0.8 to om2:9862 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/192.168.0.8:33724 remote=om2/192.168.0.9:9862]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking $Proxy21.submitRequest over nodeId=om2,nodeAddress=om2:9862. Trying to failover after sleeping for 2000ms.
> 2021-05-11 16:51:28,499 [pool-2-thread-8] INFO retry.RetryInvocationHandler: com.google.protobuf.ServiceException: java.net.SocketTimeoutException: Call From afd4a99f917c/192.168.0.8 to om2:9862 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/192.168.0.8:33724 remote=om2/192.168.0.9:9862]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking $Proxy21.submitRequest over nodeId=om2,nodeAddress=om2:9862. Trying to failover immediately.
> 2021-05-11 16:51:28,499 [pool-2-thread-9] INFO retry.RetryInvocationHandler: com.google.protobuf.ServiceException: java.net.SocketTimeoutException: Call From afd4a99f917c/192.168.0.8 to om2:9862 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/192.168.0.8:33724 remote=om2/192.168.0.9:9862]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking $Proxy21.submitRequest over nodeId=om2,nodeAddress=om2:9862. Trying to failover immediately.
> 2021-05-11 16:51:28,502 [pool-2-thread-7] INFO retry.RetryInvocationHandler: com.google.protobuf.ServiceException: java.net.SocketTimeoutException: Call From afd4a99f917c/192.168.0.8 to om2:9862 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/192.168.0.8:33724 remote=om2/192.168.0.9:9862]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking $Proxy21.submitRequest over nodeId=om2,nodeAddress=om2:9862. Trying to failover immediately.
> 2021-05-11 16:51:28,502 [pool-2-thread-8] WARN retry.RetryInvocationHandler: A failover has occurred since the start of call #8573 $Proxy21.submitRequest over nodeId=om2,nodeAddress=om2:9862
> 2021-05-11 16:51:28,505 [pool-2-thread-4] INFO retry.RetryInvocationHandler: com.google.protobuf.ServiceException: java.net.SocketTimeoutException: Call From afd4a99f917c/192.168.0.8 to om2:9862 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/192.168.0.8:33724 remote=om2/192.168.0.9:9862]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking $Proxy21.submitRequest over nodeId=om3,nodeAddress=om3:9862. Trying to failover immediately.
> 2021-05-11 16:51:28,506 [pool-2-thread-3] INFO retry.RetryInvocationHandler: com.google.protobuf.ServiceException: java.net.SocketTimeoutException: Call From afd4a99f917c/192.168.0.8 to om2:9862 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/192.168.0.8:33724 remote=om2/192.168.0.9:9862]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking $Proxy21.submitRequest over nodeId=om3,nodeAddress=om3:9862. Trying to failover after sleeping for 2000ms.
> 2021-05-11 16:51:28,506 [pool-2-thread-1] INFO retry.RetryInvocationHandler: com.google.protobuf.ServiceException: java.net.SocketTimeoutException: Call From afd4a99f917c/192.168.0.8 to om2:9862 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/192.168.0.8:33724 remote=om2/192.168.0.9:9862]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking $Proxy21.submitRequest over nodeId=om3,nodeAddress=om3:9862. Trying to failover immediately.
> 2021-05-11 16:51:28,506 [pool-2-thread-4] WARN retry.RetryInvocationHandler: A failover has occurred since the start of call #8574 $Proxy21.submitRequest over nodeId=om3,nodeAddress=om3:9862
> 2021-05-11 16:51:28,507 [pool-2-thread-6] INFO retry.RetryInvocationHandler: com.google.protobuf.ServiceException: java.net.SocketTimeoutException: Call From afd4a99f917c/192.168.0.8 to om2:9862 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/192.168.0.8:33724 remote=om2/192.168.0.9:9862]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking $Proxy21.submitRequest over nodeId=om3,nodeAddress=om3:9862. Trying to failover after sleeping for 2000ms.
> 2021-05-11 16:51:28,507 [pool-2-thread-1] WARN retry.RetryInvocationHandler: A failover has occurred since the start of call #8569 $Proxy21.submitRequest over nodeId=om3,nodeAddress=om3:9862
> 2021-05-11 16:51:28,509 [pool-2-thread-7] WARN retry.RetryInvocationHandler: A failover has occurred since the start of call #8572 $Proxy21.submitRequest over nodeId=om3,nodeAddress=om3:9862
> 2021-05-11 16:51:28,510 [pool-2-thread-10] INFO retry.RetryInvocationHandler: com.google.protobuf.ServiceException: java.net.SocketTimeoutException: Call From afd4a99f917c/192.168.0.8 to om2:9862 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/192.168.0.8:33724 remote=om2/192.168.0.9:9862]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout, while invoking $Proxy21.submitRequest over nodeId=om3,nodeAddress=om3:9862. Trying to failover immediately.
> 2021-05-11 16:51:28,510 [pool-2-thread-10] WARN retry.RetryInvocationHandler: A failover has occurred since the start of call #8577 $Proxy21.submitRequest over nodeId=om3,nodeAddress=om3:9862
> 2021-05-11 16:51:28,510 [pool-2-thread-9] WARN retry.RetryInvocationHandler: A failover has occurred since the start of call #8578 $Proxy21.submitRequest over nodeId=om3,nodeAddress=om3:9862
> 2021-05-11 16:51:28,579 [pool-2-thread-8] ERROR freon.RandomKeyGenerator: Exception while adding key: key-7-39708 in bucket: bucket-27-13459 of volume: vol-3-91505.
> KEY_NOT_FOUND org.apache.hadoop.ozone.om.exceptions.OMException: Failed to commit key, as /vol-3-91505/bucket-27-13459/key-7-39708/106217536723227301entry is not found in the OpenKey table
> at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:604)
> at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.commitKey(OzoneManagerProtocolClientSideTranslatorPB.java:664)
> at org.apache.hadoop.ozone.client.io.BlockOutputStreamEntryPool.commitKey(BlockOutputStreamEntryPool.java:263)
> at org.apache.hadoop.ozone.client.io.KeyOutputStream.close(KeyOutputStream.java:528)
> at org.apache.hadoop.ozone.client.io.OzoneOutputStream.close(OzoneOutputStream.java:61)
> at org.apache.hadoop.ozone.freon.RandomKeyGenerator.createKey(RandomKeyGenerator.java:726)
> at org.apache.hadoop.ozone.freon.RandomKeyGenerator.access$1100(RandomKeyGenerator.java:86)
> at org.apache.hadoop.ozone.freon.RandomKeyGenerator$ObjectCreator.run(RandomKeyGenerator.java:621)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2021-05-11 16:51:28,580 [pool-2-thread-4] ERROR freon.RandomKeyGenerator: Exception while adding key: key-7-95957 in bucket: bucket-27-74790 of volume: vol-5-95335.
> KEY_NOT_FOUND org.apache.hadoop.ozone.om.exceptions.OMException: Failed to commit key, as /vol-5-95335/bucket-27-74790/key-7-95957/106217536724013735entry is not found in the OpenKey table
> at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:604)
> at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.commitKey(OzoneManagerProtocolClientSideTranslatorPB.java:664)
> at org.apache.hadoop.ozone.client.io.BlockOutputStreamEntryPool.commitKey(BlockOutputStreamEntryPool.java:263)
> at org.apache.hadoop.ozone.client.io.KeyOutputStream.close(KeyOutputStream.java:528)
> at org.apache.hadoop.ozone.client.io.OzoneOutputStream.close(OzoneOutputStream.java:61)
> at org.apache.hadoop.ozone.freon.RandomKeyGenerator.createKey(RandomKeyGenerator.java:726)
> at org.apache.hadoop.ozone.freon.RandomKeyGenerator.access$1100(RandomKeyGenerator.java:86)
> at org.apache.hadoop.ozone.freon.RandomKeyGenerator$ObjectCreator.run(RandomKeyGenerator.java:621)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2021-05-11 16:51:28,582 [pool-2-thread-1] ERROR freon.RandomKeyGenerator: Exception while adding key: key-7-69461 in bucket: bucket-26-04246 of volume: vol-9-59350.
> KEY_NOT_FOUND org.apache.hadoop.ozone.om.exceptions.OMException: Failed to commit key, as /vol-9-59350/bucket-26-04246/key-7-69461/106217536720868001entry is not found in the OpenKey table
> at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:604)
> at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.commitKey(OzoneManagerProtocolClientSideTranslatorPB.java:664)
> at org.apache.hadoop.ozone.client.io.BlockOutputStreamEntryPool.commitKey(BlockOutputStreamEntryPool.java:263)
> at org.apache.hadoop.ozone.client.io.KeyOutputStream.close(KeyOutputStream.java:528)
> at org.apache.hadoop.ozone.client.io.OzoneOutputStream.close(OzoneOutputStream.java:61)
> at org.apache.hadoop.ozone.freon.RandomKeyGenerator.createKey(RandomKeyGenerator.java:726)
> at org.apache.hadoop.ozone.freon.RandomKeyGenerator.access$1100(RandomKeyGenerator.java:86)
> at org.apache.hadoop.ozone.freon.RandomKeyGenerator$ObjectCreator.run(RandomKeyGenerator.java:621)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2021-05-11 16:51:28,579 [pool-2-thread-9] ERROR freon.RandomKeyGenerator: Exception while adding key: key-7-47910 in bucket: bucket-27-13990 of volume: vol-8-92485.
> KEY_NOT_FOUND org.apache.hadoop.ozone.om.exceptions.OMException: Failed to commit key, as /vol-8-92485/bucket-27-13990/key-7-47910/106217536727487146entry is not found in the OpenKey table
> at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:604)
> at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.commitKey(OzoneManagerProtocolClientSideTranslatorPB.java:664)
> at org.apache.hadoop.ozone.client.io.BlockOutputStreamEntryPool.commitKey(BlockOutputStreamEntryPool.java:263)
> at org.apache.hadoop.ozone.client.io.KeyOutputStream.close(KeyOutputStream.java:528)
> at org.apache.hadoop.ozone.client.io.OzoneOutputStream.close(OzoneOutputStream.java:61)
> at org.apache.hadoop.ozone.freon.RandomKeyGenerator.createKey(RandomKeyGenerator.java:726)
> at org.apache.hadoop.ozone.freon.RandomKeyGenerator.access$1100(RandomKeyGenerator.java:86)
> at org.apache.hadoop.ozone.freon.RandomKeyGenerator$ObjectCreator.run(RandomKeyGenerator.java:621)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2021-05-11 16:51:28,584 [pool-2-thread-10] ERROR freon.RandomKeyGenerator: Exception while adding key: key-7-67604 in bucket: bucket-27-46498 of volume: vol-7-16026.
> KEY_NOT_FOUND org.apache.hadoop.ozone.om.exceptions.OMException: Failed to commit key, as /vol-7-16026/bucket-27-46498/key-7-67604/106217536727421609entry is not found in the OpenKey table
> at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:604)
> at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.commitKey(OzoneManagerProtocolClientSideTranslatorPB.java:664)
> at org.apache.hadoop.ozone.client.io.BlockOutputStreamEntryPool.commitKey(BlockOutputStreamEntryPool.java:263)
> at org.apache.hadoop.ozone.client.io.KeyOutputStream.close(KeyOutputStream.java:528)
> at org.apache.hadoop.ozone.client.io.OzoneOutputStream.close(OzoneOutputStream.java:61)
> at org.apache.hadoop.ozone.freon.RandomKeyGenerator.createKey(RandomKeyGenerator.java:726)
> at org.apache.hadoop.ozone.freon.RandomKeyGenerator.access$1100(RandomKeyGenerator.java:86)
> at org.apache.hadoop.ozone.freon.RandomKeyGenerator$ObjectCreator.run(RandomKeyGenerator.java:621)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2021-05-11 16:51:28,586 [pool-2-thread-5] ERROR freon.RandomKeyGenerator: Exception while adding key: key-7-51341 in bucket: bucket-27-00966 of volume: vol-4-81053.
> KEY_NOT_FOUND org.apache.hadoop.ozone.om.exceptions.OMException: Failed to commit key, as /vol-4-81053/bucket-27-00966/key-7-51341/106217536723620518entry is not found in the OpenKey table
> at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:604)
> at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.commitKey(OzoneManagerProtocolClientSideTranslatorPB.java:664)
> at org.apache.hadoop.ozone.client.io.BlockOutputStreamEntryPool.commitKey(BlockOutputStreamEntryPool.java:263)
> at org.apache.hadoop.ozone.client.io.KeyOutputStream.close(KeyOutputStream.java:528)
> at org.apache.hadoop.ozone.client.io.OzoneOutputStream.close(OzoneOutputStream.java:61)
> at org.apache.hadoop.ozone.freon.RandomKeyGenerator.createKey(RandomKeyGenerator.java:726)
> at org.apache.hadoop.ozone.freon.RandomKeyGenerator.access$1100(RandomKeyGenerator.java:86)
> at org.apache.hadoop.ozone.freon.RandomKeyGenerator$ObjectCreator.run(RandomKeyGenerator.java:621)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2021-05-11 16:51:28,588 [pool-2-thread-7] ERROR freon.RandomKeyGenerator: Exception while adding key: key-7-36072 in bucket: bucket-27-27098 of volume: vol-2-31542.
> KEY_NOT_FOUND org.apache.hadoop.ozone.om.exceptions.OMException: Failed to commit key, as /vol-2-31542/bucket-27-27098/key-7-36072/106217536723227300entry is not found in the OpenKey table
> at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:604)
> at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.commitKey(OzoneManagerProtocolClientSideTranslatorPB.java:664)
> at org.apache.hadoop.ozone.client.io.BlockOutputStreamEntryPool.commitKey(BlockOutputStreamEntryPool.java:263)
> at org.apache.hadoop.ozone.client.io.KeyOutputStream.close(KeyOutputStream.java:528)
> at org.apache.hadoop.ozone.client.io.OzoneOutputStream.close(OzoneOutputStream.java:61)
> at org.apache.hadoop.ozone.freon.RandomKeyGenerator.createKey(RandomKeyGenerator.java:726)
> at org.apache.hadoop.ozone.freon.RandomKeyGenerator.access$1100(RandomKeyGenerator.java:86)
> at org.apache.hadoop.ozone.freon.RandomKeyGenerator$ObjectCreator.run(RandomKeyGenerator.java:621)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 15.12% |???????????????? | 3779/25000 Time: 0:15:482021-05-11 16:51:30,501 [pool-2-thread-2] WARN retry.RetryInvocationHandler: A failover has occurred since the start of call #8575 $Proxy21.submitRequest over nodeId=om3,nodeAddress=om3:9862
> 2021-05-11 16:51:30,506 [pool-2-thread-3] WARN retry.RetryInvocationHandler: A failover has occurred since the start of call #8571 $Proxy21.submitRequest over nodeId=om3,nodeAddress=om3:9862
> 2021-05-11 16:51:30,509 [pool-2-thread-6] WARN retry.RetryInvocationHandler: A failover has occurred since the start of call #8570 $Proxy21.submitRequest over nodeId=om3,nodeAddress=om3:9862
> 2021-05-11 16:51:30,511 [pool-2-thread-2] ERROR freon.RandomKeyGenerator: Exception while adding key: key-7-58453 in bucket: bucket-27-15691 of volume: vol-6-95831.
> KEY_NOT_FOUND org.apache.hadoop.ozone.om.exceptions.OMException: Failed to commit key, as /vol-6-95831/bucket-27-15691/key-7-58453/106217536724669096entry is not found in the OpenKey table
> at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:604)
> at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.commitKey(OzoneManagerProtocolClientSideTranslatorPB.java:664)
> at org.apache.hadoop.ozone.client.io.BlockOutputStreamEntryPool.commitKey(BlockOutputStreamEntryPool.java:263)
> at org.apache.hadoop.ozone.client.io.KeyOutputStream.close(KeyOutputStream.java:528)
> at org.apache.hadoop.ozone.client.io.OzoneOutputStream.close(OzoneOutputStream.java:61)
> at org.apache.hadoop.ozone.freon.RandomKeyGenerator.createKey(RandomKeyGenerator.java:726)
> at org.apache.hadoop.ozone.freon.RandomKeyGenerator.access$1100(RandomKeyGenerator.java:86)
> at org.apache.hadoop.ozone.freon.RandomKeyGenerator$ObjectCreator.run(RandomKeyGenerator.java:621)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2021-05-11 16:51:30,515 [pool-2-thread-3] ERROR freon.RandomKeyGenerator: Exception while adding key: key-7-86181 in bucket: bucket-27-73522 of volume: vol-1-21313.
> KEY_NOT_FOUND org.apache.hadoop.ozone.om.exceptions.OMException: Failed to commit key, as /vol-1-21313/bucket-27-73522/key-7-86181/106217536722178723entry is not found in the OpenKey table
> at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:604)
> at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.commitKey(OzoneManagerProtocolClientSideTranslatorPB.java:664)
> at org.apache.hadoop.ozone.client.io.BlockOutputStreamEntryPool.commitKey(BlockOutputStreamEntryPool.java:263)
> at org.apache.hadoop.ozone.client.io.KeyOutputStream.close(KeyOutputStream.java:528)
> at org.apache.hadoop.ozone.client.io.OzoneOutputStream.close(OzoneOutputStream.java:61)
> at org.apache.hadoop.ozone.freon.RandomKeyGenerator.createKey(RandomKeyGenerator.java:726)
> at org.apache.hadoop.ozone.freon.RandomKeyGenerator.access$1100(RandomKeyGenerator.java:86)
> at org.apache.hadoop.ozone.freon.RandomKeyGenerator$ObjectCreator.run(RandomKeyGenerator.java:621)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2021-05-11 16:51:30,519 [pool-2-thread-6] ERROR freon.RandomKeyGenerator: Exception while adding key: key-7-21473 in bucket: bucket-27-93062 of volume: vol-0-55753.
> KEY_NOT_FOUND org.apache.hadoop.ozone.om.exceptions.OMException: Failed to commit key, as /vol-0-55753/bucket-27-93062/key-7-21473/106217536721785506entry is not found in the OpenKey table
> at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:604)
> at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.commitKey(OzoneManagerProtocolClientSideTranslatorPB.java:664)
> at org.apache.hadoop.ozone.client.io.BlockOutputStreamEntryPool.commitKey(BlockOutputStreamEntryPool.java:263)
> at org.apache.hadoop.ozone.client.io.KeyOutputStream.close(KeyOutputStream.java:528)
> at org.apache.hadoop.ozone.client.io.OzoneOutputStream.close(OzoneOutputStream.java:61)
> at org.apache.hadoop.ozone.freon.RandomKeyGenerator.createKey(RandomKeyGenerator.java:726)
> at org.apache.hadoop.ozone.freon.RandomKeyGenerator.access$1100(RandomKeyGenerator.java:86)
> at org.apache.hadoop.ozone.freon.RandomKeyGenerator$ObjectCreator.run(RandomKeyGenerator.java:621)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 100.00% |?????????????????????????????????????????????????????????????????????????????????????????????????????| 25000/25000 Time: 0:15:51
> 2021-05-11 16:51:36,051 [main] WARN grpc.GrpcUtil: Timed out gracefully shutting down connection: ManagedChannelOrphanWrapper\{delegate=ManagedChannelImpl{logId=1, target=192.168.0.3:9858}}.
> KEY_NOT_FOUND org.apache.hadoop.ozone.om.exceptions.OMException: Failed to commit key, as /vol-0-55753/bucket-27-93062/key-7-21473/106217536721785506entry is not found in the OpenKey table
>
> ***************************************************
> Status: Failed
> Git Base Revision: 7a3bc90b05f257c8ace2f76d74264906f0f7a932
> Number of Volumes created: 10
> Number of Buckets created: 500
> Number of Keys added: 3779
> Ratis replication factor: THREE
> Ratis replication type: RATIS
> Average Time spent in volume creation: 00:00:00,089
> Average Time spent in bucket creation: 00:00:01,643
> Average Time spent in key creation: 00:00:09,633
> Average Time spent in key write: 00:00:01,184
> Total bytes written: 38696960
> Total Execution time: 00:15:54,754
> ***************************************************
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org