You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Mukul Kumar Singh (JIRA)" <ji...@apache.org> on 2017/08/22 12:50:00 UTC
[jira] [Created] (HDFS-12337) Ozone: Concurrent RocksDB open calls
fail because of "No locks available"
Mukul Kumar Singh created HDFS-12337:
----------------------------------------
Summary: Ozone: Concurrent RocksDB open calls fail because of "No locks available"
Key: HDFS-12337
URL: https://issues.apache.org/jira/browse/HDFS-12337
Project: Hadoop HDFS
Issue Type: Sub-task
Components: ozone
Affects Versions: HDFS-7240
Reporter: Mukul Kumar Singh
Assignee: Mukul Kumar Singh
Fix For: HDFS-7240
HDFS-12216 fixes the issue with the static container cache and re-using the same container port on datanode restart. However TestKeys still fails after HDFS-12216 is fixed.
The test is now failing because Concurrent RocksDB open calls fail. In the current code BlockDeleting service and Dispatcher tries to open the db concurrently.
This jira will also fix the keepPort property for Ratis Container port and set the reuse address property for XceiverServerRatis correctly.
{code}
2017-08-22 16:51:34,453 [BlockDeletingService#1] INFO utils.RocksDBStore (RocksDBStore.java:<init>(64)) - opening db file /Users/msingh/code/work/apache/cblock/ozone_review/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/dn0_data0/containers/621a3b15-b9fc-4d49-a6e6-29d4c40cc91f/metadata/container.db
2017-08-22 16:51:34,460 [nioEventLoopGroup-9-1] INFO logging.LoggingHandler (Slf4JLogger.java:info(101)) - [id: 0x8822cd3d, /0.0.0.0:57044] RECEIVED: [id: 0x61367e6f, /127.0.0.1:57173 => /127.0.0
.1:57044]
2017-08-22 16:51:34,461 [nioEventLoopGroup-10-1] INFO utils.RocksDBStore (RocksDBStore.java:<init>(64)) - opening db file /Users/msingh/code/work/apache/cblock/ozone_review/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/dn0_data0/containers/621a3b15-b9fc-4d49-a6e6-29d4c40cc91f/metadata/container.db
2017-08-22 16:51:34,465 [nioEventLoopGroup-10-1] INFO utils.RocksDBStore (RocksDBStore.java:<init>(67)) - Failed init RocksDB, db path : /Users/msingh/code/work/apache/cblock/ozone_review/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/dn0_data0/containers/621a3b15-b9fc-4d49-a6e6-29d4c40cc91f/metadata/container.dbexception org.rocksdb.RocksDBException: lock /Users/msingh/code/work/apache/cblock/ozone_review/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/dn0_data0/containers/621a3b15-b9fc-4d49-a6e6-29d4c40cc91f/metadata/container.db/LOCK: No locks available
2017-08-22 16:51:34,465 [BlockDeletingService#1] INFO background.BlockDeletingService (BlockDeletingService.java:call(203)) - The elapsed time of task@70a576ee for deleting blocks: 12ms.
2017-08-22 16:51:34,474 [nioEventLoopGroup-10-1] INFO impl.Dispatcher (ContainerUtils.java:logAndReturnError(129)) - Operation: GetKey : Trace ID: 73f19131-f63b-459a-8f09-9a3db893a296 : Message: 621a3b15-b9fc-4d49-a6e6-29d4c40cc91f : Result: UNABLE_TO_READ_METADATA_DB
2017-08-22 16:51:34,475 [Thread-382] INFO exceptions.OzoneExceptionMapper (OzoneExceptionMapper.java:toResponse(39)) ozone c2a23759-c76f-49ea-b574-f0802a4e5b75/c0df3a48-f75b-4b5e-b1bd-c189ce698056/13b3d486-3d7a-49e4-bc9d-1ef63e674548 hdfs 73f19131-f63b-459a-8f09-9a3db893a296 - Returning exception. ex: {"httpCode":500,"shortMessage":"internalServerError","resource":"hdfs","message":"621a3b15-b9fc-4d49-a6e6-29d4c40cc91f","requestID":"73f19131-f63b-459a-8f09-9a3db893a296","hostName":"hw13605.local"}
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org