You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by "Mukul Kumar Singh (JIRA)" <ji...@apache.org> on 2018/10/10 07:30:03 UTC
[jira] [Resolved] (HDDS-610) On restart of SCM it fails to register
DataNodes
[ https://issues.apache.org/jira/browse/HDDS-610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mukul Kumar Singh resolved HDDS-610.
------------------------------------
Resolution: Duplicate
> On restart of SCM it fails to register DataNodes
> ------------------------------------------------
>
> Key: HDDS-610
> URL: https://issues.apache.org/jira/browse/HDDS-610
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Reporter: Namit Maheshwari
> Assignee: Hanisha Koneru
> Priority: Major
>
> {code:java}
> 2018-10-09 23:34:11,105 INFO org.apache.hadoop.hdds.scm.server.StorageContainerManager: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting StorageContainerManager
> STARTUP_MSG: host = ctr-e138-1518143905142-510793-01-000004.hwx.site/172.27.79.197
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 3.3.0-SNAPSHOT
> STARTUP_MSG: classpath = /tmp/ozone-0.3.0-SNAPSHOT/etc/hadoop:/tmp/ozone-0.3.0-SNAPSHOT/share/hadoop/common/*:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/kerb-simplekdc-1.0.1.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jackson-core-2.9.5.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/protobuf-java-2.5.0.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/netty-3.10.5.Final.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/commons-beanutils-1.9.3.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/okio-1.6.0.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/kerb-core-1.0.1.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jetty-security-9.3.19.v20170502.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/bcpkix-jdk15on-1.54.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/ratis-client-0.3.0-eca3531-SNAPSHOT.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jsr305-3.0.0.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/leveldbjni-all-1.8.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/j2objc-annotations-1.3.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jctools-core-2.1.2.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/auto-common-0.8.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/ratis-common-0.3.0-eca3531-SNAPSHOT.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jzlib-1.1.3.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/htrace-core4-4.1.0-incubating.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/auto-value-1.6.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/kerb-util-1.0.1.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/hadoop-hdds-server-framework-0.3.0-SNAPSHOT.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jetty-servlet-9.3.19.v20170502.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jackson-jaxrs-1.9.13.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/kerby-xdr-1.0.1.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/hadoop-hdds-client-0.3.0-SNAPSHOT.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jackson-core-asl-1.9.13.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/lz4-1.3.0.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/commons-net-3.6.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/ratis-netty-0.3.0-eca3531-SNAPSHOT.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/log4j-api-2.11.0.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/netty-all-4.0.52.Final.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/hamcrest-all-1.3.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/error_prone_annotations-2.2.0.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/slf4j-log4j12-1.7.25.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/kerby-config-1.0.1.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/auto-service-1.0-rc4.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/metrics-core-3.2.4.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/kerby-pkix-1.0.1.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/xz-1.0.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/rocksdbjni-5.14.2.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/nimbus-jose-jwt-4.41.1.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/hadoop-hdds-container-service-0.3.0-SNAPSHOT.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/hadoop-auth-3.3.0-SNAPSHOT.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jsr311-api-1.1.1.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jackson-annotations-2.9.5.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/javax.servlet-api-3.1.0.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/kerb-server-1.0.1.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/zookeeper-3.4.13.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/avro-1.7.7.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/kerb-client-1.0.1.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jackson-xc-1.9.13.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/disruptor-3.4.2.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/guava-11.0.2.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/curator-framework-2.12.0.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jackson-databind-2.9.5.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/dnsjava-2.1.7.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/hadoop-common-3.3.0-SNAPSHOT.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/commons-configuration2-2.1.1.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/json-smart-2.3.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/token-provider-1.0.1.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/log4j-1.2.17.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/commons-daemon-1.0.13.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jetty-util-ajax-9.3.19.v20170502.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/picocli-3.5.2.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/ratis-grpc-0.3.0-eca3531-SNAPSHOT.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/commons-lang3-3.7.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/commons-compress-1.4.1.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/accessors-smart-1.2.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/bcprov-jdk15on-1.54.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jaxb-impl-2.2.3-1.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/re2j-1.1.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/curator-recipes-2.12.0.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/commons-math3-3.1.1.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jetty-xml-9.3.19.v20170502.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/httpclient-4.5.2.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/asm-5.0.4.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/hadoop-hdds-common-0.3.0-SNAPSHOT.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/curator-client-2.12.0.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/audience-annotations-0.5.0.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/kerby-util-1.0.1.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jetty-server-9.3.19.v20170502.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jackson-mapper-asl-1.9.13.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jaxb-api-2.2.11.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jsp-api-2.1.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/ratis-server-0.3.0-eca3531-SNAPSHOT.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jsch-0.1.54.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/compress-lzf-1.0.3.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jetty-util-9.3.19.v20170502.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/netty-tcnative-2.0.8.Final-osx-x86_64.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/paranamer-2.3.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/log4j-core-2.11.0.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jetty-http-9.3.19.v20170502.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/commons-codec-1.11.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/commons-pool2-2.6.0.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/gson-2.2.4.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jcip-annotations-1.0-1.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/kerb-admin-1.0.1.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/commons-io-2.5.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/snakeyaml-1.16.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jersey-core-1.19.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/javapoet-1.10.0.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jetty-webapp-9.3.19.v20170502.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jboss-marshalling-1.4.11.Final.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/aalto-xml-1.0.0.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/hadoop-hdfs-3.3.0-SNAPSHOT.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/commons-cli-1.2.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/woodstox-core-5.0.3.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/snappy-java-1.0.5.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/ratis-proto-shaded-0.3.0-eca3531-SNAPSHOT.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/kerby-asn1-1.0.1.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/stax2-api-3.1.4.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/commons-collections-3.2.2.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jettison-1.1.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/commons-text-1.4.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jersey-servlet-1.19.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/hadoop-hdfs-client-3.3.0-SNAPSHOT.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jersey-json-1.19.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/slf4j-api-1.7.25.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/okhttp-2.7.5.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/commons-logging-1.1.3.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jetty-io-9.3.19.v20170502.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/kerb-identity-1.0.1.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/animal-sniffer-annotations-1.16.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/kerb-common-1.0.1.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/jersey-server-1.19.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/auto-value-annotations-1.6.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/kerb-crypto-1.0.1.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/httpcore-4.4.4.jar:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/web:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/hadoop-hdds-server-scm-0.3.0-SNAPSHOT.jar:/opt/byteman.jar
> STARTUP_MSG: build = git@github.com:apache/hadoop.git -r 347ea385817766a5c418017009728cd8b9959776; compiled by 'nmaheshwari' on 2018-10-09T00:14Z
> STARTUP_MSG: java = 1.8.0_112
> ************************************************************/
> 2018-10-09 23:34:11,119 INFO org.apache.hadoop.hdds.scm.server.StorageContainerManager: registered UNIX signal handlers for [TERM, HUP, INT]
> 2018-10-09 23:34:11,376 INFO org.apache.hadoop.hdds.scm.node.SCMNodeManager: Entering startup chill mode.
> 2018-10-09 23:34:12,106 INFO org.apache.hadoop.hdds.scm.server.SCMChillModeManager: SCM exiting chill mode.
> 2018-10-09 23:34:12,107 WARN org.apache.hadoop.hdds.server.events.EventQueue: No event handler registered for event TypedEvent{payloadType=Boolean, name='Boolean'}
> 2018-10-09 23:34:12,107 WARN org.apache.hadoop.hdds.server.events.EventQueue: No event handler registered for event TypedEvent{payloadType=Boolean, name='Boolean'}
> 2018-10-09 23:34:12,202 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> 2018-10-09 23:34:12,420 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue, queueCapacity: 1000, scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler, ipcBackoff: false.
> 2018-10-09 23:34:12,448 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 9861
> 2018-10-09 23:34:12,508 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue, queueCapacity: 1000, scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler, ipcBackoff: false.
> 2018-10-09 23:34:12,509 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 9863
> 2018-10-09 23:34:12,539 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue, queueCapacity: 1000, scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler, ipcBackoff: false.
> 2018-10-09 23:34:12,541 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 9860
> 2018-10-09 23:34:12,580 INFO org.apache.hadoop.hdfs.DFSUtil: Starting Web-server for scm at: http://0.0.0.0:9876
> 2018-10-09 23:34:12,626 INFO org.eclipse.jetty.util.log: Logging initialized @2524ms
> 2018-10-09 23:34:12,800 INFO org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets.
> 2018-10-09 23:34:12,824 INFO org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.scm is not defined
> 2018-10-09 23:34:12,840 INFO org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
> 2018-10-09 23:34:12,844 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context scm
> 2018-10-09 23:34:12,845 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs
> 2018-10-09 23:34:12,845 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
> 2018-10-09 23:34:12,960 INFO org.apache.hadoop.hdds.scm.server.StorageContainerManager: StorageContainerLocationProtocol RPC server is listening at /0.0.0.0:9860
> 2018-10-09 23:34:13,113 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: Loaded properties from hadoop-metrics2.properties
> 2018-10-09 23:34:13,134 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
> 2018-10-09 23:34:13,134 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: StorageContainerManager metrics system started
> 2018-10-09 23:34:13,189 INFO org.apache.hadoop.hdds.scm.server.SCMClientProtocolServer: RPC server for Client is listening at /0.0.0.0:9860
> 2018-10-09 23:34:13,190 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
> 2018-10-09 23:34:13,190 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 9860: starting
> 2018-10-09 23:34:13,193 INFO org.apache.hadoop.hdds.scm.server.StorageContainerManager: ScmBlockLocationProtocol RPC server is listening at /0.0.0.0:9863
> 2018-10-09 23:34:13,193 INFO org.apache.hadoop.hdds.scm.server.SCMBlockProtocolServer: RPC server for Block Protocol is listening at /0.0.0.0:9863
> 2018-10-09 23:34:13,193 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
> 2018-10-09 23:34:13,194 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 9863: starting
> 2018-10-09 23:34:13,197 INFO org.apache.hadoop.hdds.scm.server.StorageContainerManager: ScmDatanodeProtocl RPC server is listening at /0.0.0.0:9861
> 2018-10-09 23:34:13,197 INFO org.apache.hadoop.hdds.scm.server.SCMDatanodeProtocolServer: RPC server for DataNodes is listening at /0.0.0.0:9861
> 2018-10-09 23:34:13,198 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
> 2018-10-09 23:34:13,199 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 9861: starting
> 2018-10-09 23:34:13,208 INFO org.apache.hadoop.http.HttpServer2: Jetty bound to port 9876
> 2018-10-09 23:34:13,209 INFO org.eclipse.jetty.server.Server: jetty-9.3.19.v20170502
> 2018-10-09 23:34:13,264 INFO org.eclipse.jetty.server.handler.ContextHandler: Started o.e.j.s.ServletContextHandler@2484f433{/logs,file:///tmp/ozone-0.3.0-SNAPSHOT/logs/,AVAILABLE}
> 2018-10-09 23:34:13,265 INFO org.eclipse.jetty.server.handler.ContextHandler: Started o.e.j.s.ServletContextHandler@5f59185e{/static,jar:file:/tmp/ozone-0.3.0-SNAPSHOT/share/ozone/lib/hadoop-hdds-server-scm-0.3.0-SNAPSHOT.jar!/webapps/static,AVAILABLE}
> 2018-10-09 23:34:13,379 INFO org.eclipse.jetty.server.handler.ContextHandler: Started o.e.j.w.WebAppContext@24528a25{/,file:///tmp/jetty-0.0.0.0-9876-scm-_-any-4060966483262888676.dir/webapp/,AVAILABLE}{/scm}
> 2018-10-09 23:34:13,387 INFO org.eclipse.jetty.server.AbstractConnector: Started ServerConnector@46292372{HTTP/1.1,[http/1.1]}{0.0.0.0:9876}
> 2018-10-09 23:34:13,387 INFO org.eclipse.jetty.server.Server: Started @3284ms
> 2018-10-09 23:34:13,388 INFO org.apache.hadoop.hdds.server.BaseHttpServer: HTTP server of SCM is listening at http://0.0.0.0:9876
> 2018-10-09 23:34:13,440 WARN org.apache.hadoop.hdds.scm.node.SCMNodeManager: SCM receive heartbeat from unregistered datanode 636d2ebe-b1c0-4885-b07d-b6624d583258{ip: 172.27.23.193, host: ctr-e138-1518143905142-510793-01-000006.hwx.site}
> 2018-10-09 23:34:13,455 ERROR org.apache.hadoop.hdds.scm.container.ContainerReportHandler: Error on processing container report from datanode 636d2ebe-b1c0-4885-b07d-b6624d583258{ip: 172.27.23.193, host: ctr-e138-1518143905142-510793-01-000006.hwx.site}
> org.apache.hadoop.hdds.scm.exceptions.SCMException: No such datanode
> at org.apache.hadoop.hdds.scm.node.states.Node2ContainerMap.setContainersForDatanode(Node2ContainerMap.java:82)
> at org.apache.hadoop.hdds.scm.node.NodeStateManager.setContainersForDatanode(NodeStateManager.java:506)
> at org.apache.hadoop.hdds.scm.node.SCMNodeManager.setContainersForDatanode(SCMNodeManager.java:410)
> at org.apache.hadoop.hdds.scm.container.ContainerReportHandler.onMessage(ContainerReportHandler.java:97)
> at org.apache.hadoop.hdds.scm.container.ContainerReportHandler.onMessage(ContainerReportHandler.java:45)
> at org.apache.hadoop.hdds.server.events.SingleThreadExecutor.lambda$onMessage$1(SingleThreadExecutor.java:85)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 2018-10-09 23:34:31,325 WARN org.apache.hadoop.hdds.scm.node.SCMNodeManager: SCM receive heartbeat from unregistered datanode 06edf838-e6b1-4dea-9557-a008007cfb1b{ip: 172.27.17.133, host: ctr-e138-1518143905142-510793-01-000010.hwx.site}
> 2018-10-09 23:34:40,241 WARN org.apache.hadoop.hdds.scm.node.SCMNodeManager: SCM receive heartbeat from unregistered datanode 5086b8e0-1aa4-4c50-b4db-859f9a9519f7{ip: 172.27.87.64, host: ctr-e138-1518143905142-510793-01-000005.hwx.site}
> 2018-10-09 23:34:40,242 ERROR org.apache.hadoop.hdds.scm.container.ContainerReportHandler: Error on processing container report from datanode 5086b8e0-1aa4-4c50-b4db-859f9a9519f7{ip: 172.27.87.64, host: ctr-e138-1518143905142-510793-01-000005.hwx.site}
> org.apache.hadoop.hdds.scm.exceptions.SCMException: No such datanode
> at org.apache.hadoop.hdds.scm.node.states.Node2ContainerMap.setContainersForDatanode(Node2ContainerMap.java:82)
> at org.apache.hadoop.hdds.scm.node.NodeStateManager.setContainersForDatanode(NodeStateManager.java:506)
> at org.apache.hadoop.hdds.scm.node.SCMNodeManager.setContainersForDatanode(SCMNodeManager.java:410)
> at org.apache.hadoop.hdds.scm.container.ContainerReportHandler.onMessage(ContainerReportHandler.java:97)
> at org.apache.hadoop.hdds.scm.container.ContainerReportHandler.onMessage(ContainerReportHandler.java:45)
> at org.apache.hadoop.hdds.server.events.SingleThreadExecutor.lambda$onMessage$1(SingleThreadExecutor.java:85)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 2018-10-09 23:34:40,888 INFO org.apache.hadoop.hdds.scm.node.SCMNodeManager: Data node with ID: 636d2ebe-b1c0-4885-b07d-b6624d583258 Registered.
> 2018-10-09 23:34:43,789 WARN org.apache.hadoop.hdds.scm.node.SCMNodeManager: SCM receive heartbeat from unregistered datanode 1baa3f7c-d85a-4ab5-b005-d2a740233e32{ip: 172.27.52.130, host: ctr-e138-1518143905142-510793-01-000007.hwx.site}
> 2018-10-09 23:34:43,790 ERROR org.apache.hadoop.hdds.scm.container.ContainerReportHandler: Error on processing container report from datanode 1baa3f7c-d85a-4ab5-b005-d2a740233e32{ip: 172.27.52.130, host: ctr-e138-1518143905142-510793-01-000007.hwx.site}
> org.apache.hadoop.hdds.scm.exceptions.SCMException: No such datanode
> at org.apache.hadoop.hdds.scm.node.states.Node2ContainerMap.setContainersForDatanode(Node2ContainerMap.java:82)
> at org.apache.hadoop.hdds.scm.node.NodeStateManager.setContainersForDatanode(NodeStateManager.java:506)
> at org.apache.hadoop.hdds.scm.node.SCMNodeManager.setContainersForDatanode(SCMNodeManager.java:410)
> at org.apache.hadoop.hdds.scm.container.ContainerReportHandler.onMessage(ContainerReportHandler.java:97)
> at org.apache.hadoop.hdds.scm.container.ContainerReportHandler.onMessage(ContainerReportHandler.java:45)
> at org.apache.hadoop.hdds.server.events.SingleThreadExecutor.lambda$onMessage$1(SingleThreadExecutor.java:85)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 2018-10-09 23:34:49,416 WARN org.apache.hadoop.hdds.scm.node.SCMNodeManager: SCM receive heartbeat from unregistered datanode e9c4a12f-db11-4e19-98fc-d1ec7b9d1aeb{ip: 172.27.68.128, host: ctr-e138-1518143905142-510793-01-000008.hwx.site}
> 2018-10-09 23:35:01,324 INFO org.apache.hadoop.hdds.scm.node.SCMNodeManager: Data node with ID: 06edf838-e6b1-4dea-9557-a008007cfb1b Registered.
> 2018-10-09 23:35:03,358 WARN org.apache.hadoop.hdds.scm.node.SCMNodeManager: SCM receive heartbeat from unregistered datanode cf462de6-6709-48ef-b822-f9270e13bc6f{ip: 172.27.16.145, host: ctr-e138-1518143905142-510793-01-000011.hwx.site}
> 2018-10-09 23:35:03,359 ERROR org.apache.hadoop.hdds.scm.container.ContainerReportHandler: Error on processing container report from datanode cf462de6-6709-48ef-b822-f9270e13bc6f{ip: 172.27.16.145, host: ctr-e138-1518143905142-510793-01-000011.hwx.site}
> org.apache.hadoop.hdds.scm.exceptions.SCMException: No such datanode
> at org.apache.hadoop.hdds.scm.node.states.Node2ContainerMap.setContainersForDatanode(Node2ContainerMap.java:82)
> at org.apache.hadoop.hdds.scm.node.NodeStateManager.setContainersForDatanode(NodeStateManager.java:506)
> at org.apache.hadoop.hdds.scm.node.SCMNodeManager.setContainersForDatanode(SCMNodeManager.java:410)
> at org.apache.hadoop.hdds.scm.container.ContainerReportHandler.onMessage(ContainerReportHandler.java:97)
> at org.apache.hadoop.hdds.scm.container.ContainerReportHandler.onMessage(ContainerReportHandler.java:45)
> at org.apache.hadoop.hdds.server.events.SingleThreadExecutor.lambda$onMessage$1(SingleThreadExecutor.java:85)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> Once the DN's are restarted they are registered fine
> {code:java}
> 2018-10-09 23:35:10,239 INFO org.apache.hadoop.hdds.scm.node.SCMNodeManager: Data node with ID: 5086b8e0-1aa4-4c50-b4db-859f9a9519f7 Registered.
> 2018-10-09 23:35:13,786 INFO org.apache.hadoop.hdds.scm.node.SCMNodeManager: Data node with ID: 1baa3f7c-d85a-4ab5-b005-d2a740233e32 Registered.
> 2018-10-09 23:35:19,407 INFO org.apache.hadoop.hdds.scm.node.SCMNodeManager: Data node with ID: e9c4a12f-db11-4e19-98fc-d1ec7b9d1aeb Registered.
> 2018-10-09 23:35:33,357 INFO org.apache.hadoop.hdds.scm.node.SCMNodeManager: Data node with ID: cf462de6-6709-48ef-b822-f9270e13bc6f Registered.{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org