Cloudera CDH/CDP 및 Hadoop EcoSystem, Semantic IoT등의 개발/운영 기술을 정리합니다. gooper@gooper.com로 문의 주세요.
서버#1에서 namenode를 format했는데 다른 서버에서 namenode를 format하게 되면 경고 메세지가 나오지만 'Y'로 응답하고 진행하면
cluster ID등의 바뀌게 되므로 아래와 같은 문제점이 발생한다.
이때는 다시 format을 해주고 데몬을 전부 재기동을 하며 각DataNode에서는 data디렉토리를 지우고 DataNode를 다시 기동해준다.
*DataNode 각각에서 해주어야 하는작업(필요시 여러번 반복해준다)
- sbin/hadoop-daemon.sh stop datanode
- rm -r /engine/bigdata/hadoop-2.7.2/dfs/datanode (필요시 : mkdir /engine/bigdata/hadoop-2.7.2/dfs/ )
- sbin/hadoop-daemon.sh start datanode
-------------------- Namenode가 다운되는 문제의 오류내용 #1---
XXX.XXX.XXX.32:8485: Incompatible namespaceID for journal Storage Directory /engine/bigdata/hadoop/journal/data/mycluster: NameNode has nsId 1574650046 but storage has nsId 18123
69513
at org.apache.hadoop.hdfs.qjournal.server.JNStorage.checkConsistentNamespace(JNStorage.java:234)
at org.apache.hadoop.hdfs.qjournal.server.Journal.newEpoch(Journal.java:289)
at org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.newEpoch(JournalNodeRpcServer.java:135)
at org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolServerSideTranslatorPB.newEpoch(QJournalProtocolServerSideTranslatorPB.java:133)
at org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos$QJournalProtocolService$2.callBlockingMethod(QJournalProtocolProtos.java:25417)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
at java.security.AccessController.doPrivileged(AccessController.java:377)
at javax.security.auth.Subject.doAs(Subject.java:572)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
at org.apache.hadoop.hdfs.qjournal.client.QuorumException.create(QuorumException.java:81)
at org.apache.hadoop.hdfs.qjournal.client.QuorumCall.rethrowException(QuorumCall.java:223)
at org.apache.hadoop.hdfs.qjournal.client.AsyncLoggerSet.waitForWriteQuorum(AsyncLoggerSet.java:142)
--------------------DataNode가 다운되는 문제의 오류내용 #2---
java.io.IOException: Cluster IDs not matched: dn cid=CID-8395f825-0009-4668-862b-3a9e8d184b54 but ns cid=CID-732a108d-6a42-44e4-9b2a-313750cc4c35; bpid=BP-1493757477-XXX.XXX.XXX.32-1474
523575034
at org.apache.hadoop.hdfs.server.datanode.DataNode.setClusterId(DataNode.java:717)
at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1316)
at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:317)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:223)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:802)
at java.lang.Thread.run(Thread.java:801)
2016-09-22 22:51:28,082 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool service for: Block pool <registering> (Datanode Uuid unassigned) service to IUDGTMP01/XXX.XXX.XXX.31:9000
2016-09-22 22:51:28,082 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool service for: Block pool <registering> (Datanode Uuid unassigned) service to IUDGTMP02/XXX.XXX.XXX.32:9000
2016-09-22 22:51:28,190 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Removed Block pool <registering> (Datanode Uuid unassigned)
2016-09-22 22:51:30,197 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Exiting Datanode
2016-09-22 22:51:30,199 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 0
2016-09-22 22:51:30,201 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at IUDGTMP02/XXX.XXX.XXX.XXX