Cloudera CDH/CDP 및 Hadoop EcoSystem, Semantic IoT등의 개발/운영 기술을 정리합니다. gooper@gooper.com로 문의 주세요.
datanode기동시 Incompatible clusterIDs오류가 발생하는 이유는 "hadoop namenode -format"이 다시 한번 이루어지면서
namenode는 새로운 ID를 받고 datanode는 새로운 ID를 발급받지 못하여 불일치가 발생한다.
그래서 format전에 /hadoop/dfs/datanode/VERSION파일을 지우고 format하면 해결된다.
---------------------------datanode 로그파일 내용------------------
2016-04-01 10:18:29,872 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /hadoop/dfs/datanode/in_use.lock acquired by nodename 2642@so-db2
2016-04-01 10:18:29,876 WARN org.apache.hadoop.hdfs.server.common.Storage: java.io.IOException: Incompatible clusterIDs in /hadoop/dfs/datanode: namenode clusterID = CID-114223dd-58a8-4145-8359-990ebc169f13; datanode clusterID = CID-0a8e7038-7c96-45ab-bf73-1ad00fcbd192
2016-04-01 10:18:29,878 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for Block pool <registering> (Datanode Uuid unassigned) service to sda2/xxx.xxx.xxx.44:9000. Exiting.
java.io.IOException: All specified directories are failed to load.
at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:478)
at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1358)
at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1323)
at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:317)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:223)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:802)
at java.lang.Thread.run(Thread.java:745)
2016-04-01 10:18:29,878 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for Block pool <registering> (Datanode Uuid unassigned) service to sda1/xxx.xxx.xxx.43:9000. Exiting.
java.io.IOException: Cluster IDs not matched: dn cid=CID-114223dd-58a8-4145-8359-990ebc169f13 but ns cid=CID-0a8e7038-7c96-45ab-bf73-1ad00fcbd192; bpid=BP-503361220-xxx.xxx.xxx.43-1459412244660
at org.apache.hadoop.hdfs.server.datanode.DataNode.setClusterId(DataNode.java:717)
at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1316)
at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:317)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:223)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:802)
at java.lang.Thread.run(Thread.java:745)
2016-04-01 10:18:29,881 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool service for: Block pool <registering> (Datanode Uuid unassigned) service to sda2/xxx.xxx.xxx.44:9000
2016-04-01 10:18:29,881 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool service for: Block pool <registering> (Datanode Uuid unassigned) service to sda1/xxx.xxx.xxx.43:9000
2016-04-01 10:18:29,983 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Removed Block pool <registering> (Datanode Uuid unassigned)
2016-04-01 10:18:31,983 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Exiting Datanode
2016-04-01 10:18:31,986 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 0
2016-04-01 10:18:31,988 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at so-db2/xxx.xxx.xxx.38
************************************************************/