Cloudera CDH/CDP 및 Hadoop EcoSystem, Semantic IoT등의 개발/운영 기술을 정리합니다. gooper@gooper.com로 문의 주세요.
hadoop클러스터를 구성 하던 서버중 HA를 담당하는 서버의 hostname등이 변경되었을때는 "hadoop-daemon.sh start zkfc"를 수행할때
아래와 같은 오류가 발생할 수 있는데 zookeeper내의 /hadoop-ha/mycluster 노드에 있는 정보가 변경된 사항을 반영하지 못해서 문제가
발생한것이다.
이때는 "hdfs zkfc -formatZK"를 실행하여 ZKFC정보를 재생성해준다.
-----------------hdfs zkfc -formatZK실행 로그----------------
16/07/29 19:30:43 INFO zookeeper.ZooKeeper: Client environment:java.library.path=:/svc/apps/sda/bin/hadoop/hadoop/lib/native
16/07/29 19:30:43 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
16/07/29 19:30:43 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
16/07/29 19:30:43 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
16/07/29 19:30:43 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
16/07/29 19:30:43 INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.32-573.22.1.el6.x86_64
16/07/29 19:30:43 INFO zookeeper.ZooKeeper: Client environment:user.name=root
16/07/29 19:30:43 INFO zookeeper.ZooKeeper: Client environment:user.home=/root
16/07/29 19:30:43 INFO zookeeper.ZooKeeper: Client environment:user.dir=/home/gooper/svc/apps/sda/bin/hadoop/hadoop-2.7.2/bin
16/07/29 19:30:43 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=gsda1:2181,gsda2:2181,gsda3:2181 sessionTimeout=5000 watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@1e178745
16/07/29 19:30:43 INFO zookeeper.ClientCnxn: Opening socket connection to server sda2/XXX.XXX.XXX.44:2181. Will not attempt to authenticate using SASL (unknown error)
16/07/29 19:30:43 INFO zookeeper.ClientCnxn: Socket connection established to sda2/XXX.XXX.XXX.44:2181, initiating session
16/07/29 19:30:43 INFO zookeeper.ClientCnxn: Session establishment complete on server sda2/XXX.XXX.XXX.44:2181, sessionid = 0x25634f0fabb0300, negotiated timeout = 5000
16/07/29 19:30:43 INFO ha.ActiveStandbyElector: Session connected.
===============================================
The configured parent znode /hadoop-ha/mycluster already exists.
Are you sure you want to clear all failover information from
ZooKeeper?
WARNING: Before proceeding, ensure that all HDFS services and
failover controllers are stopped!
===============================================
Proceed formatting /hadoop-ha/mycluster? (Y or N) Y
16/07/29 19:36:25 INFO ha.ActiveStandbyElector: Recursively deleting /hadoop-ha/mycluster from ZK...
16/07/29 19:36:25 INFO ha.ActiveStandbyElector: Successfully deleted /hadoop-ha/mycluster from ZK.
16/07/29 19:36:26 INFO ha.ActiveStandbyElector: Successfully created /hadoop-ha/mycluster in ZK.
16/07/29 19:36:26 INFO zookeeper.ZooKeeper: Session: 0x25634f0fabb0300 closed
16/07/29 19:36:26 INFO zookeeper.ClientCnxn: EventThread shut down
------------------오류내용-------------
2016-07-29 18:33:32,857 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.library.path=/home/gooper/svc/apps/sda/bin/hadoop/hadoop-2.7.2/lib
2016-07-29 18:33:32,857 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
2016-07-29 18:33:32,857 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2016-07-29 18:33:32,857 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.name=Linux
2016-07-29 18:33:32,857 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.arch=amd64
2016-07-29 18:33:32,857 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.version=2.6.32-573.12.1.el6.x86_64
2016-07-29 18:33:32,857 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.name=root
2016-07-29 18:33:32,857 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.home=/root
2016-07-29 18:33:32,857 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.dir=/home/gooper/svc/apps/sda/bin/hadoop/hadoop-2.7.2
2016-07-29 18:33:32,858 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=gsda1:2181,gsda2:2181,gsda3:2181 sessionTimeout=5000 watcher=org.apache.hadoop.ha.Ac
tiveStandbyElector$WatcherWithClientRef@52bf72b5
2016-07-29 18:33:32,936 FATAL org.apache.hadoop.hdfs.tools.DFSZKFailoverController: Got a fatal error, exiting now
java.net.UnknownHostException: gsda3: unknown error
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
at java.net.InetAddress.getAllByName(InetAddress.java:1192)
at java.net.InetAddress.getAllByName(InetAddress.java:1126)
at org.apache.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:61)
at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:445)
at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:380)
at org.apache.hadoop.ha.ActiveStandbyElector.getNewZooKeeper(ActiveStandbyElector.java:631)
at org.apache.hadoop.ha.ActiveStandbyElector.createConnection(ActiveStandbyElector.java:775)
at org.apache.hadoop.ha.ActiveStandbyElector.<init>(ActiveStandbyElector.java:229)
at org.apache.hadoop.ha.ZKFailoverController.initZK(ZKFailoverController.java:350)
at org.apache.hadoop.ha.ZKFailoverController.doRun(ZKFailoverController.java:191)
at org.apache.hadoop.ha.ZKFailoverController.access$000(ZKFailoverController.java:61)
at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:172)
at org.apache.hadoop.ha.ZKFailoverController$1.run(ZKFailoverController.java:168)
at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:415)
at org.apache.hadoop.ha.ZKFailoverController.run(ZKFailoverController.java:168)
at org.apache.hadoop.hdfs.tools.DFSZKFailoverController.main(DFSZKFailoverController.java:181)
