메뉴 건너뛰기

Cloudera, BigData, Semantic IoT, Hadoop, NoSQL

Cloudera CDH/CDP 및 Hadoop EcoSystem, Semantic IoT등의 개발/운영 기술을 정리합니다. gooper@gooper.com로 문의 주세요.


./hadoop-daemon.sh start namenode로 namenode기동시 EditLog의 custerId, namespaceId가 달라서 오류가 발생하면
"bin/hadoop namenode -recover"를 실행하면 문제를 해결해준다.

recover를 실행하면 아래의 짊문을 하는데 s를 선택한다.
Enter 'c' to continue, skipping the bad section in the log
Enter 's' to stop reading the edit log here, abandoning any later edits
Enter 'q' to quit without saving
Enter 'a' to always select the first choice in the future without prompting. (c/s/q/a)



-------------------------오류내용-----------------
16/09/24 10:57:03 INFO namenode.FSImageFormatPBINode: Loading 1 INodes.
16/09/24 10:57:03 INFO namenode.FSImageFormatProtobuf: Loaded FSImage in 0 seconds.
16/09/24 10:57:03 INFO namenode.FSImage: Loaded image for txid 0 from /engine/bigdata/hadoop/dfs/namenode/current/fsimage_                                              0000000000000000000
16/09/24 10:57:03 INFO namenode.FSImage: Reading org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream@830730                                              f8 expecting start txid #1
16/09/24 10:57:03 INFO namenode.FSImage: Start loading edits file http://IUDGTMP03:8480/getJournal?jid=mycluster&segmentTx                                              Id=1&storageInfo=-63%3A1812369513%3A0%3ACID-732a108d-6a42-44e4-9b2a-313750cc4c35, http://IUDGTMP04:8480/getJournal?jid=myc                                              luster&segmentTxId=1&storageInfo=-63%3A1812369513%3A0%3ACID-732a108d-6a42-44e4-9b2a-313750cc4c35
16/09/24 10:57:03 INFO namenode.EditLogInputStream: Fast-forwarding stream 'http://IUDGTMP03:8480/getJournal?jid=mycluster                                              &segmentTxId=1&storageInfo=-63%3A1812369513%3A0%3ACID-732a108d-6a42-44e4-9b2a-313750cc4c35, http://IUDGTMP04:8480/getJourn                                              al?jid=mycluster&segmentTxId=1&storageInfo=-63%3A1812369513%3A0%3ACID-732a108d-6a42-44e4-9b2a-313750cc4c35' to transaction                                               ID 1
16/09/24 10:57:03 INFO namenode.EditLogInputStream: Fast-forwarding stream 'http://IUDGTMP03:8480/getJournal?jid=mycluster                                              &segmentTxId=1&storageInfo=-63%3A1812369513%3A0%3ACID-732a108d-6a42-44e4-9b2a-313750cc4c35' to transaction ID 1
16/09/24 10:57:03 ERROR namenode.EditLogInputStream: caught exception initializing http://IUDGTMP03:8480/getJournal?jid=my                                              cluster&segmentTxId=1&storageInfo=-63%3A1812369513%3A0%3ACID-732a108d-6a42-44e4-9b2a-313750cc4c35
org.apache.hadoop.hdfs.server.namenode.TransferFsImage$HttpGetFailedException: Fetch of http://IUDGTMP03:8480/getJournal?j                                              id=mycluster&segmentTxId=1&storageInfo=-63%3A1812369513%3A0%3ACID-732a108d-6a42-44e4-9b2a-313750cc4c35 failed with status                                               code 403
Response message:
This node has namespaceId '671989181 and clusterId 'CID-8395f825-0009-4668-862b-3a9e8d184b54' but the requesting node expe                                              cted '1812369513' and 'CID-732a108d-6a42-44e4-9b2a-313750cc4c35'
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:468)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:456)
        at java.security.AccessController.doPrivileged(AccessController.java:377)
        at javax.security.auth.Subject.doAs(Subject.java:572)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
        at org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:448)
        at org.apache.hadoop.security.SecurityUtil.doAsCurrentUser(SecurityUtil.java:442)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog.getInputStream(EditLogFileInputStream.java                                              :455)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.init(EditLogFileInputStream.java:141)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOpImpl(EditLogFileInputStream.java:192)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOp(EditLogFileInputStream.java:250)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
        at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
        at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:190)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:143)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:837)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:692)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:294)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:975)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:681)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.doRecovery(NameNode.java:1382)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1472)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1554)
16/09/24 10:57:03 ERROR namenode.EditLogInputStream: Got error reading edit log input stream http://IUDGTMP03:8480/getJour                                              nal?jid=mycluster&segmentTxId=1&storageInfo=-63%3A1812369513%3A0%3ACID-732a108d-6a42-44e4-9b2a-313750cc4c35; failing over                                               to edit log http://IUDGTMP04:8480/getJournal?jid=mycluster&segmentTxId=1&storageInfo=-63%3A1812369513%3A0%3ACID-732a108d-6                                              a42-44e4-9b2a-313750cc4c35
org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream$PrematureEOFException: got premature end-of-file at txi                                              d 0; expected file to go up to 2
        at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:194)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
        at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:190)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:143)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:837)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:692)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:294)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:975)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:681)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.doRecovery(NameNode.java:1382)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1472)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1554)
16/09/24 10:57:03 INFO namenode.EditLogInputStream: Fast-forwarding stream 'http://IUDGTMP04:8480/getJournal?jid=mycluster                                              &segmentTxId=1&storageInfo=-63%3A1812369513%3A0%3ACID-732a108d-6a42-44e4-9b2a-313750cc4c35' to transaction ID 1
16/09/24 10:57:03 ERROR namenode.EditLogInputStream: caught exception initializing http://IUDGTMP04:8480/getJournal?jid=my                                              cluster&segmentTxId=1&storageInfo=-63%3A1812369513%3A0%3ACID-732a108d-6a42-44e4-9b2a-313750cc4c35
org.apache.hadoop.hdfs.server.namenode.TransferFsImage$HttpGetFailedException: Fetch of http://IUDGTMP04:8480/getJournal?j                                              id=mycluster&segmentTxId=1&storageInfo=-63%3A1812369513%3A0%3ACID-732a108d-6a42-44e4-9b2a-313750cc4c35 failed with status                                               code 403
Response message:
This node has namespaceId '671989181 and clusterId 'CID-8395f825-0009-4668-862b-3a9e8d184b54' but the requesting node expe                                              cted '1812369513' and 'CID-732a108d-6a42-44e4-9b2a-313750cc4c35'
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:468)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:456)
        at java.security.AccessController.doPrivileged(AccessController.java:377)
        at javax.security.auth.Subject.doAs(Subject.java:572)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
        at org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:448)
        at org.apache.hadoop.security.SecurityUtil.doAsCurrentUser(SecurityUtil.java:442)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog.getInputStream(EditLogFileInputStream.java                                              :455)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.init(EditLogFileInputStream.java:141)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOpImpl(EditLogFileInputStream.java:192)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOp(EditLogFileInputStream.java:250)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
        at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
        at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:190)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:143)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:837)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:692)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:294)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:975)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:681)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.doRecovery(NameNode.java:1382)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1472)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1554)
16/09/24 10:57:03 ERROR namenode.FSImage: Error replaying edit log at offset 0.  Expected transaction ID was 1
org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream$PrematureEOFException: got premature end-of-file at txi                                              d 0; expected file to go up to 2
        at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:194)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
        at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:190)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:143)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:837)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:692)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:294)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:975)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:681)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.doRecovery(NameNode.java:1382)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1472)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1554)
16/09/24 10:57:03 ERROR namenode.MetaRecoveryContext: We failed to read txId 1

Enter 'c' to continue, skipping the bad section in the log
Enter 's' to stop reading the edit log here, abandoning any later edits
Enter 'q' to quit without saving
Enter 'a' to always select the first choice in the future without prompting. (c/s/q/a)
s
16/09/24 10:57:52 WARN namenode.MetaRecoveryContext: Stopped reading edit log at 0/-1
16/09/24 10:57:52 INFO namenode.FSImage: Edits file http://IUDGTMP03:8480/getJournal?jid=mycluster&segmentTxId=1&storageInfo=-63%3A1812369513%3A0%3ACI                  D-732a108d-6a42-44e4-9b2a-313750cc4c35, http://IUDGTMP04:8480/getJournal?jid=mycluster&segmentTxId=1&storageInfo=-63%3A1812369513%3A0%3ACID-732a108d-6                  a42-44e4-9b2a-313750cc4c35 of size -1 edits # 0 loaded in 49 seconds
16/09/24 10:57:52 INFO namenode.FSImage: Reading org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream@fc20cc3e expecting start txid #3
16/09/24 10:57:52 INFO namenode.FSImage: Start loading edits file http://IUDGTMP04:8480/getJournal?jid=mycluster&segmentTxId=3&storageInfo=-63%3A18123                  69513%3A0%3ACID-732a108d-6a42-44e4-9b2a-313750cc4c35, http://IUDGTMP03:8480/getJournal?jid=mycluster&segmentTxId=3&storageInfo=-63%3A1812369513%3A0%3A                  CID-732a108d-6a42-44e4-9b2a-313750cc4c35
16/09/24 10:57:52 INFO namenode.EditLogInputStream: Fast-forwarding stream 'http://IUDGTMP04:8480/getJournal?jid=mycluster&segmentTxId=3&storageInfo=-                  63%3A1812369513%3A0%3ACID-732a108d-6a42-44e4-9b2a-313750cc4c35, http://IUDGTMP03:8480/getJournal?jid=mycluster&segmentTxId=3&storageInfo=-63%3A1812369                  513%3A0%3ACID-732a108d-6a42-44e4-9b2a-313750cc4c35' to transaction ID 1
16/09/24 10:57:52 INFO namenode.EditLogInputStream: Fast-forwarding stream 'http://IUDGTMP04:8480/getJournal?jid=mycluster&segmentTxId=3&storageInfo=-                  63%3A1812369513%3A0%3ACID-732a108d-6a42-44e4-9b2a-313750cc4c35' to transaction ID 1
16/09/24 10:57:52 ERROR namenode.EditLogInputStream: caught exception initializing http://IUDGTMP04:8480/getJournal?jid=mycluster&segmentTxId=3&storag                  eInfo=-63%3A1812369513%3A0%3ACID-732a108d-6a42-44e4-9b2a-313750cc4c35
org.apache.hadoop.hdfs.server.namenode.TransferFsImage$HttpGetFailedException: Fetch of http://IUDGTMP04:8480/getJournal?jid=mycluster&segmentTxId=3&s                  torageInfo=-63%3A1812369513%3A0%3ACID-732a108d-6a42-44e4-9b2a-313750cc4c35 failed with status code 403
Response message:
This node has namespaceId '671989181 and clusterId 'CID-8395f825-0009-4668-862b-3a9e8d184b54' but the requesting node expected '1812369513' and 'CID-7                  32a108d-6a42-44e4-9b2a-313750cc4c35'
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:468)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:456)
        at java.security.AccessController.doPrivileged(AccessController.java:377)
        at javax.security.auth.Subject.doAs(Subject.java:572)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
        at org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:448)
        at org.apache.hadoop.security.SecurityUtil.doAsCurrentUser(SecurityUtil.java:442)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog.getInputStream(EditLogFileInputStream.java:455)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.init(EditLogFileInputStream.java:141)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOpImpl(EditLogFileInputStream.java:192)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOp(EditLogFileInputStream.java:250)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
        at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
        at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:190)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:143)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:837)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:692)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:294)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:975)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:681)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.doRecovery(NameNode.java:1382)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1472)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1554)
16/09/24 10:57:52 ERROR namenode.EditLogInputStream: Got error reading edit log input stream http://IUDGTMP04:8480/getJournal?jid=mycluster&segmentTxI                  d=3&storageInfo=-63%3A1812369513%3A0%3ACID-732a108d-6a42-44e4-9b2a-313750cc4c35; failing over to edit log http://IUDGTMP03:8480/getJournal?jid=myclust                  er&segmentTxId=3&storageInfo=-63%3A1812369513%3A0%3ACID-732a108d-6a42-44e4-9b2a-313750cc4c35
org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream$PrematureEOFException: got premature end-of-file at txid 0; expected file to go up                   to 127
        at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:194)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
        at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:190)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:143)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:837)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:692)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:294)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:975)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:681)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.doRecovery(NameNode.java:1382)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1472)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1554)
16/09/24 10:57:52 INFO namenode.EditLogInputStream: Fast-forwarding stream 'http://IUDGTMP03:8480/getJournal?jid=mycluster&segmentTxId=3&storageInfo=-                  63%3A1812369513%3A0%3ACID-732a108d-6a42-44e4-9b2a-313750cc4c35' to transaction ID 1
16/09/24 10:57:52 ERROR namenode.EditLogInputStream: caught exception initializing http://IUDGTMP03:8480/getJournal?jid=mycluster&segmentTxId=3&storag                  eInfo=-63%3A1812369513%3A0%3ACID-732a108d-6a42-44e4-9b2a-313750cc4c35
org.apache.hadoop.hdfs.server.namenode.TransferFsImage$HttpGetFailedException: Fetch of http://IUDGTMP03:8480/getJournal?jid=mycluster&segmentTxId=3&s                  torageInfo=-63%3A1812369513%3A0%3ACID-732a108d-6a42-44e4-9b2a-313750cc4c35 failed with status code 403
Response message:
This node has namespaceId '671989181 and clusterId 'CID-8395f825-0009-4668-862b-3a9e8d184b54' but the requesting node expected '1812369513' and 'CID-7                  32a108d-6a42-44e4-9b2a-313750cc4c35'
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:468)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:456)
        at java.security.AccessController.doPrivileged(AccessController.java:377)
        at javax.security.auth.Subject.doAs(Subject.java:572)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
        at org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:448)
        at org.apache.hadoop.security.SecurityUtil.doAsCurrentUser(SecurityUtil.java:442)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog.getInputStream(EditLogFileInputStream.java:455)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.init(EditLogFileInputStream.java:141)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOpImpl(EditLogFileInputStream.java:192)
        at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOp(EditLogFileInputStream.java:250)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
        at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
        at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:190)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:143)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:837)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:692)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:294)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:975)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:681)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.doRecovery(NameNode.java:1382)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1472)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1554)
16/09/24 10:57:52 ERROR namenode.FSImage: Error replaying edit log at offset 0.  Expected transaction ID was 3
org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream$PrematureEOFException: got premature end-of-file at txid 0; expected file to go up                   to 127
        at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:194)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
        at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
        at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:190)
        at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:143)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:837)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:692)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:294)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:975)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:681)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.doRecovery(NameNode.java:1382)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1472)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1554)
16/09/24 10:57:52 ERROR namenode.MetaRecoveryContext: We failed to read txId 3

Enter 'c' to continue, skipping the bad section in the log
Enter 's' to stop reading the edit log here, abandoning any later edits
Enter 'q' to quit without saving
Enter 'a' to always select the first choice in the future without prompting. (c/s/q/a)
s
16/09/24 10:57:54 WARN namenode.MetaRecoveryContext: Stopped reading edit log at 0/-1
16/09/24 10:57:54 INFO namenode.FSImage: Edits file http://IUDGTMP04:8480/getJournal?jid=mycluster&segmentTxId=3&storageInfo=-63%3A1812369513%3A0%3ACI                  D-732a108d-6a42-44e4-9b2a-313750cc4c35, http://IUDGTMP03:8480/getJournal?jid=mycluster&segmentTxId=3&storageInfo=-63%3A1812369513%3A0%3ACID-732a108d-6                  a42-44e4-9b2a-313750cc4c35 of size -1 edits # 0 loaded in 1 seconds
16/09/24 10:57:54 INFO namenode.FSNamesystem: Need to save fs image? false (staleImage=true, haEnabled=true, isRollingUpgrade=false)
16/09/24 10:57:54 INFO namenode.NameCache: initialized with 0 entries 0 lookups
16/09/24 10:57:54 INFO namenode.FSNamesystem: Finished loading FSImage in 52039 msecs
16/09/24 10:57:54 INFO namenode.FSImage: Save namespace ...
16/09/24 10:57:54 INFO namenode.NNStorageRetentionManager: Going to retain 2 images with txid >= 0
16/09/24 10:57:55 INFO namenode.MetaRecoveryContext: RECOVERY COMPLETE
16/09/24 10:57:55 INFO namenode.FSNamesystem: Stopping services started for active state
16/09/24 10:57:55 INFO namenode.FSNamesystem: Stopping services started for standby state
16/09/24 10:57:55 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at IUDGTMP02/XXX.XXX.XXX.32
************************************************************/

번호 제목 날짜 조회 수
304 How-to: Build a Complex Event Processing App on Apache Spark and Drools file 2016.10.31 552
303 How-to: Tune Your Apache Spark Jobs (Part 2) file 2016.10.31 478
302 mybatis와 spring을 org.apache.commons.dbcp2.BasicDataSource의 DataSource로 연동할때 DB설정(참고) 2016.10.31 1257
301 Caused by: java.sql.SQLNonTransientConnectionException: Could not read resultset: unexpected end of stream, read 0 bytes from 4 오류시 확인/조치할 내용 2016.10.31 4387
300 VisualVM 1.3.9을 이용한 spark-submit JVM 모니터링을 위한 설정및 spark-submit실행 옵션 2016.10.28 2414
299 VisualVM 1.3.9을 이용한 JVM 모니터링 file 2016.10.27 891
298 운영중인 상태에서 kafka topic삭제하고 재생성하여 처리되지 않은 메세지 모두 삭제하기 2016.10.24 1242
297 producer / consumer구현시 설정 옵션 설명 2016.10.19 777
296 java.lang.OutOfMemoryError: unable to create new native thread오류 발생지 조치사항 2016.10.17 1205
295 동시에 많은 요청이 endpoint로 몰려서java.net.NoRouteToHostException가 발생하는 경우의 처리방법 2016.10.17 811
294 AIX 7.1에 Python 2.7.11설치하기 2016.10.06 1254
293 프로그래밍 언어별 딥러닝 라이브러리 정리 file 2016.10.05 692
292 모두를 위한 머신러닝과 딥러닝의 강의 file 2016.09.27 750
291 DBCP Datasource(org.apache.commons.dbcp.BasicDataSource) 설정 및 속성 설명 2016.09.26 966
290 AIX 7.1에서 hive실행시 "hive: line 86: readlink: command not found" 오류가 발생시 임시 조치사항 2016.09.25 1014
289 hive기동시 Caused by: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D 오류 발생시 조치사항 2016.09.25 1054
288 schema설정없이 hive를 최초에 실행했을때 발생하는 오류메세지및 처리방법 2016.09.25 1492
287 파일끝에 붙는 ^M 일괄 지우기(linux, unix(AIX)) 혹은 파일내에 있는 ^M지우기 2016.09.24 238
286 AIX 7.1에 MariaDB 10.2 소스 설치 2016.09.24 2938
» ./hadoop-daemon.sh start namenode로 namenode기동시 EditLog의 custerId, namespaceId가 달라서 발생하는 오류 해결방법 2016.09.24 506
위로