메뉴 건너뛰기

Cloudera, BigData, Semantic IoT, Hadoop, NoSQL

Cloudera CDH/CDP 및 Hadoop EcoSystem, Semantic IoT등의 개발/운영 기술을 정리합니다. gooper@gooper.com로 문의 주세요.


기타 ubuntu에 hadoop 2.0.5설치하기

총관리자 2013.12.16 22:09 조회 수 : 1977

출처 : http://www.spikyjohn.com/cribsheets/20130609_hadoopinstall.html

 

Just the command lines to get hadoop 2 installed on Ubuntu. These are all cribbed from the following source notes, and I am preserving them here for my own benefit so I can quickly repeat what I did. Note many of these instructions are also in the main hadoop docs from apache.

Source material

Use Michael-noll's guide for version 1 & ssh
http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster/
http://hadoop.apache.org/docs/r1.1.2/single_node_setup.html

Or this one for Hadoop 2
http://jugnu-life.blogspot.com/2012/05/hadoop-20-install-tutorial-023x.html
http://hadoop.apache.org/docs/r2.0.5-alpha/

Create the hadoop user and ssh

sudo apt-get install openssh-server openssh-client

sudo addgroup hadoop
sudo adduser --ingroup hadoop hduser
su - hduser

If you cannot ssh to localhost without a passphrase, execute the following commands:
ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

Testing your SSH
ssh localhost
Say yes
#exit

Get hadoop all set up

As the hduser, after downloading the tar

tar -xvf hadoop-2.0.5-alpha.tar.gz
ln -s hadoop-2.0.5-alpha hadoop
#edit .bashrc
export JAVA_HOME=/usr/lib/jvm/jdk1.7.0_21/
export HADOOP_PREFIX="/home/hduser/hadoop"
export PATH=$PATH:$HADOOP_PREFIX/bin
export PATH=$PATH:$HADOOP_PREFIX/sbin

export HADOOP_MAPRED_HOME=${HADOOP_PREFIX}
export HADOOP_COMMON_HOME=${HADOOP_PREFIX}
export HADOOP_HDFS_HOME=${HADOOP_PREFIX}
export YARN_HOME=${HADOOP_PREFIX}

Stolen entirely from JJ, but with path changed for my Ubuntu

Stolen from http://jugnu-life.blogspot.com/2012/05/hadoop-20-install-tutorial-023x.html Please click on his blog.

Login again so bash has paths above. In Hadoop 2.x version /etc/hadoop is the default conf directory. We need to modify / create following property files in the /etc/hadoop directory

cd ~
mkdir -p /home/hduser/workspace/hadoop_space/hadoop23/dfs/name;mkdir -p /home/hduser/workspace/hadoop_space/hadoop23/dfs/data;mkdir -p /home/hduser/workspace/hadoop_space/hadoop23/mapred/system;mkdir -p /home/hduser/workspace/hadoop_space/hadoop23/mapred/local

Edit core-site.xml with following contents

<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:8020</value>
<description>The name of the default file system. Either the literal string "local" or a host:port for NDFS.</description>
<final>true</final>
</property>
</configuration>

Edit hdfs-site.xml with following contents

<configuration>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/home/hduser/workspace/hadoop_space/hadoop23/dfs/name</value>
<description>Determines where on the local filesystem the DFS name node
should store the name table. If this is a comma-delimited list of directories then the name table is replicated in all of the
directories, for redundancy. </description>
<final>true</final>
</property>

<property>
<name>dfs.datanode.data.dir</name>
<value>file:/home/hduser/workspace/hadoop_space/hadoop23/dfs/data</value>
<description>Determines where on the local filesystem an DFS data node
should store its blocks. If this is a comma-delimited list of directories, then data will be stored in all named
directories, typically on different devices. Directories that do not exist are ignored.
</description>
<final>true</final>
</property>

<property>
<name>dfs.replication</name>
<value>1</value>
</property>

<property>
<name>dfs.permissions</name>
<value>false</value>
</property>

</configuration>

The path
file:/home/hduser/workspace/hadoop_space/hadoop23/dfs/name AND
file:/home/hduser/workspace/hadoop_space/hadoop23/dfs/data
are some folders in your computer which would give space to store data and name edit files

Path should be specified as URI
Create a file mapred-site.xml inside /etc/hadoop with following contents

<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>

<property>
<name>mapred.system.dir</name>
<value>file:/home/hduser/workspace/hadoop_space/hadoop23/mapred/system</value>
<final>true</final>
</property>

<property>
<name>mapred.local.dir</name>
<value>file:/home/hduser/workspace/hadoop_space/hadoop23/mapred/local</value>
<final>true</final>
</property>

</configuration>

The path

file:/home/hduser/workspace/hadoop_space/hadoop23/mapred/system AND
file:/home/hduser/workspace/hadoop_space/hadoop23/mapred/local
are some folders in your computer which would give space to store data

Path should be specified as URI

Edit yarn-site.xml with following contents

<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce.shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>

Format the namenode

# hdfs namenode –format

Say Yes and let it complete the format

Time to start the daemons

# hadoop-daemon.sh start namenode
# hadoop-daemon.sh start datanode

You can also start both of them together by

# start-dfs.sh

Start Yarn Daemons

# yarn-daemon.sh start resourcemanager
# yarn-daemon.sh start nodemanager

You can also start all yarn daemons together by

# start-yarn.sh

Time to check if Daemons have started

Enter the command

# jps
2539 NameNode
2744 NodeManager
3075 Jps
3030 DataNode
2691 ResourceManager

Time to launch UI

Open the localhost:8088 to see the Resource Manager page

Done :)

Happy Hadooping :)

번호 제목 날짜 조회 수
81 hadoop 2.6.0에 sqoop2 (1.99.5) server및 client설치 == fail 2015.06.11 1914
80 [sbt] sbt-assembly를 이용하여 실행에 필요한 모든 j라이브러리를 포함한 fat jar파일 만들기 2016.07.11 1929
79 VirtualBox에 ubuntu 설치 하기 (12.10) file 2013.03.04 1948
78 MySQL 다운로드 및 리눅스에서 간단 컴파일 설치 2013.03.08 1950
77 java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error: Unable to deserialize reduce input key from...오류해결방법 2015.06.16 1976
» ubuntu에 hadoop 2.0.5설치하기 2013.12.16 1977
75 통계자료 구할수 있는 곳 2014.04.16 2011
74 List<Map<String, String>>형태의 데이타에서 중복제거 하는 방법 2016.12.23 2046
73 Spark 2.1.1 clustering(5대) 설치(YARN기반) 2016.04.22 2057
72 Hadoop wordcount 소스 작성 file 2013.03.06 2067
71 RHadoop을 통해서 웹사이트의 방문자수를 예측하는 방법 2014.09.26 2075
70 Hadoop 설치 및 시작하기 file 2013.03.06 2077
69 VisualVM 1.3.9을 이용한 spark-submit JVM 모니터링을 위한 설정및 spark-submit실행 옵션 2016.10.28 2122
68 hbase shell 필드 검색 방법 2015.05.24 2126
67 ../depcomp: line 512 exec : g++ : not found 2013.03.08 2146
66 Hive java connection 설정 file 2013.04.01 2170
65 hbase에 필요한 jar들 2013.04.01 2189
64 solr설치및 적용관련 file 2014.09.27 2207
63 W/F수행후 Logs not available for 1. Aggregation may not to complete. 표시되며 로그내용이 보이지 않은 경우 2020.05.08 2217
62 hadoop설치시 참고사항 2013.03.08 2295
위로