集群的规划
[1] 集群规划
| IP | 角色 |
|---|---|
| 192.168.130.3 | NameNode1主节点, ResourceManager1主节点, Journalnode |
| 192.168.130.4 | NameNode2备用主节点, ResourceManager2备用主节点, Journalnode |
| 192.168.130.5 | DataNode1, NodeManager1 |
[2] 相关版本
| JDK | Hadoop | Zookeeper |
|---|---|---|
| /home/jdk1.8.0_211 | /home/hadoop-2.8.5 | /home/zookeeper3.4.14 |
[3] 登录方式1
2sudo su
ssh -p 22 user@192.168.130.*
安装Zookeeper集群
首先在192.168.130.3上：
[1] 修改配置文件/conf/zoo.cfg:1
dataDir=/home/zookeeper3.4.14/tmp
在最后一行添加：1
2
3server.1=192.168.130.3:2888:3888
server.2=192.168.130.4:2888:3888
server.3=192.168.130.5:2888:3888
[2] 修改myid配置文件
在/home/zookeeper3.4.14目录下创建一个/tmp/myid空文件1
2
3
4mkdir tmp
cd tmp
touch myid
echo 1 > myid
[3] 将配置好的zookeeper拷贝到其他节点1
2scp -r /home/zookeeper3.4.14/ user@192.168.130.4:/home/
scp -r /home/zookeeper3.4.14/ user@192.168.130.5:/home/
进入192.168.130.4和192.168.130.5两台机器中，找到myid文件，修改各自的myid文件：在192.168.130.4中输入2，192.168.130.5中输入3
安装Hadoop集群
首先在192.168.130.3上：
[1] 修改core-site.xml1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://ns1</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/sunzesong/hadoop-2.8.5/tmp</value>
</property>
<property>
<name>ha.zookeeper.quorum</name>
<value>192.168.130.3:2181, 192.168.130.4:2181, 192.168.130.5:2181</value>
</property>
</configuration>
[2] 修改hdfs-site.xml1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72<configuration>
<property>
<name>dfs.nameservices</name>
<value>ns1</value>
</property>
<property>
<name>dfs.ha.namenodes.ns1</name>
<value>nn1,nn2</value>
</property>
<property>
<name>dfs.namenode.rpc-address.ns1.nn1</name>
<value>192.168.130.3:9000</value>
</property>
<property>
<name>dfs.namenode.http-address.ns1.nn1</name>
<value>192.168.130.3:50070</value>
</property>
<property>
<name>dfs.namenode.rpc-address.ns1.nn2</name>
<value>192.168.130.4:9000</value>
</property>
<property>
<name>dfs.namenode.http-address.ns1.nn2</name>
<value>192.168.130.4:50070</value>
</property>
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://192.168.130.3:8485;192.168.130.4:8485;/ns1</value>
</property>
<property>
<name>dfs.journalnode.edits.dir</name>
<value>/home/sunzesong/hadoop-2.8.5/journal</value>
</property>
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.client.failover.proxy.provider.ns1</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<!-- 配置隔离机制方法，多个机制用换行分割，即每个机制暂用一行-->
<property>
<name>dfs.ha.fencing.methods</name>
<value>
sshfence
shell(/bin/true)
</value>
</property>
<!-- 使用sshfence隔离机制时需要ssh免登陆 -->
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/root/.ssh/id_rsa</value>
</property>
<!-- 配置sshfence隔离机制超时时间 -->
<property>
<name>dfs.ha.fencing.ssh.connect-timeout</name>
<value>30000</value>
</property>
</configuration>
[3] 将配置好的hadoop拷贝到其他节点1
2scp -r /home/hadoop-2.8.5/ user@192.168.130.4:/home/
scp -r /home/hadoop-2.8.5/ user@192.168.130.5:/home/
启动Zookeeper集群和Hadoop集群
[1] 启动Zookeeper集群
在每一台机器上：1
./zkServer.sh start
[2] 启动journalnode
在192.168.130.3和192.168.130.4两台节点上启动journalnode节点：1
hadoop-daemon.sh start journalnode
[3] 格式化
格式化HDFS：hdfs namenode -format
格式化zookeeper：hdfs zkfc -formatZK手动输入，不可以复制粘贴
[4] 启动Hadoop集群
在192.168.130.3：start-dfs.sh
输出：1
2
3
4
5
6
7
8
9
10Starting namenodes on [192.168.130.3 192.168.130.4]
192.168.130.3: starting namenode, logging to /home/hadoop-2.8.5/logs/hadoop-namenode-s03.out
192.168.130.4: starting namenode, logging to /home/hadoop-2.8.5/logs/hadoop-namenode-pc04.out
192.168.130.5: starting datanode, logging to /home/hadoop-2.8.5/logs/hadoop-datanode-s5.out
Starting journal nodes [192.168.130.3 192.168.130.4 ]
192.168.130.3: starting journalnode, logging to /home/hadoop-2.8.5/logs/hadoop-journalnode-s03.out
192.168.130.4: starting journalnode, logging to /home/hadoop-2.8.5/logs/hadoop-journalnode-pc04.out
Starting ZK Failover Controllers on NN hosts [192.168.130.3 192.168.130.4]
192.168.130.3: starting zkfc, logging to /home/hadoop-2.8.5/logs/hadoop-zkfc-s03.out
192.168.130.4: starting zkfc, logging to /home/hadoop-2.8.5/logs/hadoop-zkfc-pc04.out
参考资料
[1] https://yq.aliyun.com/articles/606740
[2] https://www.cnblogs.com/yousen/p/5191075.html
[3] https://www.cnblogs.com/shizhijie/p/9878078.html