懶人記錄 Hadoop2.7.1 集群搭建過程，懶人hadoop2.7.1

懶人記錄 Hadoop2.7.1 集群搭建過程，懶人hadoop2.7.1

懶人記錄 Hadoop2.7.1 集群搭建過程

2016-07-02 13:15:45

• 總結
  • 除了配置hosts ，和免密碼互連之外，先在一台機器上裝好所有東西
  • 配置好之後，拷貝虛擬機，配置hosts和免密碼互連
  • 之前在公司裝的時候jdk用的32位，hadoop的native包不能正常加載，浪費好多時間自己編譯，所以jdk務必64位
  • 配置免密碼互連
  其它也沒什麼了,注意下文件的用戶組，不一定是"hadoop"，根據自己的情況設置
  • sudo chown -R hadoop /opt
  • sudo chgrp -R hadoop /opt

• 准備文件
  
• 虛擬機安裝和配置

　　　　　　我們需要三台虛擬機，可以先裝一台虛擬機，下載好hadoop，配置好JDK，設置好環境變量後拷貝虛擬機

• 到此第一個虛擬機配置的差不多了，把這個虛擬機拷貝兩份(注意是完全復制，並且需要重置mac地址)，就有了三台虛擬機，分別為 master,slave1,slave2
  • 　　
  • 修改slave1和slave2的hostname為slave1-hadoop,slave2-hadoop
  • 修改三台機器的hosts
    • 192.168.56.101 master
      192.168.56.102 slave1
      192.168.56.103 slave2
  • ip不一定，需要自己看下虛機的ip
• 配置master可以免密碼登錄其它兩台機器和自己
  • 在master上操作
  • ssh-keygen -t rsa -P ''，一切都選擇默認操作，該輸密碼輸密碼
  • ssh-copy-id hadoop@master
  • ssh-copy-id hadoop@slave1
  • ssh-copy-id hadoop@slave2
  • 完成之後測試一下ssh slave1 正常情況下應該不用密碼就直接連接到slave1上

    hadoop@master-hadoop ~ $ ssh-keygen -t rsa -P ''
    Generating public/private rsa key pair.
    Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
    Created directory '/home/hadoop/.ssh'.
    Your identification has been saved in /home/hadoop/.ssh/id_rsa.
    Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
    The key fingerprint is:
    5c:c9:4c:0c:b6:28:eb:21:b9:6f:db:6e:3f:ee:0d:9a hadoop@master-hadoop
    The key's randomart image is:
    +--[ RSA 2048]----+
    |        oo.      |
    |       o =..     |
    |    . . . =      |
    |   . o . .       |
    |  o o   S        |
    |   + .           |
    |  . .   .        |
    |   ....o.o       |
    |   .o+E++..      |
    +-----------------+
    hadoop@master-hadoop ~ $ ssh-copy-id hadoop@slave1
    The authenticity of host 'slave1 (192.168.56.102)' can't be established.
    ECDSA key fingerprint is d8:fc:32:ed:a7:2c:e1:c7:d7:15:89:b9:f6:97:fb:c3.
    Are you sure you want to continue connecting (yes/no)? yes
    /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
    /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
    hadoop@slave1's password:
    
    Number of key(s) added: 1
    
    Now try logging into the machine, with:   "ssh 'hadoop@slave1'"
    and check to make sure that only the key(s) you wanted were added.

     

     

• 　格式化namenode
  • 　　./bin/hdfs namenode –format 
• 　　啟動hadoop驗證一下
  • ./sbin/start-all.sh
  • 正常的日志應該是這樣:

    hadoop@master-hadoop /opt/hadoop/sbin $ ./start-all.sh
    This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
    Starting namenodes on [master]
    master: starting namenode, logging to /opt/hadoop/logs/hadoop-hadoop-namenode-master-hadoop.out
    slave1: starting datanode, logging to /opt/hadoop/logs/hadoop-hadoop-datanode-slave1-hadoop.out
    slave2: starting datanode, logging to /opt/hadoop/logs/hadoop-hadoop-datanode-slave2-hadoop.out
    Starting secondary namenodes [master]
    master: starting secondarynamenode, logging to /opt/hadoop/logs/hadoop-hadoop-secondarynamenode-master-hadoop.out
    starting yarn daemons
    starting resourcemanager, logging to /opt/hadoop/logs/yarn-hadoop-resourcemanager-master-hadoop.out
    slave1: starting nodemanager, logging to /opt/hadoop/logs/yarn-hadoop-nodemanager-slave1-hadoop.out
    slave2: starting nodemanager, logging to /opt/hadoop/logs/yarn-hadoop-nodemanager-slave2-hadoop.out

     

  • 　　看下三個節點的jps

    hadoop@master-hadoop /opt/hadoop/sbin $ jps
    5858 ResourceManager
    5706 SecondaryNameNode
    5514 NameNode
    6108 Jps
    
    
    hadoop@slave2-hadoop ~ $ jps
    3796 Jps
    3621 NodeManager
    3510 DataNode
    
    
    hadoop@slave1-hadoop ~ $ jps
    3786 Jps
    3646 NodeManager
    3535 DataNode

• 　　一切正常，安裝完畢