Hadoop 分布式环境搭建

Hadoop
分布式环境搭建


一、前期环境


  • 安装概览
























IP Host Name Software Node
192.168.23.128 ae01 JDK 1.7 NameNode, SecondaryNameNode, DataNode, JobTracker, TaskTracker
192.168.23.129 ae02 JDK 1.7 DataNode, TaskTracker
192.168.23.130 ae03 JDK 1.7 DataNode, TaskTracker

    若使用虚拟机安装，可以安装samba, smbfs方面对于文件的控制。

  • 系统环境： ubuntu-12.04.2-server-amd64

  • 安装目录： /usr/local/ae

  • JDK 安装目录： export
    JAVA_HOME=/usr/local/ae/jdk1.7.0_51

  • Hadoop版本： hadoop-1.2.1

二、服务器间无密码登录


进入每台服务器，分别安装SSH，并生成ssh-key。下面安装步骤只讲解在ae01安装SSH和生成SSH-KEY，ae02、ae03重复此步骤。

  • 安装SSH

    [email protected]:~$ sudo apt-get install openssh-server

  • 生成SSH-KEY


    [email protected]:~# ssh-keygen -t rsa -P ""
    
Generating public/private rsa key pair.
    
Enter file in which to save the key (/root/.ssh/id_rsa):
    
Created directory ‘/root/.ssh‘.
    
Your identification has been saved in /root/.ssh/id_rsa.
    
Your public key has been saved in /root/.ssh/id_rsa.pub.
    
The key fingerprint is:
    
64:3d:a4:70:94:c4:33:64:6b:6b:1c:7c:e9:8f:15:93 [email protected]
    
The key‘s randomart image is:
    
+--[ RSA 2048]----+
    
|      .=*..      |
    
|       === . .   |
    
|        Oo= E    |
    
|       = = . o   |
    
|        S . .    |
    
|       .   +     |
    
|          . .    |
    
|                 |
    
|                 |
    
+-----------------+


  • 配置无密码SSH登录
    如果你希望服务器a1不需要密码就能SSH登录服务器a2，你需要将a1生成的公共密码添加到a2的~/.ssh/authorized_keys文件.
    此次安装中，ae01是NameNode，需要无密码登录到DataNode（ae02，ae03）的服务器，所以我们需要分别将ae01生成的公钥，添加到ae02，ae03的authorized_keys文件。
    修改ae01的公钥名字为id_rsa_ae01.pub。

    [email protected]:~/.ssh$ sudo cp id_rsa.pub id_rsa_ae01.pub

    复制id_rsa_ae01.pub到服务器ae02

    [email protected]:~/.ssh$ scp ./id_rsa_ae01.pub [email protected]192.168.23.129:~/.ssh/

    登录到ae02，将id_rsa_ae01.pub 添加到authorized_keys

    [email protected]:~/.ssh$ cat id_rsa_ae01.pub >> authorized_keys

    重新登录到ae01，并尝试无密码方面ae02


    [email protected]:~/.ssh$ ssh ae02
    
Welcome to Ubuntu 12.04.2 LTS (GNU/Linux 3.5.0-23-generic x86_64)
    

    * Documentation:  https://help.ubuntu.com/
    

    System information as of Thu Jun 12 12:21:45 CST 2014
    

    System load:  0.0                Processes:           86
    
  Usage of /:   10.3% of 18.45GB   Users logged in:     1
    
  Memory usage: 35%                IP address for eth0: 192.168.23.129
    
  Swap usage:   0%
    

    Graph this data and manage this system at https://landscape.canonical.com/
    

    122 packages can be updated.
    
69 updates are security updates.
    

    Last login: Tue Jun 10 20:06:57 2014 from 192.168.23.128

    对以上机器都进行如上操作，确保两两之间可以实现无密码ssh.


三、安装Hadoop


  • 修改host文件，添加3台服务器的host

    [email protected]:/usr/local/ae$ sudo vim /etc/hosts

    127.0.0.1         localhost
    
192.168.23.128    ae01
    
192.168.23.129    ae02
    
192.168.23.129    ae03

  • 解压Hadoop
    将hadoop-1.2.1.tar.gz
    复制到 /usr/local/ae，解压

    [email protected]:/usr/local/ae$ sudo tar -zxvf hadoop-1.2.1.tar.gz

  • 添加Hadoop环境变量

    export HADOOP_HOME=/usr/local/ae/hadoop-1.2.1
    
export HADOOP_HOME_WARN_SUPPRESS=1
    
export PATH=$PATH:$HADOOP_HOME/bin

  • 配置Hadoop
    core-site.xml是全局配置,hdfs-site.xml和mapred-site.xml分别是hdfs和mapred的局部配置

    修改$HADOOP_HOME/conf/hadoop-env.sh
    添加JAVA_HOME

    export JAVA_HOME=/usr/local/ae/jdk1.7.0_51

    修改$HADOOP_HOME/conf/core-site.xml
    加入以下文件到<configuration>节点


    <property>
    
        <name>hadoop.tmp.dir</name>
    
        <value>/usr/local/ae/storage/hadoop/temp</value>
    
        <description>A base for other temporary directories.</description>
    
</property>
    

    <property>
    
        <name>fs.default.name</name>
    
        <value>hdfs://ae01:9000</value>
    
        <description>
    
The name of the default file system.  A URI whose scheme and authority determine the FileSystem implementation.  The uri‘s scheme determines the config property (fs.SCHEME.impl) naming the FileSystem implementation class.  The uri‘s authority is used to
    
    determine the host, port, etc. for a filesystem.
    
        </description>
    
</property>
    
<property>
    
        <name>fs.checkpoint.period</name>
    
        <value>3600</value>
    
        <description>
    
The number of seconds between two periodic checkpoints.
    
        </description>
    
</property>
    

    <property>
    
        <name>fs.checkpoint.size</name>
    
        <value>67108864</value>
    
        <description>
    
The size of the current edit log (in bytes) that triggers a periodic checkpoint even if the fs.checkpoint.period hasn‘t expired.
    
        </description>
    
</property>

    修改$HADOOP_HOME/conf/hdfs-site.xml
    加入以下文件到<configuration>节点


    <property>
    
        <name>dfs.name.dir</name>
    
        <value>/usr/local/ae/storage/hadoop/name</value>
    
        <description>
    
Determines where on the local filesystem the DFS name node should store the name table(fsimage).  If this is a comma-delimited list of directories then the name table is replicated in all of the directories, for redundancy.
    
        </description>
    
</property>
    

    <property>
    
        <name>dfs.data.dir</name>
    
        <value>/usr/local/ae/storage/hadoop/data</value>
    
        <description>]
    
Determines where on the local filesystem the DFS name node should store the name table(fsimage).  If this is a comma-delimited list of directories then the name table is replicated in all of the directories, for redundancy.
    
        </description>
    
</property>
    

    <property>
    
        <name>dfs.http.address</name>
    
        <value>ae01:50070</value>
    
        <description>
    
The address and the base port where the dfs namenode web ui will listen on. If the port is 0 then the server will start on a free port.
    
        </description>
    
</property>
    

    <property>
    
        <name>dfs.permissions</name>
    
        <value>false</value>
    
        <description>
    
If "true", enable permission checking in HDFS. If "false", permission checking is turned off,
    
but all other behavior is unchanged. Switching from one parameter value to the other does not change the mode, owner or group of files or directories.
    
        </description>
    
</property>
    

    <property>
    
        <name>dfs.replication</name>
    
        <value>1</value>
    
        <description>Default block replication. The actual number of replications can be specified when the file is created. The default is used if replication is not specified in create time.
    
        </description>
    
</property>

    修改$HADOOP_HOME/conf/hdfs-site.xml
    加入以下文件到<configuration>节点


    <property>
    
        <name>mapred.job.tracker</name>
    
        <value>ae01:9001</value>
    
        <description>
    
The host and port that the MapReduce job tracker runs  at.  If "local", then jobs are run in-process as a single map and reduce task.
    
        </description>
    
   </property>

    修改$HADOOP_HOME/conf/masters

    ae01

    修改$HADOOP_HOME/conf/slaves

    ae01
    
ae02
    
ae03

    在上述配置中：
    fs.default.name的值hdfs://ae23:9000 用来决定NameNode
    mapred.job.tracker的值ae23:9001 用来决定JobTracker
    masters文件的值决定SecondaryNameNode
    slaves文件的值决定DataNode和TaskTracker

    创建文件目录/usr/local/ae/storage/hadoop, 并赋予hadoop
    文件夹足够的权限

    [email protected]: ~$ /usr/local/ae$ sudo chmod 777 ./storage/hadoop/

　　　将配置好的Hadoop复制到ae02和ae03，并在ae02和ae03上创建文件目录/usr/local/ae/storage/hadoop

  • 初始化和启动Hadoop

    登录ae01，初始化

    [email protected]：~$ hadoop namenode -format

    启动Hadoop

    [email protected]：~$ start-all.sh

    使用jps查看java进程
    ae01


    [email protected]:/usr/local/ae$ jps
    
26239 JobTracker
    
26158 SecondaryNameNode
    
36052 Jps
    
26468 TaskTracker
    
25687 NameNode
    
25926 DataNode

    ae02

    [email protected]:~$ jps
    
25021 Jps
    
18999 TaskTracker
    
18791 DataNode

    ae03

    [email protected]:~$ jps
    
3901 DataNode
    
9485 Jps
    
4106 TaskTracker

时间： 2024-03-08 13:47:54

Hadoop 分布式环境搭建的相关文章

Hadoop分布式环境搭建

作者:gqk: 使用多个服务器访问hadoop的各个进程: Hadoop分布式环境搭建准备工作: 克隆三个虚拟机: 机器规划: 更改两个机器的MAC的地址 更改网卡并配置: 1)修改网卡:克隆的机子网卡默认为eth1 改为eth0,,MAC改为本机的 vim /etc/udev/rules.d/70-persistent-net.rules  (注意:出现修改后 不管用的 ) eth0配文件中的mac地址改成了ifocnfig中出来的mac地址,然后再次删除/etc/udev/rules.d/7

WMware 中CentOS系统Hadoop 分布式环境搭建（一）——Hadoop安装环境准备

1.创建3台虚拟机并装好系统,这里使用64位CentOS. 2.Ping测试[确保两两能ping通]: [ping xxx.xxx.xxx.xxx] 3.安装SSH:[yum install ssh rsync] 4.按下图重命名主机.修改host文件: master: slave1: slave2: 具体虚拟机的IP地址根据实际情况配置. 5.重启每台机器,检查ip和host配置是否正常,然后使用主机名来进行ping测试: 6.创建hadoop的用户和用户组[每台机器上执行] sudo gro

一、Hadoop伪分布式环境搭建

Hadoop 2.x伪分布式环境搭建步骤: 1.修改hadoop-env.sh.yarn-env.sh.mapred-env.sh 方法:使用notepad++(beifeng用户)代开这三个文件 添加代码:export JAVA_HOME=/opt/modules/jdk1.7.0_67 2.修改core-site.xml.hdfs-site.xml.yarn-site.xml.mapred-site.xml配置文件 1)修改core-site.xml <configuration> <

【转】Hadoop HDFS分布式环境搭建

原文地址  http://blog.sina.com.cn/s/blog_7060fb5a0101cson.html Hadoop HDFS分布式环境搭建 最近选择给大家介绍Hadoop HDFS系统,因此研究了一下如何在Linux 下配置一个HDFS Clust.小记一下,以备将来进一步研究和记忆. HDFS简介 全称 Hadoop Distributed File System, Hadoop分布式文件系统. 根据Google的GFS论文,由Doug Cutting使用JAVA开发的开源项目

Hadoop学习笔记(3)——分布式环境搭建

Hadoop学习笔记(3) ——分布式环境搭建 前面,我们已经在单机上把Hadoop运行起来了,但我们知道Hadoop支持分布式的,而它的优点就是在分布上突出的,所以我们得搭个环境模拟一下. 在这里,我们采用这样的策略来模拟环境,我们使用3台ubuntu机器,1台为作主机(master),另外2台作为从机(slaver).同时,这台主机,我们就用第一章中搭建好的环境来. 我们采用与第一章中相似的步骤来操作: 运行环境搭建 在前面,我们知道,运行hadoop是在linux上运行的.所以我们单机就在

【Hadoop基础教程】4、Hadoop之完全分布式环境搭建

上一篇blog我们完成了Hadoop伪分布式环境的搭建,伪分布式模式也叫单节点集群模式, NameNode.SecondaryNameNode.DataNode.JobTracker.TaskTracker所有的守护进程全部运行在K-Master节点之上.在本篇blog我们将搭建完全分布式环境,运行NameNode.SecondaryNameNode.JobTracker守护进程在主节点上,运行DataNode.TaskTracker在从节点上. 开发环境 硬件环境:Centos 6.5 服务器

【Hadoop】伪分布式环境搭建、验证

Hadoop伪分布式环境搭建: 自动部署脚本: #!/bin/bash set -eux export APP_PATH=/opt/applications export APP_NAME=Ares # 安装apt依赖包 apt-get update -y && apt-get install supervisor -y && apt-get install python-dev python-pip libmysqlclient-dev -y # 安装pip.python

Hadoop 2.x伪分布式环境搭建测试

Hadoop 2.x伪分布式环境搭建测试 标签(空格分隔): hadoop hadoop,spark,kafka交流群:459898801 1,搭建hadoop所需环境 卸载open JDK rpm -qa |grep java rpm -e –nodeps [java] 1.1,在/opt/目录下创建四个目录: modules/ software/ datas/ tools/ 解压hadoop-2.5.0及jdk-7u67-linux-x64.tar.gz至modules目录下. $tar -

Hadoop之环境搭建

初学Hadoop之环境搭建 阅读目录 1.安装CentOS7 2.安装JDK1.7.0 3.安装Hadoop2.6.0 4.SSH无密码登陆 本文仅作为学习笔记,供大家初学Hadoop时学习参考.初学Hadoop,欢迎有经验的朋友进行指导与交流! 1.安装CentOS7 准备 CentOS系统镜像CentOS-7.0-1406-x86_64-Everything.iso vmware workstation 11 安装 1.点击创建一个新的虚拟机,选择"典型"配置,下一步. 2.选择&