CentOS 64位上编译 Hadoop 2.6.0

1.操作系统编译环境

yum install cmake lzo-devel zlib-devel gcc gcc-c++ autoconf automake libtool ncurses-devel openssl-devel libXtst

2.安装JDK

下载JDK1.7，注意只能用1.7，否则编译会出错

http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html

tar zxvf jdk-7u75-linux-x64.tar.gz -C /app

export JAVA_HOME=/app/jdk1.7.0_75

export JRE_HOME=$JAVA_HOME/jre

export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

PATH=$PATH:$JAVA_HOME/bin

3.安装protobuf

下载protobuf-2.5.0，不能用高版本，否则Hadoop编译不能通过

wget https://protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz

tar xvf protobuf-2.5.0.tar.gz

cd protobuf-2.5.0

./configure

make

make install

ldconfig

protoc --version

4.安装ANT

wget http://mirror.bit.edu.cn/apache/ant/binaries/apache-ant-1.9.4-bin.tar.gz

tar zxvf apache-ant-1.9.4-bin.tar.gz -C /app

vim /etc/profile

export ANT_HOME=/app/apache-ant-1.9.4

PATH=$PATH:$ANT_HOME/bin

5.安装maven

wget http://mirror.bit.edu.cn/apache/maven/maven-3/3.3.1/binaries/apache-maven-3.3.1-bin.tar.gz

tar zxvf apache-maven-3.3.1-bin.tar.gz -C /app

vim /etc/profile

export MAVEN_HOME=/app/apache-maven-3.3.1

export PATH=$PATH:$MAVEN_HOME/bin

修改配置文件

vi /usr/local/apache-maven-3.0.5/conf/settings.xml

更改maven资料库，在<mirrors></mirros>里添加如下内容：

<mirror>

<id>nexus-osc</id>

<mirrorOf>*</mirrorOf>

<name>Nexusosc</name>

<url>http://maven.oschina.net/content/groups/public/</url>

</mirror>

在<profiles></profiles>内新添加

<profile>

<id>jdk-1.7</id>

<activation>

<jdk>1.7</jdk>

</activation>

<repositories>

<repository>

<id>nexus</id>

<name>local private nexus</name>

<url>http://maven.oschina.net/content/groups/public/</url>

<releases>

<enabled>true</enabled>

</releases>

<snapshots>

<enabled>false</enabled>

</snapshots>

</repository>

</repositories>

<pluginRepositories>

<pluginRepository>

<id>nexus</id>

<name>local private nexus</name>

<url>http://maven.oschina.net/content/groups/public/</url>

<releases>

<enabled>true</enabled>

</releases>

<snapshots>

<enabled>false</enabled>

</snapshots>

</pluginRepository>

</pluginRepositories>

</profile>

6.安装findbugs（非必须）

wget http://prdownloads.sourceforge.net/findbugs/findbugs-3.0.1.tar.gz?download

tar zxvf findbugs-3.0.1.tar.gz -C /app

vim /etc/profile

export FINDBUGS_HOME=/app/findbugs-3.0.1

PATH=$PATH:$FINDBUGS_HOME/bin

export PATH

7.编译 Hadoop2.6.0

wget http://mirror.bit.edu.cn/apache/hadoop/core/hadoop-2.6.0/hadoop-2.6.0-src.tar.gz

cd hadoop-2.6.0-src

mvn package -DskipTests -Pdist,native -Dtar

[INFO] Reactor Summary:

[INFO]

[INFO] Apache Hadoop Main ................................. SUCCESS [  4.401 s]

[INFO] Apache Hadoop Project POM .......................... SUCCESS [  3.864 s]

[INFO] Apache Hadoop Annotations .......................... SUCCESS [  7.591 s]

[INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.535 s]

[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  3.585 s]

[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [  6.623 s]

[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [  4.722 s]

[INFO] Apache Hadoop Auth ................................. SUCCESS [  7.787 s]

[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [  5.500 s]

[INFO] Apache Hadoop Common ............................... SUCCESS [02:47 min]

[INFO] Apache Hadoop NFS .................................. SUCCESS [ 12.793 s]

[INFO] Apache Hadoop KMS .................................. SUCCESS [ 20.443 s]

[INFO] Apache Hadoop Common Project ....................... SUCCESS [  0.111 s]

[INFO] Apache Hadoop HDFS ................................. SUCCESS [04:35 min]

[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 29.896 s]

[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [ 11.100 s]

[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [  8.262 s]

[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [  0.069 s]

[INFO] hadoop-yarn ........................................ SUCCESS [  0.066 s]

[INFO] hadoop-yarn-api .................................... SUCCESS [02:05 min]

[INFO] hadoop-yarn-common ................................. SUCCESS [ 46.132 s]

[INFO] hadoop-yarn-server ................................. SUCCESS [  0.123 s]

[INFO] hadoop-yarn-server-common .......................... SUCCESS [ 19.166 s]

[INFO] hadoop-yarn-server-nodemanager ..................... SUCCESS [ 25.552 s]

[INFO] hadoop-yarn-server-web-proxy ....................... SUCCESS [  5.456 s]

[INFO] hadoop-yarn-server-applicationhistoryservice ....... SUCCESS [ 11.781 s]

[INFO] hadoop-yarn-server-resourcemanager ................. SUCCESS [ 30.557 s]

[INFO] hadoop-yarn-server-tests ........................... SUCCESS [  9.765 s]

[INFO] hadoop-yarn-client ................................. SUCCESS [ 14.016 s]

[INFO] hadoop-yarn-applications ........................... SUCCESS [  0.101 s]

[INFO] hadoop-yarn-applications-distributedshell .......... SUCCESS [  4.116 s]

[INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SUCCESS [  2.993 s]

[INFO] hadoop-yarn-site ................................... SUCCESS [  0.093 s]

[INFO] hadoop-yarn-registry ............................... SUCCESS [  9.036 s]

[INFO] hadoop-yarn-project ................................ SUCCESS [  6.557 s]

[INFO] hadoop-mapreduce-client ............................ SUCCESS [  0.267 s]

[INFO] hadoop-mapreduce-client-core ....................... SUCCESS [ 36.775 s]

[INFO] hadoop-mapreduce-client-common ..................... SUCCESS [ 28.049 s]

[INFO] hadoop-mapreduce-client-shuffle .................... SUCCESS [  7.285 s]

[INFO] hadoop-mapreduce-client-app ........................ SUCCESS [ 17.333 s]

[INFO] hadoop-mapreduce-client-hs ......................... SUCCESS [ 15.283 s]

[INFO] hadoop-mapreduce-client-jobclient .................. SUCCESS [  7.110 s]

[INFO] hadoop-mapreduce-client-hs-plugins ................. SUCCESS [  3.843 s]

[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ 12.559 s]

[INFO] hadoop-mapreduce ................................... SUCCESS [  6.331 s]

[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ 45.863 s]

[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [ 46.304 s]

[INFO] Apache Hadoop Archives ............................. SUCCESS [  3.575 s]

[INFO] Apache Hadoop Rumen ................................ SUCCESS [ 12.991 s]

[INFO] Apache Hadoop Gridmix .............................. SUCCESS [ 10.105 s]

[INFO] Apache Hadoop Data Join ............................ SUCCESS [  5.021 s]

[INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [  3.804 s]

[INFO] Apache Hadoop Extras ............................... SUCCESS [  5.298 s]

[INFO] Apache Hadoop Pipes ................................ SUCCESS [ 10.290 s]

[INFO] Apache Hadoop OpenStack support .................... SUCCESS [  9.220 s]

[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [11:12 min]

[INFO] Apache Hadoop Client ............................... SUCCESS [ 10.714 s]

[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [  0.143 s]

[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [  7.664 s]

[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 29.970 s]

[INFO] Apache Hadoop Tools ................................ SUCCESS [  0.057 s]

[INFO] Apache Hadoop Distribution ......................... SUCCESS [ 49.425 s]

[INFO] ------------------------------------------------------------------------

[INFO] BUILD SUCCESS

[INFO] ------------------------------------------------------------------------

[INFO] Total time: 32:26 min

[INFO] Finished at: 2015-03-19T19:56:40+08:00

[INFO] Final Memory: 99M/298M

[INFO] ------------------------------------------------------------------------

编译成功后会打包，放在hadoop-dist/target

# ls

antrun                    dist-tar-stitching.sh  hadoop-2.6.0.tar.gz    hadoop-dist-2.6.0-javadoc.jar  maven-archiver

dist-layout-stitching.sh  hadoop-2.6.0           hadoop-dist-2.6.0.jar  javadoc-bundle-options         test-dir

时间： 2024-03-23 03:37:18

CentOS 64位上编译 Hadoop 2.6.0的相关文章

centOS 64位下编译hadoop2.4.1

1. 安装JDK a.      解压jdk [[email protected] software]$ tar -xvzf jdk-7u60-linux-x64.tar.gz b.      设置环境变量 PATH=$PATH:$HOME/bin export JAVA_HOME=/home/bigdata001/BigDataPlatform/jdk1.7.0_60 export JAVA_BIN=$JAVA_HOME/bin export CLASSPATH=.:$JAVA_HOME/li

64位CentOS上编译 Hadoop 2.2.0

1. 下载Hadoop 2.2.0 源码包,并解压 $ wget http://mirrors.hust.edu.cn/apache/hadoop/common/hadoop-2.2.0/hadoop-2.2.0-src.tar.gz $ tar zxf hadoop-2.2.0-src.tar.gz 2. 安装下面的软件 $ sudo yum install cmake lzo-devel zlib-devel gcc autoconf automake libtool ncurses-dev

64位Ubuntu 编译 hadoop源码

本人下载了最新版的ubuntu64(14.04)位桌面版系统,在安装hadoop2.6.0时,由于官方的hadoop是在32位机子上编译的,故需下载hadoop源码进行编译. 准备:hadoop-2.6.0-src jdk1.7.0_75(由于最新版的jdk是1.8.0_31版本的,而我在用这个版本编译hadoop时出现了问题,后来发现1.8.0是在hadoop出来之后出来的,所以可能有些不支持,故建议用1.7.0版本的) 1.安装JDK 安装jdk的方式大家可以网上搜一下,或者通过这个链接ht

CentOS 6.4 64位 源码编译hadoop 2.2.0

CentOS 6.4 64位 源码编译hadoop 2.2.0 搭建环境:Centos 6.4 64bit 1.安装JDK 参考这里2.安装mavenmaven官方下载地址,可以选择源码编码安装,这里就直接下载编译好的wget http://mirror.bit.edu.cn/apache/maven/maven-3/3.1.1/binaries/apache-maven-3.1.1-bin.zip解压文件后,同样在/etc/profie里配置环境变量vim /etc/profieexport

centos 64位 下hadoop-2.7.2 下编译

由于机器安装的是centos 6.7 64位 系统  从hadoop中下载是32位  hadoop 依赖的的库是libhadoop.so 是32 位  需要重新编译. 下面就是编译的步骤 1 安装jdk 1 .1下载jdk 1.2对jdk 进行解压 1.3 配置环境变量 1.3.1命令 1.3.2 修改环境变量 在文件末尾添加 如图显示 1.4 利用命令查看版本 2安装所需要的库 2.1命令 2.2第二个安装 3安装 protobuf-2.5.0.tar.gz(版本必须是2.5.0) 3.1如图

CentOS上编译hadoop 2.7

官网下载的 hadoop执行运行报警告: WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicableput: `./in': No such file or directory 主要是lib下的native-hadoop 是32位的,所以这里我们自己在64位的系统上编译hadoop 环境准备: 1

mac OS X Yosemite 上编译hadoop 2.6/2.7及TEZ 0.5.2/0.7 注意事项

1.jdk 1.7问题 hadoop 2.7.0必须要求jdk 1.7.0,而oracle官网已经声明,jdk 1.7 以后不准备再提供更新了,所以趁现在还能下载,赶紧去down一个mac版吧 http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html 应该选择mac ox 64位的版本 http://download.oracle.com/otn-pub/java/jdk/7u79-b1

Centos(64位)安装Hbase详细步骤

HBase是一个分布式的.面向列的开源数据库,该技术来源于 Fay Chang 所撰写的Google论文“Bigtable:一个结构化数据的分布式存储系统”.就像Bigtable利用了Google文件系统(File System)所提供的分布式数据存储一样,HBase在Hadoop之上提供了类似于Bigtable的能力.HBase是Apache的Hadoop项目的子项目.HBase不同于一般的关系数据库,它是一个适合于非结构化数据存储的数据库.另一个不同的是HBase基于列的而不是基于行的模式.

CentOS6.5、Hadoop-2.2.0、64 位的编译

1.安装CentOS6.5 下载地址是 http://mirror.neu.edu.cn/centos/6.5/isos/x86_64/ 选择 CentOS-6.5-x86_64-bin-DVD1.iso 下载,注意是 64 位的,大小是 4GB,需要下载一段时间的. 其实 6.x 的版本都可以,不一定是 6.5,一定要保持 linux 联网状态. 说明:我把软件下载后全部复制到/usr/local 目录下,以下命令执行的路径是在/usr/local 目录下. 请读者在阅读时,一定要注意路径.