CDH大数据集群环境搭建步骤

CDH环境搭建步骤：

一、环境检查

1、检查并关闭防火墙

2、检查并关闭selinux

3、配置nginx 文件服务并重启 nginx服务

 systemctl status firewalld.service

systemctl stop firewalld.service

cat /etc/selinux/config

vim /etc/nginx/conf.d/file.conf 添加下面内容

server

{

listen 9052;

charset utf-8;

server_name localhost;

location / {

alias /opt/;

autoindex on;

autoindex_exact_size on;

autoindex_localtime on;

}

}4、检查swappiness不能超过10

5、设置下面内容

二、安装准备

1、准备cm包

 cat /proc/sys/vm/swappiness

如果超过10

运行时执行

echo 0 > /proc/sys/vm/swappiness

保持重启之后继续生效

vim /etc/sysctl.conf

vm.swappiness=0

echo never > /sys/kernel/mm/transparent_hugepage/defrag

echo never > /sys/kernel/mm/transparent_hugepage/enabled

为了重启之后也生效

vim /etc/rc.local

添加：

echo never > /sys/kernel/mm/transparent_hugepage/defrag

echo never > /sys/kernel/mm/transparent_hugepage/enabled

1、将cm包放到安装http服务的节点

mkdir /opt/cm

allkeys.asc

cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm

cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm

cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm

cloudera-manager-server-db-2-6.3.1-1466458.el7.x86_64.rpm

enterprise-debuginfo-6.3.1-1466458.el7.x86_64.rpm

oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm

2、创建本地yum仓库

createrepo .

若createrepo命令未发现，需要安装createrepo

yum -y install createrepo

再次执行createrepo .

3、在/etc/yum.repo.d/目录下增加cm.repo文件

vim /etc/yum.repos.d/cm.repo

添加以下内容：

[cmrepo]2、准备cdh parcel包

3、在mysql创建数据库和用户并授权 name=CM

baseurl=http://hostname:9052/cm

gpgcheck=false

enabled=true

4、验证CM源是否配置成功

yum repolist

5、将cm.repo分发到其余各节点

1、mkdir /opt/cdh

将parcel包放到文件夹

CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel

CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1

CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha256

manifest.json

2、mkdir /opt/cdh/flink

将flink parcel包放到该文件下

FLINK-1.13.3-bin-scala-2.11-el7.parcel

FLINK-1.13.3-bin-scala-2.11-el7.parcel.sha

manifest.json

3、将FLINK_ON_YARN-1.13.3.jar 放到cm节点 的/opt/cloudera/csd并修改文件夹权限

sudo chown cloudera-scm:cloudera-scm /opt/cloudera/*

#scm

CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

CREATE USER 'scm'@'%' IDENTIFIED BY 'password';

GRANT ALL ON scm.* TO 'scm'@'%' ;

FLUSH PRIVILEGES;

#Activity Monitor

CREATE DATABASE am DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

CREATE USER 'am'@'%' IDENTIFIED BY 'password';

GRANT ALL ON am.* TO 'am'@'%';

FLUSH PRIVILEGES;

#Reports Manager

CREATE DATABASE rm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

CREATE USER 'rm'@'%' IDENTIFIED BY 'password';

GRANT ALL ON rm.* TO 'rm'@'%' ;

FLUSH PRIVILEGES;

#Hue

CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

CREATE USER 'hue'@'%' IDENTIFIED BY 'password';

GRANT ALL ON hue.* TO 'hue'@'%';

FLUSH PRIVILEGES;

#Hive4、在cm节点配置jdbc驱动

5、如果cms节点的/usr/java/default/下没有jdk，用下面方式安装

yum -y install oracle-j2sdk1.8-1.8.0+update181-1

三、安装

1、安装 cloudera-manager-server

yum -y install cloudera-manager-server

2、初始化cm数据库

/opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm password

3、启动cm server

systemctl start cloudera-scm-server

4、访问7180端口， admin admin CREATE DATABASE hive DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

CREATE USER 'hive'@'%' IDENTIFIED BY 'password';

GRANT ALL ON hive.* TO 'hive'@'%' ;

FLUSH PRIVILEGES;

#Oozie

CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

CREATE USER 'oozie'@'%' IDENTIFIED BY 'password';

GRANT ALL ON oozie.* TO 'oozie'@'%' ;

FLUSH PRIVILEGES;

# mkdir -p /usr/share/java

# mv mysql-connector-java-5.1.34.jar/usr/share/java/

# cd /usr/share/java/

# ln -s mysql-connector-java-5.1.34.jar mysql-connector-java.jar

配置parcel路径：http://hostname:9052/cdh