大数据管理框架CDP集群搭建

1 搭建准备

HostName IP 内存 CPU 磁盘
cdp01 172.29.0.110 16G 4核 100G
cdp02 172.29.0.111 16G 4核 100G
cdp03 172.29.0.112 16G 4核 100G

组件 版本
Cloudera Manager 7.1.4
CDH 7.1.7
OpenJDK 1.8.0_372
CentOS 7.6
MySQL 8.0.34

2 基础配置

2.1 hosts配置(所有主机)

  • 编辑hosts文件
    vim /etc/hosts
    删除原IP配置,填入以下内容
    172.29.0.110 cdp01
    172.29.0.111 cdp02
    172.29.0.112 cdp03
    
  • 设置主机名称
    cdp01节点
    hostnamectl set-hostname cdp01 && bash
    cdp02节点
    hostnamectl set-hostname cdp02 && bash
    cdp03节点
    hostnamectl set-hostname cdp03 && bash

2.2 关闭 Selinux(所有主机)

setenforce 0
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config

2.3 关闭防火墙(所有主机)

systemctl stop firewalld
systemctl disable firewalld

2.4 设置swap(所有主机)

cat >> /etc/sysctl.conf <<EOF
vm.swappiness=1
EOF

2.5 设置 limits(所有主机)

cat >> /etc/security/limits.conf <<EOF
* soft nofile 65535
* hard nofile 65535
hdfs - nofile 32768
mapred - nofile 32768
hbase - nofile 32768
hdfs - noproc 32768
mapred - noproc 32768
hbase - noproc 32768
EOF

2.6 关闭透明大页(所有主机)

vim /etc/rc.d/rc.local

echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag
if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
	echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi
if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
	echo never > /sys/kernel/mm/transparent_hugepage/defrag
fi

2.7 安装ntpd服务(所有主机)

yum install -y ntp ntpdate

  • cdp01 节点
    vim /etc/ntp.conf
    // 注:192.168.191.2 和 255.255.255.0是集群所在网段的网关和子网掩码
    restrict 192.168.191.2 mask 255.255.255.0 nomodify notrap
    server 127.127.1.0
    fudge 127.127.1.0 stratum 8
    
  • cdp02 节点
    vim /etc/ntp.conf
    restrict 192.168.191.2  mask 255.255.255.0  nomodify notrap
    server cdp02 prefer
    server 127.127.1.0
    fudge 127.127.1.0 stratum 9
    
  • cdp03 节点
    vim /etc/ntp.conf
    server cdp01 prefer
    server cdp02
    
  • 在manager( cdp01 )启动ntp之后,其余各节点启动ntp服务之前,master(cdp02 )和worker( cdp03 )节点执行命令,同步manager时间启动 cdp01 节点ntpd服务
    systemctl start ntpd
    systemctl enable ntpd
    
  • 在cdp02、cdp03 节点进行时间同步
    ntpdate cdp01
    
  • 启动cdp02、cdp03节点ntpd服务
    systemctl start ntpd
    systemctl enable ntpd
    
  • 查看ntp状态
    ntpq -p
    ’*’ 表示当前使用的时钟源,’+’ 表示这些源可作为 NTP 源
    

2.8 安装http服务(所有主机)

  • 安装httpd服务
    yum install httpd -y
  • 开启httpd服务
    systemctl start httpd.service
  • 开机⾃动启动httpd服务
    chkconfig httpd on

2.9 配置SSH免密(cdp01节点)

  • 在要免密登录的主机上生成密钥
    ssh-keygen -t rsa # 一路回车
    在~/.ssh目录下创建两个文件id_rsa(私钥),id_rsa.pub(公钥)
  • 将生成的公钥发到要远程连接的服务器上
    ssh-copy-id cdp01
    ssh-copy-id cdp02
    ssh-copy-id cdp03
    
    注:这一次需要输入密码,后面就不用了;如果主机未设置账号密码登录需要先设置账号密码;配置后可以通过将/etc/ssh/sshd_config文件中的PasswordAuthentication yes修改为PasswordAuthentication no来关闭账号密码登录
    vim /etc/ssh/sshd_config
    修改如下:
    PasswordAuthentication no
    修改后使其生效:
    systemctl restart sshd
    
  • 验证免密登录效果
    ssh root@cdp02
    退出输入:
    exit
    

3 配置本地yum源

3.1 配置cm

  • 安装createrepo
    yum install createrepo
    
  • 下载cm软件包
    mkdir -p /var/www/html/cm7.1.4
    cd /var/www/html/cm7.1.4
    
    下载所需软件包,如果下载慢可以自己下载然后上传到服务器
    wget -nd -r -l1 --no-parent https://archive.cloudera.com/cm7/7.1.4/redhat7/yum/RPMS/x86_64/  
    wget https://archive.cloudera.com/cm7/7.1.4/redhat7/yum/RPM-GPG-KEY-cloudera
    wget https://archive.cloudera.com/cm7/7.1.4/allkeys.asc
    
  • 创建repo
    createrepo .
    
  • 下载cdh软件包
    mkdir -p /var/www/html/cdh7.1.7
    cd /var/www/html/cdh7.1.7
    
    下载所需软件包,如果下载慢可以自己下载然后上传到服务器
    wget -nd -r -l1 --no-parent https://archive.cloudera.com/cdh7/7.1.7.0/parcels/
    
  • 创建repo
    createrepo .
    
  • 配置本地yum源
    cd /etc/yum.repos.d
    vim cm7.repo
    
    配置以下内容:
    [cm]
    name=cm,Version 7.1.4
    baseurl=http://172.29.0.110/cm7.1.4/
    enable=1
    gpgcheck=0
    [cdh]
    name=cdh
    baseurl=http://172.29.0.110/cdh7.1.7/
    enable=1
    gpgcheck=0
    
    退出保存后,执⾏以下命令:
    yum clean all
    yum makecache
    
    查看配置结果
    yum list |grep cloudera
    
    可以看到以下内容:

    可以通过外网地址查看配置结果
    //xxx.xxx.xxx.xxx为cdp01外网IP
    http://xxx.xxx.xxx.xxx/cm7.1.4/
    http://xxx.xxx.xxx.xxx/cdh7.1.7/
    
  • 分发到其他节点
    在cdp01上配置好本地yum仓库之后,将cm7.repo⽂件分发到其他节点/etc/yum.repos.d目录

4 安装jdk(所有主机)

yum -y install java-1.8.0-openjdk

5 安装MySQL(cdp01)

5.1 安装并启用MySQL源

使用wget从MySQL网站下载MySQL源rpm文件:

cd /root/downloads/
wget https://dev.mysql.com/get/mysql80-community-release-el7-3.noarch.rpm

使用rpm命令来安装下载的rpm包:

rpm -ivh mysql80-community-release-el7-3.noarch.rpm

检查MySQL 8.0源已经成功添加到yum程序库中:

yum repolist enabled | grep "mysql.*-community.*"

5.2 使用yum命令安装MySQL 8.0

rpm --import https://repo.mysql.com/RPM-GPG-KEY-mysql-2022
yum install -y mysql-community-server

5.3 启动并检查MySQL服务器

systemctl start mysqld
systemctl status mysqld

5.4 修改Root密码

查找密码并登陆Mysql

cat /var/log/mysqld.log | grep password
mysql -u root -p #输入密码
alter user root@localhost identified by '密码';
FLUSH PRIVILEGES;
use mysql;
update user set host='%' where user = 'root';
commit;
exit;

5.5 my.cnf配置

vi /etc/my.cnf

#设置编码为utf-8
character_set_server=utf8
init_connect='SET NAMES utf8'

5.6 重启服务

systemctl stop mysqld
systemctl enable mysqld
systemctl start mysqld

5.7 创建CM元数据库

CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE USER 'scm'@'%' IDENTIFIED BY '密码';
GRANT ALL ON scm.* TO 'scm'@'%';

CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE USER 'amon'@'%' IDENTIFIED BY '密码';
GRANT ALL ON amon.* TO 'amon'@'%';

CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE USER 'rman'@'%' IDENTIFIED BY '密码';
GRANT ALL ON rman.* TO 'rman'@'%';

CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE USER 'hue'@'%' IDENTIFIED BY '密码';
GRANT ALL ON hue.* TO 'hue'@'%';

CREATE DATABASE hive DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE USER 'hive'@'%' IDENTIFIED BY '密码';
GRANT ALL ON hive.* TO 'hive'@'%';

CREATE DATABASE ranger DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE USER 'rangeradmin'@'%' IDENTIFIED BY '密码';
GRANT ALL ON ranger.* TO 'rangeradmin'@'%';

CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE USER 'nav'@'%' IDENTIFIED BY '密码';
GRANT ALL ON nav.* TO 'nav'@'%';

CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE USER 'navms'@'%' IDENTIFIED BY '密码';
GRANT ALL ON navms.* TO 'navms'@'%';

CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE USER 'oozie'@'%' IDENTIFIED BY '密码';
GRANT ALL ON oozie.* TO 'oozie'@'%';

flush privileges;

6 安装jdbc驱动(所有主机)

cd /root/downloads/
wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz
mkdir -p /usr/share/java
tar zxvf mysql-connector-java-5.1.46.tar.gz

scp mysql-connector-java-5.1.46/mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar
scp mysql-connector-java-5.1.46/mysql-connector-java-5.1.46-bin.jar cdp02:/usr/share/java/mysql-connector-java.jar
scp mysql-connector-java-5.1.46/mysql-connector-java-5.1.46-bin.jar cdp03:/usr/share/java/mysql-connector-java.jar

7 安装cm server(cdp01)

7.1 在cdp01上通过yum安装Cloudera Manager Server

yum -y install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server

7.2 安装完后可以查看服务状态

systemctl status cloudera-scm-server

7.3 初始化mysql

sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm '之前设置的密码'

7.4 启动Cloudera Manager Server

systemctl start cloudera-scm-server.service
systemctl status cloudera-scm-server.service

7.5 检查服务日志,等待启动完成

sudo tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log

7.6 检查端口是否监听

netstat -nltp | grep 7180

7.7 访问CM 创建集群

通过 http://cdp01:7180/ 访问CM,输⼊初始账号密码 admin/admin 即可登陆

热门相关:倾心之恋:总裁的妻子   富贵不能吟   万古至尊   裙上之臣   神算大小姐