Cloudera manager ha mode construction

Preface

Programmers engaged in big data It should be all right CDH Familiar with , This is a free data platform system that is easy to use in the market , stay CDH We can install and deploy compatible hadoop sprk zookeeper kafka And other commonly used big data services , Although it does not support flink, Compatibility must be driven by the general trend in the future .
This blog is about CDH It is a very important service for monitoring and managing various data services -Cloudera Manager, For many people This should be no stranger .
generally speaking ,cloudera manager The single point mode adopted by , Deploy a single server Single point service. But in a scenario where cluster jobs are particularly busy , In order to improve the Cloudera Manager High reliability , We'd better deploy Cloudera Manager HA（ Master-slave server Master-slave service） To improve the high reliability of the system .

Cloudera Manager HA Pattern building

1. Set up the host and load balancer

At a high level , Set up... On a separate host Cloudera Manager Server and Cloudera Management Service role （ Include Cloudera Navigator）, And ensure that the load balancer is configured to match the load from other sources Cloudera Host of service and management console for network access .

It is mainly divided into 4 Step ：
1. Create hosts for primary and secondary servers
2. Set load balancer
3. Set up the database
4. Set up NFS The server

Cloudera Manager Server,Cloudera Navigator And all that use relational databases Cloudera Management Service All roles should use an external database server located outside the host . You must ensure that these databases are configured for high availability .（ Here's an explanation ： Because of the internal application of CM Single point The databases are built inside the cluster , So this class doesn't use an external database server ）

The essence of database high availability is to configure a MySQL Master slave replication server

1.1 MySQL Mutual master-slave replication and load balancing configuration

The specific process has been described in a blog before .
3. haproxy Load balancing mysql
Here we choose to host 172.17.17.92 install

1. yum install haproxy
   2. Modify the configuration
   vi /etc/haproxy/haproxy.cfg
   Add the configuration
   listen mysql
   bind 0.0.0.0:3306
   mode tcp
   option tcplog
   balance leastconn
   server 172.17.17.21 172.17.17.21:3306 check
   server 172.17.17.25 172.17.17.25:3306 check

3 start-up
Execute... In the unzip directory haproxy -f /etc/haproxy/haproxy.cfg
About configuring other CM Service role ( for example Service Monitor and Host Monitor role ), These roles use file support

About configuring other CM Service role ( for example Service Monitor and Host Monitor role ), These roles use file supported storage mechanisms to store them on a shared server NFS Storage mechanism .

1.2 Master slave server Server selected host

Cloudera The suggestion is Cloudera Manager The service uses four hosts . All these hosts must correctly resolve the forward and reverse DNS lookup ：
Cloudera Manager Server Main host （ Host name ： CMS1 [147]）
Cloudera Management Service Main host （ Host name ： MGMT1 [148]）
Cloudera Manager Server Secondary host （ Host name ： CMS2 [145]）
Cloudera Management Service Secondary host （ Host name ： MGMT2 [146]）

1.3 Set up the load balancer

In the cluster Keep two hosts as CM Server and CM Service External access host . One (CMSHostname 144) Used to access the Cloudera Manager Admin Console, Another one (MGMTName 143) For from Cloudera Manager Server and CLoudera Manager Agents Yes Cloudera Manager Service Conduct internal access .

1. Download and install haproxy: yum install haproxy
2. stay 144 143 On the configuration HAProxy Self starting ： chkconfig haproxy on
3. To configure HAProxy : edit /etc/haproxy/haproxy.cfg file , stay 144 Add 7180 7182 7183 stay 143 Add 5683 7184 7185 7186….
   remarks ：HAProxy Is a credible , High performance TCP/HTTP Load balancer .

1.4 Set up NFS Server( As a shared storage mechanism , Realize high availability remote login )

1. install nfs service ： yum install nfs-utils
2. start-up nfc rpcbind, And set it to self start ：

systemctl start rpcbind
systemctl start nfs
chkconfig nfs on

2 by HA install and configure Cloudera Manager Server

2.1 by CM Server Set up NFS mount

1. stay CMSHostname 144( Here it is regarded as NFS Server) Create directory on ：
mkdir -p /media/cloudera-scm-server
2. stay 144 Editor in chief /etc/exports Add the following To mark the mount
/media/cloudera-scm-server 172...147(rw,sync,no_root_squash,no_subtree_check)
/media/cloudera-scm-server 172...147 (rw,sync,no_root_squash,no_subtree_check)
3. stay 144 Export and mount ： exportfs –a
4. stay 145 147 Set file mount on
Make sure nfs Installation successful , If not installed , be yum install nfs-utils
Make sure rpcbind Run and restart ： systemctl restart rpcbind
5. stay 145 147 Create a mount point on
1. Load on 145 perhaps 147 Upper /var/lib/cloudera-scm-server File to NFS Server On ： scp -r /var/lib/cloudera-scm-server/ 192...144:/media/cloudera-scm-server
2. stay 145 147 Create the file name to be mounted on
rm -rf /var/lib/cloudera-scm-server
mkdir -p /var/lib/cloudera-scm-server
3. By editing the 145 147 Upper /etc/fstab file , Add something , Set up fstab To persist the mount on restart
172...144:/media/cloudera-scm-server /var/lib/cloudera-scm-server nfs auto,noatime,nolock,intr,tcp,actimeo=1800 0 0

2.2 install CM HA in Server The primary node of

stay CM Server Build on a single point HA Pattern , The suggestion of the official website is to allocate four hosts as Server Principal and subordinate Service Principal and subordinate . So here we need to open the original CM Admin Cosole Home page , The original CM Service stop it And delete

remarks ： We will 147 As HA The primary node of

2.3 install CM HA in Server The slave node

stay 145 Installation on cloudera-manager-server, And will 147 Upper /etc/cloudera-scm-server/db.properties file scp Copied to the 145 On

Here, we need to pay attention to the... On the slave node cloudera-scm-server Not on by default , Because he's just at the master node 147 Upper server After hanging up Only open . therefore , Set the slave node server Don't start automatically by default .
chkconfig cloudera-scm-server off

2.4 to update CLoudera Manager Agents To use the load balancer , This is the use of 144

1. stay 145 147 146 148 Update on /etc/cloudera-scm-agent/config.ini( change server_host value ) server_host=172...144
2. restart agent: systemctl restart cloudera-scm-agent

3 by HA install and configure Cloudera Manager Service

3.1 by CM Service Set up NFS mount

1. stay NFS Server(Service Of NFS Server What we chose is 143) Create directory

mkdir -p /media/cloudera-host-monitor
mkdir -p /media/cloudera-scm-agent
mkdir -p /media/cloudera-scm-eventserver
mkdir -p /media/cloudera-scm-headlamp
mkdir -p /media/cloudera-service-monitor
mkdir -p /media/cloudera-scm-navigator
mkdir -p /media/etc-cloudera-scm-agent

2. stay NFS Server /etc/exports Add something to the file to mark the mount

/media/cloudera-host-monitor MGMT1(rw,sync,no_root_squash,no_subtree_check)
/media/cloudera-scm-agent MGMT1(rw,sync,no_root_squash,no_subtree_check)
/media/cloudera-scm-eventserver MGMT1(rw,sync,no_root_squash,no_subtree_check)
/media/cloudera-scm-headlamp MGMT1(rw,sync,no_root_squash,no_subtree_check)
/media/cloudera-service-monitor MGMT1(rw,sync,no_root_squash,no_subtree_check)
/media/cloudera-scm-navigator MGMT1(rw,sync,no_root_squash,no_subtree_check)
/media/etc-cloudera-scm-agent MGMT1(rw,sync,no_root_squash,no_subtree_check)
/media/cloudera-host-monitor MGMT2(rw,sync,no_root_squash,no_subtree_check)
/media/cloudera-scm-agent MGMT2(rw,sync,no_root_squash,no_subtree_check)
/media/cloudera-scm-eventserver MGMT2(rw,sync,no_root_squash,no_subtree_check)
/media/cloudera-scm-headlamp MGMT2(rw,sync,no_root_squash,no_subtree_check)
/media/cloudera-service-monitor MGMT2(rw,sync,no_root_squash,no_subtree_check)
/media/cloudera-scm-navigator MGMT2(rw,sync,no_root_squash,no_subtree_check)
/media/etc-cloudera-scm-agent MGMT2(rw,sync,no_root_squash,no_subtree_check)

 remarks ： there MGMT1 MGMT2 Respectively refer to the... We selected before 146 148 host 

3. stay NFS Server Export mount on ： exportfs -a
4. stay 146 148 Set file system mount on
   1） Make sure that... Is installed nfs, without , be yum install nfs-utils
   2） stay 146 148 Create a mount point directory on ：

mkdir -p /var/lib/cloudera-host-monitor
mkdir -p /var/lib/cloudera-scm-agent
mkdir -p /var/lib/cloudera-scm-eventserver
mkdir -p /var/lib/cloudera-scm-headlamp
mkdir -p /var/lib/cloudera-service-monitor
mkdir -p /var/lib/cloudera-scm-navigator
mkdir -p /etc/cloudera-scm-agent

3） stay 146 148 through NFS( Here is 143) mount

mount -t nfs NFS:/media/cloudera-host-monitor /var/lib/cloudera-host-monitor
mount -t nfs NFS:/media/cloudera-scm-agent /var/lib/cloudera-scm-agent
mount -t nfs NFS:/media/cloudera-scm-eventserver /var/lib/cloudera-scm-eventserver
mount -t nfs NFS:/media/cloudera-scm-headlamp /var/lib/cloudera-scm-headlamp
mount -t nfs NFS:/media/cloudera-service-monitor /var/lib/cloudera-service-monitor
mount -t nfs NFS:/media/cloudera-scm-navigator /var/lib/cloudera-scm-navigator
mount -t nfs NFS:/media/etc-cloudera-scm-agent /etc/cloudera-scm-agent

4） establish fstab Files remain mounted on reboot . edit /etc/fstab file Add the following line ：

NFS:/media/cloudera-host-monitor /var/lib/cloudera-host-monitor nfs auto,noatime,nolock,intr,tcp,actimeo=1800 0 0
NFS:/media/cloudera-scm-agent /var/lib/cloudera-scm-agent nfs auto,noatime,nolock,intr,tcp,actimeo=1800 0 0
NFS:/media/cloudera-scm-eventserver /var/lib/cloudera-scm-eventserver nfs auto,noatime,nolock,intr,tcp,actimeo=1800 0 0
NFS:/media/cloudera-scm-headlamp /var/lib/cloudera-scm-headlamp nfs auto,noatime,nolock,intr,tcp,actimeo=1800 0 0
NFS:/media/cloudera-service-monitor /var/lib/cloudera-service-monitor nfs auto,noatime,nolock,intr,tcp,actimeo=1800 0 0
NFS:/media/cloudera-scm-navigator /var/lib/cloudera-scm-navigator nfs auto,noatime,nolock,intr,tcp,actimeo=1800 0 0
NFS:/media/etc-cloudera-scm-agent /etc/cloudera-scm-agent nfs auto,noatime,nolock,intr,tcp,actimeo=1800 0 0

3.2 install CM Service Master node [ What we chose is 148 node ]

In the original single point mode , The node has been installed successfully cloudera-manager-daemons cloudera-manager-agent.
Here we only need to use the original single point mode , Configure agent .

1. edit /etc/cloudera-scm-agent/config.ini file , Add the following ：

server_host=172.*.*.144
listening_hostname=172.*.*.143	

2. Make sure Cloudera-scm Users and Cloudera-scm The group can access the following installed Directory stay / var / lib in , By using chown command Cloudera-scm：

chown -R cloudera-scm:cloudera-scm /var/lib/cloudera-scm-eventserver
chown -R cloudera-scm:cloudera-scm /var/lib/cloudera-scm-navigator
chown -R cloudera-scm:cloudera-scm /var/lib/cloudera-service-monitor
chown -R cloudera-scm:cloudera-scm /var/lib/cloudera-host-monitor
chown -R cloudera-scm:cloudera-scm /var/lib/cloudera-scm-agent
chown -R cloudera-scm:cloudera-scm /var/lib/cloudera-scm-headlamp

3. stay 148 restart agent: systemctl cloudera-scm-agent restart

3.3 install CM Service From the node

Principles and steps And configuration Cm Service The master node is similar to

4 Perform automatic failover

Official website HA Mode of Server Failover implementation It uses Corosync and Pacemaker To achieve , But there are many pits in the configuration process . Here Use a similar function to supplement （pcs）：
https://clusterlabs.org/quickstart-redhat.html

HA Node reply interval setting ：

pcs resource create cloudera-scm-server systemd:cloudera-scm-server op monitor interval=10s