RabbitMQ Keepalived

发表于 更新于 分类于

概述

系统环境 IP地址 主机名 节点类型
CentOS6.5 192.168.1.92 rabbit1 Master
CentOS6.5 192.168.1.93 rabbit2 Slave
CentOS6.5 192.168.1.95 VIP
CentOS6.5 192.168.1.17 Client

主机名设置
修改92的主机名为rabbit1,93的主机名为rabbit2：

1
2
3
4
5
[root@localhost opt]# vim /etc/sysconfig/network
[root@localhost opt]# vim /etc/hosts
192.168.1.92    rabbit1
192.168.1.93    rabbit2
[root@localhost opt]# reboot

SSH设置

2台机子执行以下操作：

1
[root@rabbit1 ~]# ssh-keygen

rabbit1机子执行以下操作:

1
2
3
4
[root@rabbit1 ~]# cp ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys
[root@rabbit1 ~]# ssh root@rabbit2 "cat ~/.ssh/id_rsa.pub" >> ~/.ssh/authorized_keys
[root@rabbit1 ~]# cat ~/.ssh/authorized_keys
[root@rabbit1 ~]# scp ~/.ssh/authorized_keys root@rabbit2:~/.ssh/

安装RabbitMQ

参考文章 RabbitMQ Install on RPM-based Linux安装好RabbitMQ.

1
2
3
4
5
6
7
8
9
10
11
12
$ wget https://packages.erlang-solutions.com/erlang/esl-erlang/FLAVOUR_1_general/esl-erlang_19.1.5~centos~6_amd64.rpm
$ wget
https://github.com/jasonmcintosh/esl-erlang-compat/releases/download/1.1.1/esl-erlang-compat-18.1-1.noarch.rpm
$ yum -y install unixODBC wxGTK wxGTK-gl lksctp-tools openssl
$ rpm -ivh esl-erlang_19.1.5~centos~6_amd64.rpm
$ rpm -ivh esl-erlang-compat-18.1-1.noarch.rpm
$ erl -version
Erlang (SMP,ASYNC_THREADS,HIPE) (BEAM) emulator version 8.1
$ wget https://www.rabbitmq.com/releases/rabbitmq-server/v3.6.5/rabbitmq-server-3.6.5-1.noarch.rpm
$ wget –no-cache http://www.convirture.com/repos/definitions/rhel/6.x/convirt.repo -O /etc/yum.repos.d/convirt.repo
$ yum -y install socat
$ rpm -ivh rabbitmq-server-3.6.5-1.noarch.rpm

安装管理插件

1
$ rabbitmq-plugins enable rabbitmq_management

防火墙设置

rabbitmq一共有以下端口被使用，根据实际情况打开端口。

4369 (epmd), 25672 (Erlang distribution)
5672, 5671 (AMQP 0-9-1 without and with TLS)
15672 (if management plugin is enabled)
61613, 61614 (if STOMP is enabled)
1883, 8883 (if MQTT is enabled)
15674 (WEB-STOMP)
编辑文件/etc/sysconfig/iptables,打开端口5672,5671,15672,4369,25672, 61613, 61614:

1
2
3
4
5
6
7
8
-A INPUT -m state --state NEW -m tcp -p tcp --dport 5671 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 5672 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 15672 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 4369 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 25672 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 61613 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 61614 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 15674 -j ACCEPT

重启防火墙

1
$ service iptables restart

启动服务

1
$ service rabbitmq-server start

管理员设置

访问页面http://www.rabbitmq.com/man/rabbitmqctl.1.man.html# 讲解了各种命令，这里使用user命令添加管理员：

1
2
3
4
$ rabbitmqctl add_user admin admin123
Creating user "admin" ...
$ rabbitmqctl set_user_tags admin administrator
Setting tags for user "admin" to [administrator] ...

访问web管理页面：http://192.168.1.92:15672,输入账号和密码测试登陆.

复制var/lib/rabbitmq/.erlang.cookie到93主机上。

1
[root@rabbit1 ~]# scp /var/lib/rabbitmq/.erlang.cookie root@rabbit2:/var/lib/rabbitmq/

检查erlang.cookie是否一致。必须保持一致的cookie。

1
2
3
4
[root@rabbit1 ~]# cat /var/lib/rabbitmq/.erlang.cookie 
PHKQIFQEBDKASNQJOJKR
[root@rabbit2 ~]# cat /var/lib/rabbitmq/.erlang.cookie 
PHKQIFQEBDKASNQJOJKR

重新启动服务,如果你复制了cookie后不能够重启，那你就重启系统吧：

1
[root@rabbit2 ~]# service rabbitmq-server start

查看各自的集群状态信息,应该都只有自己的节点信息：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
[root@rabbit1 ~]# rabbitmqctl cluster_status
Cluster status of node rabbit@rabbit1 ...
[{nodes,[{disc,[rabbit@rabbit1]}]},
 {running_nodes,[rabbit@rabbit1]},
 {cluster_name,<<"rabbit@rabbit1">>},
 {partitions,[]},
 {alarms,[{rabbit@rabbit1,[]}]}]

[root@rabbit2 ~]# rabbitmqctl cluster_status
Cluster status of node rabbit@rabbit2 ...
[{nodes,[{disc,[rabbit@rabbit2]}]},
 {running_nodes,[rabbit@rabbit2]},
 {cluster_name,<<"rabbit@rabbit2">>},
 {partitions,[]},
 {alarms,[{rabbit@rabbit2,[]}]}]

将rabbit1，rabbit2组成集群环境。

1
2
3
4
5
6
[root@rabbit2 ~]# rabbitmqctl stop_app
Stopping node rabbit@rabbit2 ...
[root@rabbit2 ~]# rabbitmqctl join_cluster rabbit@rabbit1
Clustering node rabbit@rabbit2 with rabbit@rabbit1 ... 
[root@rabbit2 ~]# rabbitmqctl start_app
Starting node rabbit@rabbit2 ...

浏览器访问地址：192.168.1.92:15672,看到有2个节点信息，说明成功集群.

接下来在任意节点上设置对所有的队列进行镜像处理：

1
2
[root@rabbit2 ~]# rabbitmqctl set_policy ha-all "^" '{"ha-mode":"all","ha-sync-mode":"automatic"}'
Setting policy "ha-all" for pattern "^" to "{\"ha-mode\":\"all\"}" with priority "0" ...

LVS安装

1
2
3
$ yum install ipvsadm -y
$ ipvsadm -v
ipvsadm v1.26 2008/5/15 (compiled with popt and IPVS v1.2.1)

检查linux内核是否集成lvs模块,开启路由转发功能,以及ip_vs模块：

1
2
3
4
5
6
7
8
9
10
11
12
$ modprobe -l | grep ipvs
$ echo 1 > /proc/sys/net/ipv4/ip_forward
$ echo "echo 1 > /proc/sys/net/ipv4/ip_forward" >> /etc/rc.local
$ modprobe ip_vs
$ modprobe ip_vs_wrr
$ echo "modprobe ip_vs" >> /etc/rc.local 
$ echo "modprobe ip_vs_wrr" >> /etc/rc.local
$ lsmod | grep ip_vs
ip_vs_wrr               2179  1 
ip_vs                 126705  3 ip_vs_wrr
libcrc32c               1246  1 ip_vs
ipv6                  336368  47 ip_vs,bridge,ip6t_REJECT,nf_conntrack_ipv6,nf_defrag_ipv6

keepalived安装

以下是Master和Slave都要进行的操作：
从官网下载源码包 http://www.keepalived.org/download.html ,安装相应的依赖库：

1
2
3
4
5
6
7
[root@localhost]# yum install -y gcc gcc-c++ makepcre pcre-devel kernel-devel openssl-devel libnl-devel popt-devel libnfnetlink-devel openssl-devel popt-static
[root@localhost]# yum update glib* -y
[root@localhost opt]# wget http://www.keepalived.org/software/keepalived-1.2.24.tar.gz
[root@localhost opt]# tar zxf keepalived-1.2.24.tar.gz
[root@localhost opt]# cd keepalived-1.2.24
[root@localhost]# ./configure --prefix=/usr/local/keepalived
[root@localhost]# make && make install

创建keepalived服务：

1
2
3
4
5
6
[root@localhost ~]# ln -s /usr/local/keepalived/sbin/keepalived /usr/sbin/keepalived
[root@localhost ~]# ln -s /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/keepalived
[root@localhost ~]# ln -s /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/keepalived
[root@localhost ~]# ln -s /usr/local/keepalived/etc/keepalived /etc/keepalived
[root@localhost ~]# chmod +x /usr/local/keepalived/etc/rc.d/init.d/keepalived
[root@localhost ~]# keepalived -v

master配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
[root@localhost ~]# vim /etc/keepalived/keepalived.conf
global_defs {
   notification_email {
     jun.chen@cyberkey.cn
   }
   notification_email_from rabbitmq_master@cyberkey.cn
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id RABBITMQ-HA
}
vrrp_script chk_rabbitmq {
    script "rabbitmqctl status"
    interval 2
    timeout 4
    weight -100
    fall 2
    rise 1
}
vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 56
    priority 200
    advert_int 1
    nopreempt
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.1.95/24 dev eth0 scope global label eth0:0
    }
    track_script {
        chk_rabbitmq
    }
}

slave配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
[root@localhost ~]# vim /etc/keepalived/keepalived.conf
global_defs {
   notification_email {
     jun.chen@cyberkey.cn
   }
   notification_email_from rabbitmq_slave@cyberkey.cn
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id RABBITMQ-HA
}
vrrp_script chk_rabbitmq {
    script "rabbitmqctl status"
    interval 2
    timeout 4
    weight -100
    fall 2
    rise 1
}
vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 56
    priority 150
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.1.95/24 dev eth0 scope global label eth0:0
    }
    track_script {
        chk_rabbitmq
    }
}

防火墙设置

1
2
3
$ vim /etc/sysconfig/iptables
-A INPUT -p vrrp -j ACCEPT
$ service iptables restart

测试效果

启动master和slave的keepalived，尝试手动停止rabbitmq-server进行切换测试。