搭建Keepalived Nginx高可用Web集群

发表于 | 分类于 ，

搭建Keepalived + Nginx高可用服务集群

一、操作场景

虚拟IP（VIP）主要用于弹性云服务器的主备切换，达到高可用性HA（High Availability）的目的。当主服务器发生故障无法对外提供服务时，动态将虚拟IP切换到备服务器，继续对外提供服务。

本文档以弹性云服务器的CentOS 7.5 64位操作系统为例，实现Keepalived+Nginx高可用服务集群搭建。

二、背景知识

三、网络拓扑

数据规划如下：

表1 数据规划

序号 产品 数量 规格
1 虚拟私有云（VPC） 1 192.168.0.0/16
子网（subnet） 1 192.168.0.0/24
2 弹性云服务器（ECS） 2 4vCPUs 8GB CentOS 7.5 64bit
子网IP（subnet IP） 2 ecs-HA1：192.168.192.39
ecs-HA2：192.168.192.41
3 弹性公网IP（EIP） 1 116.63.115.186
虚拟IP（VIP） 1 192.168.192.100

实现方式如下：

  • 将2台同子网的弹性云服务器配置Keepalived，一台为主服务器，一台为备份服务器。
  • 将这2台弹性云服务器绑定同一个虚拟IP。
  • 将虚拟IP与弹性公网IP绑定，从互联网可以访问绑定了该虚拟IP地址的主备云服务器。

图1 组网图

三、操作步骤

1. nginx
1.1 在ECS-HA1和ECS-HA2分别安装Nginx

  • RPM包地址：http://nginx.org/packages/centos/7/x86_64/RPMS/

    1
    2
    $ wget http://nginx.org/packages/centos/7/x86_64/RPMS/nginx-1.20.2-1.el7.ngx.x86_64.rpm 
    $ rpm -ivh nginx-1.20.2-1.el7.ngx.x86_64.rpm

  • yum安装

    1
    2
    3
    4
    $ yum install nginx
    $ systemctl start nginx
    $ systemctl enable nginx
    $ systemctl status nginx
1.2 配置Nginx

  • 主机ECS-HA1 nginx.conf配置

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    82
    83
    84
    # For more information on configuration, see:
    #   * Official English Documentation: http://nginx.org/en/docs/
    #   * Official Russian Documentation: http://nginx.org/ru/docs/

    user nginx;
    worker_processes auto;
    error_log /mnt/sse/log/nginx/error.log;
    pid /run/nginx.pid;

    # Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.
    include /usr/share/nginx/modules/*.conf;

    events {
        worker_connections 1024;
    }

    http {
        log_format  main  '$status $body_bytes_sent "$http_referer" '
                          '"$http_user_agent" "$http_x_forwarded_for"'
                          '"rl=$request_length" $remote_addr - $remote_user [$time_local] "$request"'
                          'rt=$request_time uct="$upstream_connect_time" uht="$upstream_header_time" urt="$upstream_response_time"';

        access_log  /mnt/sse/log/nginx/access.log  main;

        sendfile            on;
        tcp_nopush          on;
        tcp_nodelay         on;
        keepalive_timeout   65;
        types_hash_max_size 4096;

        include             /etc/nginx/mime.types;
        default_type        application/octet-stream;

        # Load modular configuration files from the /etc/nginx/conf.d directory.
        # See http://nginx.org/en/docs/ngx_core_module.html#include
        # for more information.
        include /etc/nginx/conf.d/*.conf;

        server {
            listen       80;
            listen       [::]:80;
            server_name  _;
            root         /usr/share/nginx/html;

            # Load configuration files for the default server block.
            include /etc/nginx/default.d/*.conf;

            error_page 404 /404.html;
            location = /404.html {
            }

            error_page 500 502 503 504 /50x.html;
            location = /50x.html {
            }
        }

    # Settings for a TLS enabled server.
    #
    #    server {
    #        listen       443 ssl http2;
    #        listen       [::]:443 ssl http2;
    #        server_name  _;
    #        root         /usr/share/nginx/html;
    #
    #        ssl_certificate "/etc/pki/nginx/server.crt";
    #        ssl_certificate_key "/etc/pki/nginx/private/server.key";
    #        ssl_session_cache shared:SSL:1m;
    #        ssl_session_timeout  10m;
    #        ssl_ciphers HIGH:!aNULL:!MD5;
    #        ssl_prefer_server_ciphers on;
    #
    #        # Load configuration files for the default server block.
    #        include /etc/nginx/default.d/*.conf;
    #
    #        error_page 404 /404.html;
    #            location = /40x.html {
    #        }
    #
    #        error_page 500 502 503 504 /50x.html;
    #            location = /50x.html {
    #        }
    #    }

    }

  • 主机ECS-HA1 /usr/share/nginx/html/index.html修改

    1
    Welcome to ECS-HA1

  • 备机ECS-HA2 nginx.conf配置

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    82
    83
    84
    # For more information on configuration, see:
    #   * Official English Documentation: http://nginx.org/en/docs/
    #   * Official Russian Documentation: http://nginx.org/ru/docs/

    user nginx;
    worker_processes auto;
    error_log /mnt/sse/log/nginx/error.log;
    pid /run/nginx.pid;

    # Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.
    include /usr/share/nginx/modules/*.conf;

    events {
        worker_connections 1024;
    }

    http {
        log_format  main  '$status $body_bytes_sent "$http_referer" '
                          '"$http_user_agent" "$http_x_forwarded_for"'
                          '"rl=$request_length" $remote_addr - $remote_user [$time_local] "$request"'
                          'rt=$request_time uct="$upstream_connect_time" uht="$upstream_header_time" urt="$upstream_response_time"';

        access_log  /mnt/sse/log/nginx/access.log  main;

        sendfile            on;
        tcp_nopush          on;
        tcp_nodelay         on;
        keepalive_timeout   65;
        types_hash_max_size 4096;

        include             /etc/nginx/mime.types;
        default_type        application/octet-stream;

        # Load modular configuration files from the /etc/nginx/conf.d directory.
        # See http://nginx.org/en/docs/ngx_core_module.html#include
        # for more information.
        include /etc/nginx/conf.d/*.conf;

        server {
            listen       80;
            listen       [::]:80;
            server_name  _;
            root         /usr/share/nginx/html;

            # Load configuration files for the default server block.
            include /etc/nginx/default.d/*.conf;

            error_page 404 /404.html;
            location = /404.html {
            }

            error_page 500 502 503 504 /50x.html;
            location = /50x.html {
            }
        }

    # Settings for a TLS enabled server.
    #
    #    server {
    #        listen       443 ssl http2;
    #        listen       [::]:443 ssl http2;
    #        server_name  _;
    #        root         /usr/share/nginx/html;
    #
    #        ssl_certificate "/etc/pki/nginx/server.crt";
    #        ssl_certificate_key "/etc/pki/nginx/private/server.key";
    #        ssl_session_cache shared:SSL:1m;
    #        ssl_session_timeout  10m;
    #        ssl_ciphers HIGH:!aNULL:!MD5;
    #        ssl_prefer_server_ciphers on;
    #
    #        # Load configuration files for the default server block.
    #        include /etc/nginx/default.d/*.conf;
    #
    #        error_page 404 /404.html;
    #            location = /40x.html {
    #        }
    #
    #        error_page 500 502 503 504 /50x.html;
    #            location = /50x.html {
    #        }
    #    }

    }

  • 备机ECS-HA2 /usr/share/nginx/html/index.html修改

    1
    Welcome to ECS-HA2
2. Keepalived
1.1 在ECS-HA1和ECS-HA2分别安装Keepalived
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
$ wget --no-check-certificate https://www.keepalived.org/software/keepalived-2.2.4.tar.gz
$ tar -xvzf keepalived-2.2.4.tar.gz
$ yum -y install openssl-devel
$ cd keepalived-2.2.4/
$ ./configure --prefix=/usr/local/keepalived
$ make
$ make install
$ mkdir /etc/keepalived
$ cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf
$ cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/keepalived
# 修改/lib/systemd/system/keepalived.service，设置EnvironmentFile=-/etc/sysconfig/keepalived
$ vim /lib/systemd/system/keepalived.service
$ systemctl start keepalived
$ systemctl enable keepalived
$ systemctl status keepalived.service
1.2 配置主机ECS-HA1Keepalived

  • 编辑keepalived.conf

    1
    vim /etc/keepalived/keepalived.conf

  • 修改keepalived.conf

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    ! Configuration File for keepalived
    global_defs {
        router_id master-node
    }
    vrrp_script chk_http_port {
        script  "/etc/keepalived/chk_nginx.sh"
        interval 2
        weight -5
        fall 2
        rise 1
    }
    vrrp_instance VI_1 {
         state MASTER
         interface eth0
         mcast_src_ip 192.168.192.39
         virtual_router_id 51
         priority 101
         advert_int 1
         authentication {
            auth_type PASS
            auth_pass 1111
         }
         unicast_src_ip 192.168.192.39
         virtual_ipaddress {
             192.168.192.100
         }
        notify_master "/etc/keepalived/keepalived_notify.sh MASTER" # 当切换到master状态时执行脚本
        notify_backup "/etc/keepalived/keepalived_notify.sh BACKUP" # 当切换到backup状态时执行脚本
        notify_fault "/etc/keepalived/keepalived_notify.sh FAULT" # 当切换到fault状态时执行脚本
        notify_stop "/etc/keepalived/keepalived_notify.sh STOP" # 当切换到stop状态时执行脚本
        garp_master_delay 1    # 设置当切为主状态后多久更新ARP缓存
        garp_master_refresh 5   # 设置主节点发送ARP报文的时间间隔
        # 跟踪接口，里面任意一块网卡出现问题，都会进入故障(FAULT)状态
        track_interface {
          eth0
        }
         track_script {
             chk_http_port
         }
    }

  • 创建/etc/keepalived/chk_nginx.sh脚本文件，使用chmod +x /etc/keepalived/chk_nginx.sh将文件变成可执行文件，并增加如下代码

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    #!/bin/bash
    counter=$(ps -C nginx --no-heading|wc -l)
    if [ "${counter}" = "0"  ]; then
         systemctl start nginx.service
         sleep 2
         counter=$(ps -C nginx  --no-heading|wc -l)
         if [ "${counter}" =  "0" ]; then
              systemctl stop keepalived.service
         fi
    fi

  • 创建/etc/keepalived/keepalived_notify脚本文件，使用chmod +x /etc/keepalived/keepalived_notify将文件变成可执行文件，并增加如下代码

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    #!/usr/bin/env bash

    # Copyright 2021 He Chen <chenhe@zuoyejia.com>. All rights reserved.
    # Use of this source code is governed by a MIT style
    # license that can be found in the LICENSE file.

    # /etc/keepalived/keepalived_notify.sh
    log_file=/var/log/keepalived.log

    sse::keepalived::mail() {
      # 这里可以添加email逻辑，当keepalived变动时及时告警
      :
     sendEmail -f chenhe@zuoyejia.com -t chenhe@zuoyejia.com -s "smtp.exmail.qq.com:587" -u "作业家Nginx+Keepalived状态通知" -o message-charset=utf-8 -xu chenhe@zuoyejia.com -xp agcCeRn3v42Cohkk -m "$1"
    }
    sse::keepalived::log() {
        echo "[`date '+%Y-%m-%d %T'`] $1" >> ${log_file}
    }

    [ ! -d /var/keepalived/ ] && mkdir -p /var/keepalived/

    case "$1" in
        "MASTER" )
            sse::keepalived::log "notify_master"
            sse::keepalived::mail "notify_master"
            ;;
        "BACKUP" )
            sse::keepalived::log "notify_backup"
            sse::keepalived::mail "notify_backup"
            ;;
        "FAULT" )
            sse::keepalived::log "notify_fault"
            sse::keepalived::mail "notify_fault"
            ;;
        "STOP" )
            sse::keepalived::log "notify_stop"
            sse::keepalived::mail "notify_stop"
            ;;
        *)
            sse::keepalived::log "keepalived_notify.sh: state error!"
            ;;
    esac
1.3 配置备机ECS-HA2 Keepalived

  • 编辑keepalived.conf

    1
    vim /etc/keepalived/keepalived.conf

  • 修改keepalived.conf

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    ! Configuration File for keepalived
    global_defs {
        router_id master-node
        script_user root
        enable_script_security
    }
    vrrp_script chk_http_port {
         script  "/etc/keepalived/chk_nginx.sh"
         interval 2
         weight -5
         fall 2
         rise 1
    }
    vrrp_instance VI_1 {
         state BACKUP
         interface eth0
         mcast_src_ip 192.168.192.41
         virtual_router_id 51
         priority 100
         advert_int 1
         authentication {
              auth_type PASS
              auth_pass 1111
         }
        unicast_src_ip 192.168.192.41
        virtual_ipaddress {
            192.168.192.100
        }
        notify_master "/etc/keepalived/keepalived_notify.sh MASTER" # 当切换到master状态时执行脚本
        notify_backup "/etc/keepalived/keepalived_notify.sh BACKUP" # 当切换到backup状态时执行脚本
        notify_fault "/etc/keepalived/keepalived_notify.sh FAULT" # 当切换到fault状态时执行脚本
        notify_stop "/etc/keepalived/keepalived_notify.sh STOP" # 当切换到stop状态时执行脚本
        garp_master_delay 1    # 设置当切为主状态后多久更新ARP缓存
        garp_master_refresh 5   # 设置主节点发送ARP报文的时间间隔
        # 跟踪接口，里面任意一块网卡出现问题，都会进入故障(FAULT)状态
        track_interface {
            eth0
        }
        track_script {
            chk_http_port
        }
    }

  • 创建/etc/keepalived/chk_nginx.sh脚本文件，使用chmod +x /etc/keepalived/chk_nginx.sh将文件变成可执行文件，并增加如下代码

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    #!/bin/bash
    counter=$(ps -C nginx --no-heading|wc -l)
    if [ "${counter}" = "0"  ]; then
         systemctl start nginx.service
         sleep 2
         counter=$(ps -C nginx  --no-heading|wc -l)
         if [ "${counter}" =  "0" ]; then
              systemctl stop keepalived.service
         fi
    fi

  • 创建/etc/keepalived/keepalived_notify脚本文件，使用chmod +x /etc/keepalived/keepalived_notify将文件变成可执行文件，并增加如下代码

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    #!/usr/bin/env bash

    # Copyright 2021 He Chen <chenhe@zuoyejia.com>. All rights reserved.
    # Use of this source code is governed by a MIT style
    # license that can be found in the LICENSE file.

    # /etc/keepalived/keepalived_notify.sh
    log_file=/var/log/keepalived.log

    sse::keepalived::mail() {
      # 这里可以添加email逻辑，当keepalived变动时及时告警
      :
     sendEmail -f chenhe@zuoyejia.com -t chenhe@zuoyejia.com -s "smtp.exmail.qq.com:587" -u "作业家Nginx+Keepalived备机状态通知" -o message-charset=utf-8 -xu chenhe@zuoyejia.com -xp agcCeRn3v42Cohkk -m "$1"
    }
    sse::keepalived::log() {
        echo "[`date '+%Y-%m-%d %T'`] $1" >> ${log_file}
    }

    [ ! -d /var/keepalived/ ] && mkdir -p /var/keepalived/

    case "$1" in
        "MASTER" )
            sse::keepalived::log "nginx backpu notify_master"
            sse::keepalived::mail "nginx backpu notify_master"
            ;;
        "BACKUP" )
            sse::keepalived::log "nginx backpu notify_backup"
            sse::keepalived::mail "nginx backpu notify_backup"
            ;;
        "FAULT" )
            sse::keepalived::log "nginx backpu notify_fault"
            sse::keepalived::mail "nginx backpu notify_fault"
            ;;
        "STOP" )
            sse::keepalived::log "nginx backpu notify_stop"
            sse::keepalived::mail "nginx backpu notify_stop"
            ;;
        *)
            sse::keepalived::log "nginx backpu keepalived_notify.sh: state error!"
            ;;
    esac
3. sendEmail
1.1 在ECS-HA1和ECS-HA2分别安装sendEmail
1
yum install sendEmail
1.2. 以腾讯企业邮箱为例，配置安全密码

  • 登录腾讯企业邮箱，打开设置

  • 选择安全设置，生成专用密码。（如果没开启安全登录，点击开启）

  • 将安全密码写入/etc/keepalived/keepalived_notify脚本文件中

4. 重启ECS-HA1和ECS-HA2 Nginx、Keepalived
1
2
$ nginx -s reload
$ systemctl restart keepalived
5. 绑定虚拟IP
  1. 登录管理控制台。
  2. 选择“服务列表 > 网络 > 虚拟私有云”。
  3. 在左侧导航栏选择“子网”。
  4. 在“子网”列表中，单击子网名称。
  5. 选择“IP地址管理”页签，在虚拟IP所在行的操作列下单击“绑定服务器”。
  6. 在弹出的页面，选择ecs HA1和ecs HA2服务器。
  7. 选择“IP地址管理”页签，在虚拟IP所在行的操作列下单击“绑定弹性公网IP”。
  8. 在弹出的页面，选择弹性公网IP 116.63.115.186
6. 域名解析

添加A类DNS域名解析，将sseserverzj2.zuoyejia.com绑定为上述公网ip

7. 验证结果

  1. 通过管理控制台远程登录到ecs-HA1。

  2. 执行以下命令，查看虚拟IP是否有绑定到ecs-HA1的eth0网卡上。
    ip addr show

    如下图所示表示虚拟IP已经绑定到ecs-HA1的eth0网卡上。

  3. 访问验证ecs-HA1

  4. 执行以下命令，停止ecs-HA1上的keepalived服务。
    systemctl stop keepalived.service

  5. 执行以下命令，查看服务器ecs-HA2是否有接管虚拟IP。
    ip addr show
    如果所示

  6. 访问验证ecs-HA2

  7. 查看自己的邮箱是否有keepalived的状态邮件
    image-20211208144744870

四、常用命令

Nginx
1
2
# 不停止对客户服务的情况下，重新加载配置文件
nginx -s reload