wiki

# Protonmail - System Test Report

This report was made to explain all the steps I did during the System Test provided by Protonmail during the recruitment process.

# Create Mail Account

  • Go to https://protonmail.com
  • Create a free account
  • share with the examinator

# Create a VPN Account

  • Go to https://account.protonvpn.com/signup
  • create a new free account
  • if you already have a protonmail account, it will work.

NOTE: a IKEv2 configuration (username/password) is available on the account page. You can use it to create your own openvpn configuration.

# Using ProtonVPN cli on OpenBSD

ProtonVPN is based on OpenVPN. To use it you must install these packages:

doas pkg_add python3 openvpn

ProtonVPN is an open-source project. The code source is available on Github.

mkdir src
git clone https://github.com/protonvpn/protonvpn-cli.git
git clone https://github.com/protonvpn/protonvpn-cli-ng.git

# ProtonVPN-cli

This version is not supported anymore. A ticket was open in 2019 to port the shell script on OpenBSD.

# ProtonVPN-cli-ng

After created the clone of the repository.

cd protonvpn-cli-ng
python setup.py build
doas python setup.py intsall

protonvpn-cli-ng require some Linux commands to work, like ip (used to manage route table). Here a quick and dirty patch to solve this problem.

diff --git a/protonvpn_cli/utils.py b/protonvpn_cli/utils.py
index 076f64d..f05b380 100644
--- a/protonvpn_cli/utils.py
+++ b/protonvpn_cli/utils.py
@@ -333,7 +333,7 @@ def check_root():
         sys.exit(1)
     else:
         # Check for dependencies
-        dependencies = ["openvpn", "ip", "sysctl", "pgrep", "pkill"]
+        dependencies = ["openvpn"] # , "ip", "sysctl", "pgrep", "pkill"]
         for program in dependencies:
             check = subprocess.run(["which", program],
                                    stdout=subprocess.PIPE,

This patch is useful only to initialize a configuration with protonvpn init. Note: the file /usr/local/bin/protonvpn was not working due to the shebang (set to #!.). A simple modification of the file make it usable by adding a correct shebang (#!/usr/local/bin/python3).

To understand how this tool is working, I will with python debugger module (pdb) and try to see if something goes wrong.

doas python3 -m pdb $(which protonvpn)

This tool create all the configuration in /root/.pvpn-cli. OpenVPN configuration file is stored in /root/.pvpn-cli/pvpn-cli.cfg.

[USER]
username = ${USERNAME}
tier = 0
default_protocol = udp
initialized = 1
dns_leak_protection = 1
custom_dns = None
check_update_interval = 3
killswitch = 0

[metadata]
last_api_pull = 1587137886
last_update_check = 1587137881

The user's credentials are stored in raw (!) in the /root/.pvpn-cli/pvpnpass file!

${USER_PASSWORD}

Another file is created, probably from a remote API call, called /root/.vpn-cli/serverinfo.json and contain all available server. That pretty interesting. Here some command to make some queries with jq.

# list available servers in a country (e.g. CA)
cat serverinfo.json \
  | jq '.LogicalServers[] | select(.EntryCountry == "CA")'

# select all IP in a country
cat serverinfo.json \
  | jq '.LogicalServers[] | select(.EntryCountry == "CA") | .Servers[].EntryIP'

# list all available IP
cat serverinfo.json \
  | jq ".LogicalServers[].Servers[].EntryIP"

A template is also created, called template.ovpn:

# ==============================================================================
# Copyright (c) 2016-2017 ProtonVPN A.G. (Switzerland)
# Email: contact@protonvpn.com
#
# The MIT License (MIT)
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all
# copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR # OTHERWISE, ARISING
# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
# IN THE SOFTWARE.
# ==============================================================================

client
dev tun


remote-random
resolv-retry infinite
nobind
cipher AES-256-CBC
auth SHA512
comp-lzo no
verb 3

tun-mtu 1500
tun-mtu-extra 32
mssfix 1450
persist-key
persist-tun

reneg-sec 0

remote-cert-tls server
auth-user-pass
pull
fast-io


<ca>
-----BEGIN CERTIFICATE-----
MIIFozCCA4ugAwIBAgIBATANBgkqhkiG9w0BAQ0FADBAMQswCQYDVQQGEwJDSDEV
MBMGA1UEChMMUHJvdG9uVlBOIEFHMRowGAYDVQQDExFQcm90b25WUE4gUm9vdCBD
QTAeFw0xNzAyMTUxNDM4MDBaFw0yNzAyMTUxNDM4MDBaMEAxCzAJBgNVBAYTAkNI
MRUwEwYDVQQKEwxQcm90b25WUE4gQUcxGjAYBgNVBAMTEVByb3RvblZQTiBSb290
IENBMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAt+BsSsZg7+AuqTq7
vDbPzfygtl9f8fLJqO4amsyOXlI7pquL5IsEZhpWyJIIvYybqS4s1/T7BbvHPLVE
wlrq8A5DBIXcfuXrBbKoYkmpICGc2u1KYVGOZ9A+PH9z4Tr6OXFfXRnsbZToie8t
2Xjv/dZDdUDAqeW89I/mXg3k5x08m2nfGCQDm4gCanN1r5MT7ge56z0MkY3FFGCO
qRwspIEUzu1ZqGSTkG1eQiOYIrdOF5cc7n2APyvBIcfvp/W3cpTOEmEBJ7/14RnX
nHo0fcx61Inx/6ZxzKkW8BMdGGQF3tF6u2M0FjVN0lLH9S0ul1TgoOS56yEJ34hr
JSRTqHuar3t/xdCbKFZjyXFZFNsXVvgJu34CNLrHHTGJj9jiUfFnxWQYMo9UNUd4
a3PPG1HnbG7LAjlvj5JlJ5aqO5gshdnqb9uIQeR2CdzcCJgklwRGCyDT1pm7eoiv
WV19YBd81vKulLzgPavu3kRRe83yl29It2hwQ9FMs5w6ZV/X6ciTKo3etkX9nBD9
ZzJPsGQsBUy7CzO1jK4W01+u3ItmQS+1s4xtcFxdFY8o/q1zoqBlxpe5MQIWN6Qa
lryiET74gMHE/S5WrPlsq/gehxsdgc6GDUXG4dk8vn6OUMa6wb5wRO3VXGEc67IY
m4mDFTYiPvLaFOxtndlUWuCruKcCAwEAAaOBpzCBpDAMBgNVHRMEBTADAQH/MB0G
A1UdDgQWBBSDkIaYhLVZTwyLNTetNB2qV0gkVDBoBgNVHSMEYTBfgBSDkIaYhLVZ
TwyLNTetNB2qV0gkVKFEpEIwQDELMAkGA1UEBhMCQ0gxFTATBgNVBAoTDFByb3Rv
blZQTiBBRzEaMBgGA1UEAxMRUHJvdG9uVlBOIFJvb3QgQ0GCAQEwCwYDVR0PBAQD
AgEGMA0GCSqGSIb3DQEBDQUAA4ICAQCYr7LpvnfZXBCxVIVc2ea1fjxQ6vkTj0zM
htFs3qfeXpMRf+g1NAh4vv1UIwLsczilMt87SjpJ25pZPyS3O+/VlI9ceZMvtGXd
MGfXhTDp//zRoL1cbzSHee9tQlmEm1tKFxB0wfWd/inGRjZxpJCTQh8oc7CTziHZ
ufS+Jkfpc4Rasr31fl7mHhJahF1j/ka/OOWmFbiHBNjzmNWPQInJm+0ygFqij5qs
51OEvubR8yh5Mdq4TNuWhFuTxpqoJ87VKaSOx/Aefca44Etwcj4gHb7LThidw/ky
zysZiWjyrbfX/31RX7QanKiMk2RDtgZaWi/lMfsl5O+6E2lJ1vo4xv9pW8225B5X
eAeXHCfjV/vrrCFqeCprNF6a3Tn/LX6VNy3jbeC+167QagBOaoDA01XPOx7Odhsb
Gd7cJ5VkgyycZgLnT9zrChgwjx59JQosFEG1DsaAgHfpEl/N3YPJh68N7fwN41Cj
zsk39v6iZdfuet/sP7oiP5/gLmA/CIPNhdIYxaojbLjFPkftVjVPn49RqwqzJJPR
N8BOyb94yhQ7KO4F3IcLT/y/dsWitY0ZH4lCnAVV/v2YjWAWS3OWyC8BFx/Jmc3W
DK/yPwECUcPgHIeXiRjHnJt0Zcm23O2Q3RphpU+1SO3XixsXpOVOYP6rJIXW9bMZ
A1gTTlpi7A==
-----END CERTIFICATE-----
</ca>

key-direction 1
<tls-auth>
# 2048 bit OpenVPN static key
-----BEGIN OpenVPN Static key V1-----
6acef03f62675b4b1bbd03e53b187727
423cea742242106cb2916a8a4c829756
3d22c7e5cef430b1103c6f66eb1fc5b3
75a672f158e2e2e936c3faa48b035a6d
e17beaac23b5f03b10b868d53d03521d
8ba115059da777a60cbfd7b2c9c57472
78a15b8f6e68a3ef7fd583ec9f398c8b
d4735dab40cbd1e3c62a822e97489186
c30a0b48c7c38ea32ceb056d3fa5a710
e10ccc7a0ddb363b08c3d2777a3395e1
0c0b6080f56309192ab5aacd4b45f55d
a61fc77af39bd81a19218a79762c3386
2df55785075f37d8c71dc8a42097ee43
344739a0dd48d03025b0450cf1fb5e8c
aeb893d9a96d1f15519bb3c4dcb40ee3
16672ea16c012664f8a9f11255518deb
-----END OpenVPN Static key V1-----
</tls-auth>

A log file is also present, called pvpn-cli.log.

# Manual connection with OpenVPN

openvpn --user _openvpn --group _openvpn \
        --config ${protonmail_template} \
        --remote {remote_ip}

Without certificate, the credentials used will be available from protonvpn account (OpenVPN / IKEv2 username section). If authentication is a success, you should receive routes:

Wed Apr 22 16:06:44 2020 TUN/TAP device /dev/tun0 opened
Wed Apr 22 16:06:44 2020 /sbin/ifconfig tun0 10.8.8.5 10.8.8.1 mtu 1500 netmask 255.255.255.0 up
Wed Apr 22 16:06:44 2020 /sbin/route add -net 10.8.8.0 -netmask 255.255.255.0 10.8.8.1
add net 10.8.8.0: gateway 10.8.8.1
Wed Apr 22 16:06:44 2020 /sbin/route add -net 194.59.249.20 -netmask 255.255.255.255 192.168.0.254
add net 194.59.249.20: gateway 192.168.0.254
Wed Apr 22 16:06:44 2020 /sbin/route add -net 0.0.0.0 -netmask 128.0.0.0 10.8.8.1
add net 0.0.0.0: gateway 10.8.8.1
Wed Apr 22 16:06:44 2020 /sbin/route add -net 128.0.0.0 -netmask 128.0.0.0 10.8.8.1
add net 128.0.0.0: gateway 10.8.8.1

# ProtonVPN API

The main URL is https://api.protonvpn.ch. Here the different end-point from the source code of protonvpn-api.

# /vpn/logicals

Used to retrieve the list of all servers.

curl https://api.protonvpn.ch/vpn/logicals

# /vpn/location

Used to retrieve the public IP address.

curl https://api.protonvpn.ch/vpn/location

# /test/ping

Used to check the connection. Return a 404 not found when tested.

curl https://api.protonvpn.ch/test/ping

# /vpn/config

Return an OpenVPN configuration template. The id param come from the /vpn/logical end-point and refer the a server id. This will generate a configuration based on the IP address(es) of this server.

platform="Linux"
protocol="tcp"
id="BzHqSTaqcpjIY9SncE5s7FpjBrPjiGOucCyJmwA6x4nTNqlElfKvCQFr9xUa2KgQxAiHv4oQQmAkcA56s3ZiGQ=="
curl "https://api.protonvpn.ch/vpn/config?Platform=${platform}&LogicalID=${id}&Protocol=${protocol}"

# ProtonMail System Test - Report

The goal of the test is to setup an automatic failover for HAProxy load balancer using keepalived. It will involve the installation and configuration of a dummy “hello, world!†HTTP server back end, the HAProxy in a VM, replication of the HAProxy to a second VM instance and a setup of the keepalived service.

The VM setup and replication themselves are not directly connected to the failover functionality but are just as important components of the test.

The whole test consists of 3 parts:

  • Setup a guest virtual machine V1 on T3 using KVM. V1 needs to have the public IP address 46.20.249.165 using bridged networking, see references for some details and guides. Install HAProxy on V1 with the 46.20.249.165 front end. Setup a simple “hello, world!†HTTP back end on T3. Configure HAProxy to use the T3 HTTP back end. Now, you should be able to request http://46.20.249.165 and get the “hello, world!†response served by the T3 back end. If you pause V1, the request should time out.

  • Replicate V1 to a 2nd virtual machine V2 with the public IP 46.20.249.166. You should now be able to request http://46.20.249.166 and get the same response served by the same back end as if you requested http://46.20.249.165.

  • Install keepalived on V1 and V2 and configure it for an automated failover from V1 to V2 through the transfer of the virtual IP 46.20.249.167 from V1 to V2. Now, you should be able to request http://46.20.249.167 and get the “hello, world!†response served by the T3 back end through V1. If you pause V1 now, the http://46.20.249.167 request should not time out anymore as in step 1. Instead, it should be served through V2, which automatically takes over the VIP 46.20.249.167. In addition to the failure of the whole V1 machine, the automatic failover should happen also in the cases when either the V1 HAProxy goes down (e.g. we intentionally kill the process) or the V1 network interface goes down.

For the static IPv4 networking of the VM public interfaces (gateway, subnet mask, DNS hosts), use the same settings as are used for T3.

Optionally, as time allows, pick one or more of the following tasks :

  • Use Vagrant for instantiating the VMs with a base Linux OS

  • Use scripts, Ansible or Puppet to automate the configuration of the VMs

  • Put the HTTP back-end in a third VM V3, in a private network

  • Improve on the security of the systems, according to your preferences

# Tasks

  • install toolbox on T3 server: done

  • install qemu on T3 server: done

  • install nginx on T3 server: done

  • install curl on T3 server: done

  • install vagrant on T3 server: done

  • install ansible on T3 server: done

  • configure nginx on T3 server: done

  • disable NetworkManager: done

  • create a bridge on T3 server: done

  • create a kvm guest V1 on T3 server: done

  • configure network on V1: done

  • install haproxy on V1: done

  • configure haproxy on V1: done

  • replicate V1 guest to V2: done

  • configure network on V2: done

  • install keepalived on V1: done

  • install keepalived on V2: done

  • configure NAT on v3 with iptables: doesn't work

    • probably due to firewalld. did not check

  • create ansible configuration for host: done

  • create a nginx role: done

  • create an haproxy role: done

  • create a keepalived role: done

  • create other ansible tasks: done

  • test network: done

  • test failover: done

# Final Configuration Summary

  • 46.20.249.165 (V1): kvm guest
    • haproxy
    • keepalived
  • 46.20.249.166 (V2): kvm guest (V1 replication)
    • haproxy
    • keepalived
  • 46.20.249.168 (T3): host server
    • user: icandidate2
    • centos
    • http backend
  • 46.20.249.171 (S1): Dell iDrac access
  • 10.0.0.2/24 (V3): backend on private network
    • nginx

# Network Map


 [46.20.249.161]       [46.20.249.168]        [10.0.0.2/24]
    ________  /   ____/     _______      ____/
   (        )/   |    |    (       )    |    |
  ( internet )---| T3 |---( private )---| V3 |
   (________)    |____|    (_______)    |____|
                    |               \
                    |                [10.0.0.1/24]
                    |
                    |               [46.20.249.165]
                  __|___       ____/
                 (      )     |    |
                ( bridge )----| V1 |
                 (______)     |____|
                    |       /
                    |      ( [46.20.249.167]
                    |       \  ____ 
                    |         |    |
                    +---------| V2 |
                              |____|
                                    \
                                     [46.20.249.166]

# Connexion T3 server

A VPN connexion is required to have the right to connect to this server. OpenVPN is used, here an example on openbsd:

doas openvpn --user _openvpn --group _openvpn --config ${config}

Ensure your SSH public key is well configured too. You should have access to t3 server with icandidate2 user.

ssh 46.20.249.168 -l icandidate2

# T3 Server configuration

uname -a
# output:
# Linux t3 3.10.0-1062.18.1.el7.x86_64 #1 SMP Tue Mar 17 23:49:17 
# UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

A quick review of the server configuration before doing important modification on it. The server uses LVM and the toolbox. It could be possible to create LVM snapshot for fast recovery.

pvdisplay
vgdisplay
lvdisplay

It has only one active network interface (em1):

ip a

The raw output of the command:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
    valid_lft forever preferred_lft forever
2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether d4:ae:52:8e:bb:50 brd ff:ff:ff:ff:ff:ff
    inet 46.20.249.168/28 brd 46.20.249.175 scope global noprefixroute em1
    valid_lft forever preferred_lft forever
    inet6 fe80::42d7:2ac0:328:cf56/64 scope link noprefixroute 
    valid_lft forever preferred_lft forever
3: em2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether d4:ae:52:8e:bb:51 brd ff:ff:ff:ff:ff:ff
4: p1p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 90:e2:ba:82:4d:70 brd ff:ff:ff:ff:ff:ff
5: p1p2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 90:e2:ba:82:4d:71 brd ff:ff:ff:ff:ff:ff

# Pre-Configuration and Toolbox installation

First thing first, ensuring the server is up to date.

sudo yum update

During the configuration, and because I don't have any other way to test or ensure the connection will be up during all the exercice, I will install tmux.

sudo yum install tmux

T3 Server will need nginx (http backend) and qemu/kvl (for virtualization part). Curl will be used to check http services.

sudo yum install nginx qemu qemu-kvm curl

Ansible was selected to install and configure the different virtual machines installed on T3 server.

sudo yum install ansible

Vagrant will also be used. Hashicorp GPG key can be retrieved on hashicorp security page (opens new window)

curl -Os https://releases.hashicorp.com/vagrant/2.2.7/vagrant_2.2.7_x86_64.rpm
curl -Os https://releases.hashicorp.com/vagrant/2.2.7/vagrant_2.2.7_SHA256SUMS
curl -Os https://releases.hashicorp.com/vagrant/2.2.7/vagrant_2.2.7_SHA256SUMS.sig

Before installing vagrant, a verifification is needed, ensuring the package was correctly downloaded from the website without alteration.

gpg --import hashicorp.asc
gpg --verify vagrant_2.2.7_SHA256SUMS.sig vagrant_2.2.7_SHA256SUMS
sha256sum -c vagrant_2.2.7_SHA256SUMS

If everything fine, vagrant RPM file can be installed

yum localinstall vagrant_2.2.7_x86_64.rpm

vagrant command is now available from the command line

vagrant

# HTTP back-end configuration on T3

The backend on T3 will have nginx configured to listen on all interfaces. The "hello, world!" application will be only a simple text file containing this string.

nginx is configuration is located in /etc/nginx/nginx.conf and files are stored by default in /usr/share/nginx/html. We can modify the configuration in many different way, but, because its a default backend application used only for test, the fastest way is to modify files in /usr/share/nginx/html:

cd /usr/share/nginx/html
sudo mv index.html index.html.bck
sudo sh -c 'echo "Hello, world!" > index.html'

nginx service can now be enabled and started.

sudo systemctl enable nginx
sudo systemctl start nginx

To check if the service is running, systemctl status command be used and to test it, curl can be used.

curl localhost
# output:
# Hello, world!

curl 46.20.249.168
# output:
# Hello, world!

nginx service is up and running, it returns the desired value.

# Static Network Configuration and NetworkManager

During the test phase, NetworkManager is only stopped.

sudo systemctl stop NetworkManager

To disable this service completely, it is also possible to reuse systemctl. It will not be used in this

sudo systemctl disable NetworkManager

# Bridge Configuration

NOTE: this part was tested before on personal virtual machine. Creating bridge and alter network configuration can result in a network loss.

#!/bin/sh
######################################################################
# bridge fallback script for testing
# will migrate the configuration for 15 seconds to ensure
# everything is fine. After that, it will fallback to the previous
# configuration.
######################################################################

WAN=em1
WAN_IP=46.20.249.168/28
BRIDGE=br0
GATEWAY=46.20.249.161

_eval() {
    printf "execute: %s\n" "${*}"
    eval ${*}
}

_init() {
    _eval ip link add ${BRIDGE} type bridge
}

_clean() {
    _eval ip link del ${BRIDGE}
}

_do() {
    _eval ip link set ${BRIDGE} up
    _eval ip link set ${WAN} master ${BRIDGE}
    _eval ip addr del dev ${WAN} ${WAN_IP} 
    _eval ip addr add dev ${BRIDGE} ${WAN_IP}
    _eval ip route add default ${GATEWAY}
    _eval ping -c1 ${GATEWAY}
    _eval ping -c1 185.70.41.32
}

_undo() {
    _eval ip addr del dev ${BRIDGE} ${WAN_IP}
    _eval ip addr add dev ${WAN} ${WAN_IP}
    _eval ip link set ${WAN} nomaster
    _eval ip link set ${BRIDGE} down
    _eval ip route add default ${GATEWAY}
    _eval ping -c1 ${GATEWAY}
    _eval ping -c1 185.70.41.32
}

case "${1}" in
    init) _init;;
    clean) _clean;;
    do) do;;
    undo) _undo;;
    restart) systemctl network restart;;
    *) echo "usage ${0}: [init|clean|do|undo|restart]";;
esac

This script is only used to check if bridge configuration works as expected. It will offer different facilities to create and migrate the network configuration to a bridge.

sudo sh script.sh do
# to test:
# sudo sh script.sh do; sleep 15; sudo sh script.sh undo

The main network files configuration are stored in /etc/sysconfig/network-scripts. To configure a bridge, the file ifcfg-br0 should be created

DEVICE=br0
TYPE=Bridge
IPADDr="46.20.249.168"
NETMASK="255.255.255.240"
GATEWAY="46.20.249.161"
ONBOOT=yes
BOOTPROTO=none
NM_CONTROLLED=no
DELAY=0

To connect ethernet interface to the bridge, /etc/sysconfig/network-scripts/ifcg-em1 should be modified too.

DEVICE=em1
TYPE=Ethernet
USERCTL=no
SLAVE=yes
MASTER=br0
BOOTPROTO=none
HWADDR=d4:ae:52:8e:bb:50
NM_CONTROLLED=no

Network configuration can be enabled by using systemctl or by rebooting the server.

sudo systemctl restart network

# Virtual Machine Configuration

All virtual machines will use vagrant and libvirt.

# Libvirt Configuration

Libvirt is an abstraction layer to help creating and managing virtual machines. All packages are directly available from centos or rhel repositories.

# install requirement for vagrant libvirt plugin
sudo yum install libvirt libvirt-devel

To use libvirt as a manager, libvirt must be enabled and started. centos use systemd as main init and the command systemctl must be used.

# ensure libvirt is enabled and started
sudo systemctl enable libvirtd
sudo systemctl start libvirtd

Vagrant runs as default user, to give it access to libvirt, this user must be in the libvirt group.

# ensure user is in libvirt group
sudo usermod ${USER} -aG libvirt

libvirt can also be manually managed with virsh.

# print the virtual machine
sudo virsh list

# print the stats about VMs
sudo virsh domstats

Eventually, all these steps can be added to an ansible playbook:

- name: install libvirt
  become: yes
  package:
    name: libvirt
    state: latest

- name: enable libvirt
  become: yes
  service:
    name: libvirtd
    state: started

- name: add current user in libvirt group
  become: yes
  command: "usermod {{ libvirtd['user'] }} -aG libvirt"
  when: libvirtd['user'] is defined

# Vagrant Configuration

Configure virtual machine with vagrant. In this part, libvirt is used because no generic boxes are available for qemu/kvm. The configuration steps can be found on the official vagrant-libvirt (opens new window) plugins on github.

# install git to manage the ansible repository
sudo yum install git ruby-devel gcc

# create the vagrant repository
mkdir vms
cd vms

# init vagrant by creating a Vagrantfile
vagrant init .

# install vagrant-libvirt plugin
vagrant plugin install vagrant-libvirt

# start the VMs
vagrant up

At least the first step (installing package) can be automated easily with an ansible playbook.

- name: install git, ruby and gcc
  become: yes
  package:
    name: [git, ruby-devel, gcc] 
    state: latest

After the creation of the Vagrantfile present in ${HOME}/vms, we can configure our three virtual machines, v1, v2 and v3. Both VMs are connected to one management network, connected on eth0 interface by default on the guest.

The bridge (br0) previously configured will be connected to each VMs on eth1 interface on the guest, and vnetX on the host. It is possible to use the bridge command to control this part.

bridge link

Here the Vagrantfile used. Please read the comment on each part. The provisioning is made with ansible and this part will be presented later.

Vagrant.configure("2") do |config|

  # v1 virtual machine with 46.20.249.165 ip address
  # the provisioning is made with ansible and it uses
  # a generic centos7 image
  config.vm.define "v1" do |v|
    v.vm.box = "generic/centos7"
    v.vm.provider :libvirt do |lib|
      lib.cpus = 2
    end
    v.vm.network :public_network, 
      :dev => "br0",
      :mode => "bridge",
      :type => "bridge",
      :ip => "46.20.249.165",
      :netmask => "255.255.255.240"
    v.vm.provision "ansible" do |ansible|
      ansible.playbook = "provisioning/site.yml"
    end
  end

  # v2 virtual machine with 46.20.249.166 ip address
  # the provisioning is made with ansible and it uses
  # a generic centos7 image
  config.vm.define "v2" do |v|
    v.vm.box = "generic/centos7"
    v.vm.provider :libvirt do |lib|
      lib.cpus = 2
    end
    v.vm.network :public_network, 
      :dev => "br0",
      :mode => "bridge",
      :type => "bridge",
      :ip => "46.20.249.166",
      :netmask => "255.255.255.240"
    v.vm.provision "ansible" do |ansible|
      ansible.playbook = "provisioning/site.yml"
    end
  end

  # v3 virtual machine is in a private network (10.0.0.0/24)
  # but share the same information than the other VMs
  config.vm.define "v3" do |v|
    v.vm.box = "generic/centos7"
    v.vm.provider :libvirt do |lib|
      lib.cpus = 2
    end
    v.vm.network :private_network,
      :dev => "br1",
      :mode => "bridge",
      :type => "bridge",
      :ip => "10.0.0.2",
      :netmask => "255.255.255.0"
    v.vm.provision "ansible" do |ansible|
      ansible.playbook = "provisioning/site.yml"
    end
  end

end

Vagrant is ready to fetch images and provision the virtual machine. If ansible playbooks and roles are not defined (yet), vagrant provisioning can be executed manually with vagrant provision.

vagrant up

# Ansible Repository

In the vagrant root where Vagrantfile is present, an ansible repository will be made. This tree is based on best practices from official ansible documentation.

# create the tree
mkdir -p provisioning/host_vars
mkdir -p provisioning/roles
mkdir -p provisioning/inventory

# configure local machine (host)
echo "localhost ansible_connection=local" \
  >> provisioning/inventory/hosts

# main playbook used to provisioning virtual machines
touch provisioning/site.yml

# used to initialize the host
touch provisioning/localhost.yml

# dedicated host variables
touch provisioning/host_vars/v1.yml
touch provisioning/host_vars/v2.yml
touch provisioning/host_vars/v3.yml

# this host variable file is dedicated to the host
# can be executed with:
touch provisioning/host_vars/localhost.yaml

Provisionning with ansible on vagrant is done with the provision subcommand.

vagrant provision ${vm}

# or with a defined provisioner
vagrant provision ${vm} --provision-with ansible

It is also possible to provision manually with ansible, directly from the provisioning directory:

cd provisioning
ansible-playbook -i inventory site.yml

# or with vagrant host file
ansible-playbook -i ../.vagrant/provisioners/ansible/inventory/vagrant_ansible_inventory \
  site.yml

Each roles will be made manually. Anyway, some of these roles can be found from galaxy like:

  • https://galaxy.ansible.com/entercloudsuite/haproxy
  • https://galaxy.ansible.com/entercloudsuite/keepalived
  • https://galaxy.ansible.com/geerlingguy/nginx

provisionning/site.yml is the main playbook used to provision each servers.

- hosts:
    - v1
    - v2
  tasks:
    - name: update packages
      become: yes
      yum:
        name: '*'
        state: latest
  roles:
    - haproxy
    - keepalived
    
- hosts: v3
  roles:
    - nginx
    - iptables

# nginx configuration

cd roles

# initialize tree
mkdir nginx
mkdir -p nginx/templates
mkdir -p nginx/tasks
mkdir -p nginx/handlers

# create files
touch nginx/README.md
touch nginx/tasks/main.yml
touch nginx/tasks/nginx.yml
touch nginx/templates/nginx.conf.j2
touch nginx/handlers/main.yml

tasks/main.yml will include nginx.yml only if nginx variable is set.

- name: include nginx
  include: nginx.yml
  when: nginx is defined

tasks/nginx.yml will install and check nginx, ensure each directory are present (from the configuration file), generate a new configuration and check it.

- name: install nginx
  become: yes
  package:
    name: nginx
    state: latest

- name: enable nginx
  become: yes
  service:
    name: nginx
    enabled: yes
    state: started

- name: ensure modules directory is present
  become: yes
  file:
    path: /usr/share/nginx/modules
    state: directory

- name: ensure conf directory is present
  become: yes
  file:
    path: /etc/nginx/conf.d
    state: directory

- name: configure nginx
  become: yes
  template:
    src: nginx.conf.j2
    dest: /tmp/nginx.conf.tmp
    owner: root
    group: root
    mode: '0644'

- name: check nginx configuration
  become: yes
  command: nginx -t -c /tmp/nginx.conf.tmp

- name: nginx configuration is valid
  become: yes
  copy:
    src: /tmp/nginx.conf.tmp
    dest: /etc/nginx/nginx.conf
    owner: root
    group: root
    mode: '0644'
    remote_src: yes
  notify: reload nginx

- name: clean nginx temporary configuration
  become: yes
  file:
    path: /tmp/nginx/conf/tmp
    state: absent

templates/nginx.conf.j2 generate a configuration file based on the default one offered by RHEL package.

user {{ nginx['user'] | default("nginx") }};
worker_processes {{ nginx['worker_processes'] | default("auto") }};
error_log {{ nginx['error_log'] | default("/var/log/nginx/error.log") }};
pid {{ nginx['pid'] | default("/run/nginx.pid") }};
include /usr/share/nginx/modules/*.conf;

events {
  worker_connections {{ nginx['events']['worker_connections'] | default("1024") }};
}

http {
  log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
    '$status $body_bytes_sent "$http_referer" '
    '"$http_user_agent" "$http_x_forwarded_for"';

  access_log  /var/log/nginx/access.log  main;

  sendfile {{ nginx['http']['sendfile'] | default("on") }};
  tcp_nopush {{ nginx['http']['tcp_nopush'] | default("on") }};
  tcp_nodelay {{ nginx['http']['tcp_nodelay'] | default("on") }};
  keepalive_timeout {{ nginx['http']['keepalive_timeout'] | default("65") }};
  types_hash_max_size {{ nginx['http']['types_hash_max_size'] | default("2048") }};

  include             /etc/nginx/mime.types;
  default_type        application/octet-stream;

  include /etc/nginx/conf.d/*.conf;

  server {
    listen       80 default_server;
    listen       [::]:80 default_server;
    server_name  _;
    root         /usr/share/nginx/html;

    # Load configuration files for the default server block.
    include /etc/nginx/default.d/*.conf;

    location / {
    }

    error_page 404 /404.html;
      location = /40x.html {
    }

    error_page 500 502 503 504 /50x.html;
      location = /50x.html {
    }
  }
}

handlers/main.yml add reload support.

- name: reload nginx
  become: yes
  service:
    name: nginx
    state: reloaded

NOTE: the website/application will be deployed manually.

# haproxy configuration

haproxy configuration will be used by a flexible pattern. This role (like the other I will create) will be enable only if the variable haproxy is set. This pattern is really interesting, because you can easily export the configuration of a server or group of server in yaml format to another tool (like salt or chef). The idea is to create a high level abstraction with defined data structure, in this case an object or dictionary in Python. Here a configuration example for the v1 server haproxy. This file is generaly stored in host_vars/v1.yml.

haproxy:
  frontends: 
    - name: test
      listen: "*:80"
      use_backend: t3
      default_backend: t3
  backends: 
    - name: t3
      balance: roundrobin
      servers:
        - t3 46.20.249.168:80 check

Before creating the role from scratch, the directory and file tree need to be initialized. Again, this is from ansible best practices pages.

cd roles

# initialize tree
mkdir haproxy
mkdir -p haproxy/templates
mkdir -p haproxy/tasks
mkdir -p haproxy/handlers

# create files
touch haproxy/README.md
touch haproxy/tasks/main.yml
touch haproxy/tasks/haproxy.yml
touch haproxy/templates/haproxy.cfg.j2
touch haproxy/handlers/main.yml

README.md will contain a small documentation about the role.

tasks/main.yml will contain the "bootstrap", a small piece of code used to include the true main file based on haproxy variable.

- name: haproxy enabled
  include: haproxy.yml
  when: haproxy is defined

tasks/haproxy.yml will contain the main code, install haproxy, enable and start it. Configure the configuration based on the data structure, check the configuration and restart the service if this last one was changed.

- name: install haproxy
  become: yes
  package:
    name: haproxy
    state: present

- name: enable haproxy
  become: yes
  service:
    name: haproxy
    state: started
    enabled: yes

- name: configure haproxy
  become: yes
  template:
    src: haproxy.cfg.j2
    dest: /tmp/haproxy.cfg.tmp
    owner: root
    group: root
    mode: '0644'

- name: check haproxy configuration
  become: yes
  command: haproxy -c -f /tmp/haproxy.cfg.tmp

- name: haproxy configuration is valid, use it
  become: yes
  copy:
    src: /tmp/haproxy.cfg.tmp
    dest: /etc/haproxy/haproxy.cfg
    owner: root
    group: root
    mode: '0644'
    remote_src: yes
  notify: restart haproxy
    
- name: clean temporary haproxy configuration
  become: yes
  file:
    path: /tmp/haproxy.cfg.tmp
    state: absent

templates/haproxy.cfg.j2 is a jinja template containing the main haproxy configuration file stored in /etc/haproxy/haproxy.cfg. This template is configured with default working values. Each values can be set from the haproxy variable present in the ansible hosts variable namespace.

global
  chroot {{ haproxy['global']['chroot'] | default("/var/lib/haproxy") }}
  daemon
  group {{ haproxy['global']['group'] | default("haproxy") }}
  log {{ haproxy['global']['log'] | default("127.0.0.1 local2") }}
  maxconn {{ haproxy['global']['maxconn'] | default("4000") }} 
  pidfile {{ haproxy['global']['pidfile'] | default("/var/run/haproxy.pid") }}
  stats socket /var/lib/haproxy/stats
  user {{ haproxy['global']['user'] | default("haproxy") }}

defaults
  log {{ haproxy['defaults']['log'] | default("global") }}
  maxconn {{ haproxy['defaults']['maxconn'] | default("3000") }}
  mode {{ haproxy['defaults']['mode'] | default("http") }}
  {% if haproxy['defaults']['option'] is defined %}
  {% else %}
  option dontlognull
  option forwardfor except 127.0.0.0/8
  option httplog
  option http-server-close
  option redispatch
  {% endif %}
  retries {{ haproxy['defaults']['retries'] | default("3") }}
  {% if haproxy['defaults']['timeout'] is defined %}
  {% else %}
  timeout check 10s
  timeout client 1m
  timeout connect 10s
  timeout http-keep-alive 10s
  timeout http-request 10s
  timeout queue 1m
  timeout server 1m
  {% endif %}

{% if haproxy['frontends'] is defined %}
{% for frontend in haproxy['frontends'] %}
frontend {{ frontend['name'] }} {{ frontend['listen'] | default("*:5000") }}
  {% if frontend['acls'] is defined %}
    {% for acl in frontend['acls'] %}
  acl {{ acl }}
    {% endfor %}
  {% endif %}
  {% if frontend['use_backend'] is defined %}
  use_backend {{ frontend['use_backend'] }}
  {% endif %}
  {% if frontend['default_backend'] is defined %}
  default_backend {{ frontend['default_backend'] }}
  {% endif %}
{% endfor %}
{% endif %}

{% if haproxy['backends'] is defined %}
{% for backend in haproxy['backends'] %}
backend {{ backend['name'] }}
  balance {{ backend['balance'] | default('roundrobin') }}
    {% if backend['servers'] is defined %}
      {% for server in backend['servers'] %}
  server {{ server }}
      {% endfor %}
    {% endif %}
{% endfor %}
{% endif %}

handlers/main.yml contain all actions like, stop, start and restart (or other events) to control haproxy service.

- name: restart haproxy
  become: yes
  service:
    name: haproxy
    state: restarted

With this role, a user can deploy an haproxy server only by editing the configuration from the data-structure stored in ansible.

# keepalived configuration

cd roles

# initialize tree
mkdir keepalived
mkdir -p keepalived/templates
mkdir -p keepalived/tasks
mkdir -p keepalived/handlers

# create files
touch keepalived/README.md
touch keepalived/tasks/main.yml
touch keepalived/tasks/keepalived.yml
touch keepalived/templates/keepalived.conf.j2
touch keepalived/handlers/main.yml

README.md is a documentation file.

tasks/main.yml is like the one create for haproxy, used only to check if the keepalived variable is set correctly?

- name: include keepalived
  include: keepalived.yml
  when: keepalived is defined

tasks/keepalived.yml contains the main ansible code. It will install, enable and configure keepalived. A configuration will be generated based on the data structure and tested. If this configuration is working, it will be placed on the default keepalived configuration path in /etc/keepalived/keepalived.conf.

- name: install keepalived
  become: yes
  package:
    name: keepalived
    state: installed

- name: enable keepalived
  become: yes
  service:
    name: keepalived
    state: started
    enabled: yes

- name: configure keepalived
  become: yes
  template:
    src: keepalived.conf.j2
    dest: /tmp/keepalived.conf.tmp
    owner: root
    group: root
    mode: '0644'

- name: check keepalived configuration
  become: yes
  command: keepalived --check /tmp/keepalived.conf.tmp

- name: install keepalived configuration
  become: yes
  copy:
    remote_src: yes
    src: /tmp/keepalived.conf.tmp
    dest: /etc/keepalived/keepalived.conf
    owner: root
    group: root
    mode: '0644'
  notify: restart keepalived

- name: clean temporary keepalived configuration
  become: yes
  file:
    path: /tmp/keepalived.conf.tmp
    state: absent

templates/keepalived.conf.j2 is the template to generate the keepalived configuration file. This is a small one supported only the required features.

global_defs {
   notification_email { 
     {{ keepalived['global']['notification_email'] | default("root") }} 
   }
   notification_email_from {{ keepalived['global']['notification_email_from'] | default("root") }}
   smtp_server {{ keepalived['global']['smtp_server'] | default("localhost") }}
   smtp_connect_timeout {{ keepalived['global']['smtp_connection_timeout'] | default("30") }}
}

{% if keepalived['instances'] is defined %}
{% for instance in keepalived['instances'] %}

vrrp_instance {{ instance['name'] }} {
  state {{ instance['state'] | default("MASTER") }}
  interface {{ instance['interface'] }}
  virtual_router_id {{ instance['virtual_router_id'] }}
  priority {{ instance['priority'] | default("1") }}
  advert_int {{ instance['advert_int'] | default("1") }}
{% if instance['authentication'] is defined %}
  authentication {
    auth_type {{ instance['authentication']['type'] | default("PASS") }}
    auth_pass {{ instance['authentication']['pass'] }}
  }
{% endif %}
{% if instance['virtual_ipaddress'] is defined %}
  virtual_ipaddress {
{% for ip in instance['virtual_ipaddress'] %}
    {{ ip }}
{% endfor %}
  }
{% endif %}
}

{% endfor %}
{% endif %}

handlers/main.yml was made to react to different event from the other part of the code. Only the restart facility was made.

- name: restart keepalived
  become: yes
  service:
    name: keepalived
    state: restarted

# iptables configuration

cd roles

# initialize tree
mkdir iptables
mkdir -p iptabes/tasks

# create files
touch iptables/README.md
touch iptables/tasks/main.yml

README.md is a documentation file.

tasks/main.yml will contain the main iptables configuration. As usual, it will be set based on the data-structure.

- name: disable firewalld
  become: yes
  service:
    name: firewalld
    state: enabled
    state: started

- name: install iptables-services
  become: yes
  package:
    name: iptables-services
    state: latest

- name: enable iptables service
  become: yes
  service:
    name: iptables
    state: started

- name: allow access to http port
  become: yes
  iptables:
    chain: INPUT
    protocol: tcp
    destination_port: 80
    jump: ACCEPT
  when: iptables['http'] is defined

- name: allow access to https port
  become: yes
  iptables:
    chain: INPUT
    protocol: tcp
    destination_port: 443
    jump: ACCEPT
  when: iptables['https'] is defined

- name: backup iptables rules
  become: yes
  command: service iptables save

WARNING: it seems firewalld is installed and enabled by default on generic/centos7 image. This can lead to some trouble with iptables compatibility.

# Libvirt role

mkdir -p provisioning/roles/libvirt/tasks
mkdir -p provisioning/roles/libvirt/handlers
touch  provisioning/roles/libvirt/tasks/main.yml
touch provisioning/roles/libvirt/tasks/libvirt.yml
touch provisioning/roles/libvirt/handlers/main.yml

tasks/main.yml:

- name: include libvirt configuration
  include: libvirt.yml
  when: libvirt is defined

tasks/libvirt.yml:

- name: install libvirt
  become: yes
  package:
    name: libvirt
    state: latest

- name: enable libvirtd
  become: yes
  service:
    name: libvirtd
    state: started
    enabled: yes

handlers/main.yml:

- name: reload libvirtd
  become: yes
  service:
    name: libvirtd
    state: reloaded

# Firewalld Role

I create a firewalld role. I was not aware that firewalld was present in centos by default now. I am old school and prefere iptables, ip6tables or even nftables. Here the code to fallback to firewalld.

firewalld/tasks/main.yml

- name: include firewalld
  include: firewalld.yml
  when: firewalld is defined

firewalld/tasks/firewalld.yml

- name: disable iptables
  service:
    name: iptables
    state: stopped
    enabled: no

- name: enable firewalld
  service:
    name: firewalld
    state: started
    enabled: yes

- name: enable https access
  become: yes
  firewalld:
    service: https
    permanent: yes
    state: enabled

- name: enable http access
  become: yes
  firewalld:
    service: http
    permanent: yes
    state: enabled

# Using Ansible Provisioning

Both virtual machines are configured through ansible host variables stored in provisioning/host_vars and called by the name of the server.

touch provisioning/host_vars/localhost.yml
touch provisioning/host_vars/v1.yml
touch provisioning/host_vars/v2.yml

# localhost configuration

- hosts: localhost
  connection: local
  roles:
    - nginx
    - libvirt

# v1 Configuration

This configuration file is located in provisioning/host_vars/v1.yml and contains the ansible host variable for v1 virtual machine. The first part define the required network configuration.

network:
  eth1:
    ip: 46.20.249.165
    netmask: 255.255.255.240

The second part defines HAproxy configuration. A frontend called test and listening to all interface on http port is created. A backend called t3 is created and forward the flow to 46.20.249.168 on port 80.

haproxy:
  frontends: 
    - name: test
      listen: "*:80"
      use_backend: t3
      default_backend: t3
  backends: 
    - name: t3
      balance: roundrobin
      servers:
        - t3 46.20.249.168:80 check

The third part of the configuration is about keepalived. One instance is created on eth1 interface. VRRP information like router id is defined. track_interface and unicast_peer are defined explicitely, due to multicast or broadcast issue. An authentication method is also defined. Finaly, 46.20.249.167/28 is shared between the two VMs.

keepalived:
  instances:
    - name: test
      interface: eth1
      virtual_router_id: 100
      priority: 100
      track_interface:
        - eth1
      unicast_peer:
        - 46.20.249.166
      authentication:
        type: PASS
        pass: 1234
      virtual_ipaddress:
        - 46.20.249.167/28

Because HAProxy is listening on port 80 (and probably one day on port 443), iptables should allow the flow.

iptables:
  http: yes
  https: yes

# v2 Configuration

Same as v1 configuration.

network:
  eth1:
    ip: 46.20.249.165
    netmask: 255.255.255.240

haproxy:
  frontends: 
    - name: test
      listen: "*:80"
      use_backend: t3
      default_backend: t3
  backends: 
    - name: t3
      balance: roundrobin
      servers:
        - t3 46.20.249.168:80 check

keepalived:
  instances:
    - name: test
      interface: eth1
      virtual_router_id: 100
      priority: 100
      track_interface:
        - eth1
      unicast_peer:
        - 46.20.249.166
      authentication:
        type: PASS
        pass: 1234
      virtual_ipaddress:
        - 46.20.249.167/28

iptables:
  http: yes
  https: yes

# v3 Configuration

v3 configuration is pretty simple, we need to enable nginx and allow http flow

nginx:

iptables:
  http: yes
  https: yes

Because it is a server in private network, we need to create a NAT rule. The flow to 46.20.249.168:80 and 46.20.249.168:443 will be forwarded to 10.0.0.2:80 and 10.0.0.2:443 respectivily. This configuration can be done with iptables (on the host). Unfortunately, due to libvirt iptables configuration, I am a bit confuse... Too many rules are present and I don't really know how to alter them without breaking the service. Usually, the rule is pretty simple:

iptables -I FORWARD -d 46.20.249.168 -p tcp -dport 80 -d 10.0.0.2
iptables -t nat -I PREROUTING -p tcp --dport 80 -d 46.20.249.168 \
  -j DNAT --to 10.0.0.2:80

Due this strange behaviour, I decided to find an alternative and use nginx as a forwarder with proxy_pass feature. Here the modification on the /etc/nginx/nginx.conf file on the host.

--- nginx.conf.old      2020-04-25 10:43:44.244383741 +0000
+++ nginx.conf  2020-04-25 10:42:36.360137288 +0000
@@ -36,6 +36,7 @@
     include /etc/nginx/default.d/*.conf;

     location / {
+      proxy_pass http://10.0.0.2;
     }

     error_page 404 /404.html;

To enable the configuration, just reload nginx.

sudo systemctl reload nginx

Maybe it is a better choice than using iptables, nginx can be easily configured dynamically with lua or other script language.

# Testing

Many different testing. Those steps should be automated with a monitoring tool.

# Network and service test

From my laptop to main server

ping -c1 46.20.249.168
# output:
# PING 46.20.249.168 (46.20.249.168): 56 data bytes
# 64 bytes from 46.20.249.168: icmp_seq=0 ttl=45 time=35.297 ms
# 
# --- 46.20.249.168 ping statistics ---
# 1 packets transmitted, 1 packets received, 0.0% packet loss
# round-trip min/avg/max/std-dev = 35.297/35.297/35.297/0.000 ms

curl 46.20.249.168
# output:
# Hello, world!

From my laptop to v1

ping -c1 46.20.249.165
# output:
# PING 46.20.249.165 (46.20.249.165): 56 data bytes
# 64 bytes from 46.20.249.165: icmp_seq=0 ttl=45 time=35.419 ms
#
# --- 46.20.249.165 ping statistics ---
# 1 packets transmitted, 1 packets received, 0.0% packet loss
# round-trip min/avg/max/std-dev = 35.419/35.419/35.419/0.000 ms

curl 46.20.249.165
# output:
# Hello, world!

From my laptop to v2

ping -c1 46.20.249.166
# output:
# PING 46.20.249.166 (46.20.249.166): 56 data bytes
# 64 bytes from 46.20.249.166: icmp_seq=0 ttl=45 time=35.253 ms
#
# --- 46.20.249.166 ping statistics ---
# 1 packets transmitted, 1 packets received, 0.0% packet loss
# round-trip min/avg/max/std-dev = 35.253/35.253/35.253/0.000 ms

curl 46.20.249.166
# output:
# Hello, world!

From my laptop to vip

ping -c1 46.20.249.167
# output:
# PING 46.20.249.167 (46.20.249.167): 56 data bytes
# 64 bytes from 46.20.249.167: icmp_seq=0 ttl=45 time=44.936 ms
#
# --- 46.20.249.167 ping statistics ---
# 1 packets transmitted, 1 packets received, 0.0% packet loss
# round-trip min/avg/max/std-dev = 44.936/44.936/44.936/0.000 ms

curl 46.20.249.167
# output:
# Hello, world!

# VIP testing

Checking IP configuration on v1 VM:

ip a s dev eth1

3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:20:6b:d4 brd ff:ff:ff:ff:ff:ff
    inet 46.20.249.165/28 brd 46.20.249.175 scope global noprefixroute eth1
        valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe20:6bd4/64 scope link
        valid_lft forever preferred_lft forever

Checking IP configuration on v2 VM:

ip a s dev eth1

3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:79:ae:c2 brd ff:ff:ff:ff:ff:ff
    inet 46.20.249.166/28 brd 46.20.249.175 scope global noprefixroute eth1
        valid_lft forever preferred_lft forever
    inet 46.20.249.167/28 scope global secondary eth1
        valid_lft forever preferred_lft forever

VIP is defined on v2. To test if the VIP will move to v1, we will suspend v2 server with vagrant suspend v2 on the host. After this action, we got this output on v1 server:

ip a s eth1

3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:20:6b:d4 brd ff:ff:ff:ff:ff:ff
    inet 46.20.249.165/28 brd 46.20.249.175 scope global noprefixroute eth1
        valid_lft forever preferred_lft forever
    inet 46.20.249.167/28 scope global secondary eth1
        valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe20:6bd4/64 scope link
        valid_lft forever preferred_lft forever

and the service is still available.

curl 46.20.249.167
# output:
# Hello, world!

# Troubleshooting // FAQ

# I lost the connection when I start the bridge

You can use iDrac and get access to the remote console. Or, just modify the configuration without saving your change (like me) if you can't use iDrac.

# I got "no route to host" message on VM

This is probably due to iptables configuration. By default, RHEL systems are using iptables rules. If you want to connect to a defined port you need to add it explicitely in the chain IN_public_allow on the host or other VMs.

iptables -A IN_public_allow -j ACCEPT -p tcp --destination-port 80

#

# Conclusion

Last time I configured a VRRP service on Linux was at Alcatel-Lucent, 10 years ago. It was really nice to recreate from scratch all the configuration.

Anyway, why I decided to create all role by myself? Because roles from ansible galaxy are not trustable for different reasons. The first one is due to the lack of announce in the community, it makes change harder. The second one is mainly because we need to control all the stack, to make that possible, we need to defined standard and convention. Ansible scripts are made by different people for different needs and are sometime not compatible with other one.

The pattern used here was used in different companies, like Afrinet, OVH and Leroy Merlin. I use it on my own project and gives the possibility to switch to another solution easily. For exemple, at OVH, this pattern was created to manage Salt, Ansible and Puppet script.

# Resources

  • https://www.centos.org/forums/viewtopic.php?t=53972

# Bridge Configuration

  • http://www.itzgeek.com/how-tos/mini-howtos/create-a-network-bridge-on-centos-7-rhel-7.html
  • https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Networking_Guide/sec-Network_Bridging_Using_the_Command_Line_Interface.html
  • https://wiki.linuxfoundation.org/networking/bridge
  • https://wiki.libvirt.org/page/Networking#Fedora.2FRHEL_Bridging
  • https://wiki.archlinux.org/index.php/Network_bridge

# Secondary IP

  • https://ma.ttias.be/how-to-add-secondary-ip-alias-on-network-interface-in-rhel-centos-7/
  • https://community.spiceworks.com/topic/545859-add-secondary-ip-to-one-interface-in-centos-7
  • https://www.ubiquityhosting.com/blog/configure-ip-ranges-on-centos-7-redhat-7/

# General Information

  • http://www.linux-kvm.org/page/Networking
  • https://wiki.libvirt.org/page/Networking#Bridged_networking_.28aka_.22shared_physical_device.22.29
  • https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Virtualization_Deployment_and_Administration_Guide/sect-bridge-mode.html

# Setup routing with iptables (host routes and guest gateways) as described here:

  • http://www.linux-kvm.org/page/Networking#Routing_with_iptables

# Vagrant

  • https://www.vagrantup.com
  • https://www.hashicorp.com/security.html
  • https://www.vagrantup.com/docs/provisioning/ansible_local.html
  • https://app.vagrantup.com/generic/boxes/centos7
  • https://github.com/vagrant-libvirt/vagrant-libvirt
  • https://www.vagrantup.com/docs/synced-folders/basic_usage.html

# Libvirt

  • https://libvirt.org/
  • https://github.com/vagrant-libvirt/vagrant-libvirt

# Ansible

  • https://docs.ansible.com/ansible/latest/user_guide/playbooks_best_practices.html
  • https://jinja.palletsprojects.com/en/2.11.x/templates/
  • https://docs.ansible.com/ansible/latest/user_guide/playbooks_conditionals.html
  • https://docs.ansible.com/ansible/latest/modules/copy_module.html
  • https://docs.ansible.com/ansible/latest/modules/service_module.html
  • https://docs.ansible.com/ansible/latest/modules/template_module.html
  • https://docs.ansible.com/ansible/latest/modules/package_module.html
  • https://docs.ansible.com/ansible/latest/user_guide/playbooks_reuse_includes.html#includes-vs-imports
  • https://docs.ansible.com/ansible/latest/scenario_guides/guide_vagrant.html

# Iptables

  • https://linux.die.net/man/8/iptables
  • https://docs.ansible.com/ansible/latest/modules/iptables_module.html
  • https://wiki.libvirt.org/page/Networking#Forwarding_Incoming_Connections