The VXLAN (Virtual eXtensible Local Area Networking) protocol is a tunnelling protocol designed to solve the problem of limited VLAN IDs (4096) in IEEE 802.1q. With VXLAN the size of the identifier is expanded to 24 bits (16777216).
VXLAN is described by IETF RFC 7348, and has been implemented by a number of vendors. The protocol runs over UDP using a single destination port. Here we will use the VXLAN linux tunnel device as an overlay network for connections between VMs on different nodes.
Preparing VXLAN Tunnel
Before we can get started configuring things we’ll need to install libvirt, bridge-utils and another package that will help us.
apt update && apt upgrade sudo apt install -y qemu-kvm virtinst libvirt-daemon-system libvirt-clients bridge-utils libguestfs-tools
We need to start the
systemctl enable --now libvirtd
Create bridge and VXLAN Interface
To enable the TCP/IP traffic to be encapsulated through these interfaces, we will create a bridge and attach the VXLAN interface to that bridge. In the end, a bridge works like a network hub and forwards the traffic to the ports that are connected to it. So the traffic that appears in the bridge will be encapsulated into the UDP multicast messages.
For the creation of the first VXLAN (with VNI 10) we will need to issue the next commands (in each of the nodes)
ip link add br-vxlan10 type bridge ip link add vxlan10 type vxlan id 10 group 18.104.22.168 dstport 0 dev ens3 ip link set vxlan10 master br-vxlan10 ip link set vxlan10 up ip link set br-vxlan10 up
Create virtual network
To enable virtual machines (VM) to use the
br-vxlan10 bridge with the attached virtual extensible LAN (VXLAN), first add a virtual network to the
libvirtd service that uses this bridge.
cat << EOF > ~/vxlan10-bridge.xml <network> <name>net-vxlan10</name> <forward mode="bridge" /> <bridge name="br-vxlan10" /> </network> EOF virsh net-define ~/vxlan10-bridge.xml virsh net-start net-vxlan10 virsh net-autostart net-vxlan10
Prepare to Guest VM
Configure storage pool
libvirt is first installed it doesn’t have any configured storage pools. Let’s create one in the default location,
virsh pool-define-as default --type dir --target /var/lib/libvirt/images
We need to mark the pool active, and we might as well configure it to activate automatically next time the system boots:
virsh pool-start default virsh pool-autostart default
Download base image
We’ll need a base image for our virtual machines. I’m going to use the ubuntu focal image, which we can download to our storage directory like this:
curl -L -o /var/lib/libvirt/images/ubuntu-focal.img \ https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img
We need to make sure
libvirt is aware of the new image:
virsh pool-refresh default
Lastly, we’ll want to set a root password on the image so that we can log in to our virtual machines:
virt-customize -a /var/lib/libvirt/images/ubuntu-focal.img \ --root-password password:secret
Create the virtual machine
We’re going to create a pair of virtual machines (one on each host). We’ll be creating each vm with
net-vxlan10 network attached. To create a virtual machine on
vm1, run the following command:
virt-install \ -r 3000 \ --network network:net-vxlan10 \ --os-variant ubuntu20.04 \ --disk pool=default,size=10,backing_store=ubuntu-focal.img,backing_format=qcow2 \ --import \ --noautoconsole \ -n vm1
The most interesting option in the above command line is probably the one used to create the virtual disk:
--disk pool=default,size=10,backing_store=ubuntu-focal.img,backing_format=qcow2 \
This creates a 10GB “copy-on-write” disk that uses
ubuntu-focal.img as a backing store. That means that reads will generally come from the
ubuntu-focal.img image, but writes will be stored in the new image. This makes it easy for us to quickly create multiple virtual machines from the same base image.
bm-2 we would run a similar command, although here we’re naming the virtual machine
virt-install \ -r 3000 \ --network network:net-vxlan10 \ --os-variant ubuntu20.04 \ --disk pool=default,size=10,backing_store=ubuntu-focal.img,backing_format=qcow2 \ --import \ --noautoconsole \ -n vm3
Accessing the vm
We can access the vm that we have created with the
virsh console like this:
virsh console vm1
For vm on bm-2 you can access it in the same way:
virsh console vm3
Now you can login with the root user and password that we have set before.
Adding an ip address
We need to add ip addresses to both vm with ip commands like the following:
# vm1 hostnamectl set-hostname vm1 ip addr add 10.10.10.11/24 dev enp1s0 ip link set enp1s0 up # vm3 hostnamectl set-hostname vm3 ip addr add 10.10.10.12/24 dev enp1s0 ip link set enp1s0 up
We can test the connection between VM with ping:
ping -c 10.10.10.11 ping -c 10.10.10.12
DHCP and Gateway
At this point, your VM can communicate with other VMs on different hosts, but cannot communicate outside or to the internet. Then we need a router as a gateway to the internet. You can make one of the nodes as a router or gateway and also as a DHCP server. On one of the nodes add IP to
br-vxlan10, here I will make
bm-1 as a gateway and DHCP server.
ip addr add 10.10.10.1/24 dev br-vxlan10
apt install -y dnsmasq
Configure dnsmasq DHCP server for a specific subnet
vi /etc/dnsmasq.d/dhcp dhcp-range=subnet10,10.10.10.5,10.10.10.250,255.255.255.0,8h dhcp-option=subnet10,3,10.10.10.1 dhcp-option=subnet10,option:dns-server,22.214.171.124
systemctl restart dnsmasq
Add nat rule to allow hosts in a network with an IP range of 10.10.10.0/24 to access the internet and translate their source IP address into the system's public IP address when packets leave the system.
iptables -t nat -A POSTROUTING -s 10.10.10.0/24 -j MASQUERADE
Login to vm and use
dhclient to get IP from DHCP server:
virsh console vm1 dhclient -v enp1s0
We should now have the IP of the DHCP server. Use the ip command to verify and try to reach the internet.
ip a ping -c3 126.96.36.199
DHCP and Gateway: Easy Way
libvirt already provides a feature to create a DHCP server,
libvirt also uses
dnsmasq for its DHCP server. This way we don't need to configure
dnsmasq ourselves. First of all, we have to let libvirt manage our bridge:
ip link del br-vxlan10
Then change the libvirt network
net-vxlan10 configuration to the following:
virsh net-edit net-vxlan10 <network> <name>net-vxlan10</name> <uuid>4a84aabc-b5cb-459d-9aab-48bcfa85c300</uuid> <forward mode='route'/> <bridge name='br-vxlan10' stp='on' delay='0'/> <ip address='10.10.10.1' netmask='255.255.255.0'> <dhcp> <range start='10.10.10.10' end='10.10.10.254'/> </dhcp> </ip> </network>
After making the change, we need to restart the
virsh net-start net-vxlan10
Since we previously deleted br-vxlan10 making the vxlan10 tunnel interface independent of br-vxlan10, we need to set master vxlan10 back to br-vxlan10:
ip link set vxlan10 master br-vxlan10 ip link set vxlan10 up
Log in to the vm then renew ip with
dhclient -r enp1s0
Use the ip command to verify and try to reach the internet:
ip a ping -c 188.8.131.52
By utilizing the VXLAN tunnel interface in Linux, we can establish an overlay network that connects KVM guest virtual machines running on different hosts. VXLAN enables these virtual machines to communicate over a separate virtual network, creating secure and flexible connectivity between the respective hosts. This is particularly valuable in virtualized environments, facilitating communication among virtual machines hosted on separate systems.