Michael Wadman

My thoughts and experiences with all things network

OpenStack Lab Network – Vagrant


This post is the second in a series that plans to document my progress through installing and configuring a OpenStack Lab, using Cumulus switches for the network.

For other posts in this series, see the Overview section in the introduction post.

This post will cover the creation of our Virtual Machines in VirtualBox using Vagrant.
Vagrant (by HashiCorp) can be described as a tool to spin up reproducible development environments using most mainstream hypervisors.


Installation of Vagrant on Ubuntu (my choice of desktop distribution) requires downloading the deb file from the official website, or (as I recently found out) adding a community repository which is a kind of proxy-repository as it just redirects to the packages from HashiCorp themselves.
For instructions on how to install Vagrant using this repo, visit the website here.

There is a Vagrant package that is available in the default repositories, but this isn’t kept up to date.


An additional step I’ll take is to copy the bash completion script for Vagrant over so that it can be used (I don’t know why this isn’t done by default, seeing as the file is included in the package anyway – see Github issue #8987).

Remember to replace the version directories with your version of Vagrant.
You’ll also need to start a new shell in order for this to take effect.

Usually with Vagrant the next step would be to install a plugin to support the hypervisor (named a Vagrant ‘provider’) you are using, but VirtualBox is supported by default.



Now we need to add a ‘Box’, which is Vagrant’s term for an image.
The difference between Vagrant boxes and most other operating system images is that boxes are pre-made and configured, from which new virtual machines are cloned (as opposed to being installed by the user).

For example, there is a Cumulus VX Vagrant box available that we’ll use in this project – named “CumulusCommunity/cumulus-vx”. We’ll add this now along with the Ubuntu 16.04 box.

During this step, if a box has more versions for more than just one provider it will prompt to ask which version to download.

These boxes are stored globally for the user in the directory “~/.vagrant.d/boxes” and the currently installed boxes can be shown using the command below



Now that we’ve got our host ready, we’ll create a ‘Vagrantfile’:

“The primary function of the Vagrantfile is to describe the type of machine required for a project, and how to configure and provision these machines” – https://www.vagrantup.com/docs/vagrantfile/

More simply put, the file marks the root directory of a project in Vagrant and some configuration associated with it. This is created using the “vagrant init” command in the directory that you want to use as your project root:


By default this file is pretty empty, with only three lines uncommented.

The “do” and “end” lines form a ‘block’ in Ruby, in which you define items for the function (the “Vagrant.configure” part) to invoke.
With the Vagrantfile, all we’re doing is defining configuration. Nothing fancy here!
Let’s change some then.

According to the Getting Started documentation, the first thing we should change is the base box to whichever of the boxes we’ve downloaded we want to use.
However because our project isn’t quite as simple as using only one VM we need to define multiple, and this is done by adding “config.vm.define” blocks inside the main configure block.
We’ll do one block for each of the VM’s we need – 2 spine switches, 4 leaf switches, 2 Ubuntu OpenStack machines and another Ubuntu VM for ZTP:

With this configured, we could go ahead and provision then boot all of the machines we’ve specified using the “vagrant up” command, and then the “vagrant destroy” to delete the VM’s.

However, there is still more configuration to do before we can be happy that our lab can be created correctly.

VirtualBox configuration

VirtualBox specific configuration can be defined both globally individually for each VM using another block – “.vm.provider”.

For example, we can set what each VM is named in VirtualBox:

We can also specify how many CPU’s and how much RAM we allocate to the VM:


Most other VirtualBox specific configuration needs to be done through the use of VBoxManage “modifyvm” commands and a “.customize” entry.
In the below, we’re adding all VM’s to a group in VirtualBox so that we can easily tell that it is a part of the OpenStack Lab.

Note that this is done at the “config” level, outside of each host.
This is an example of the global configuration that I referenced earlier.

In VirtualBox, we now have a group folder for our lab machines:

This would usually look like the following if you we’re using the VBoxManage command line utility:


We’ll apply the configuration completed above to the rest of our machines and we’re done with VirtualBox configuration:



Vagrant networking is as simple (or as hard) as the provider (hypervisor) you are using makes it.

There is one limitation in our scenario, in that VirtualBox can only support a maximum of 36 interfaces per machine
This doesn’t bother us because the maximum amount of interfaces one of our machines has is 4 (Each leaf switch has 1 management, 2 uplinks to the spine switches and 1 downlink interface to the attached OpenStack host).


One other thing to be aware of is that Vagrant sets the first network interface as a NAT interface by default.
A NAT interface in VirtualBox allows all communication outbound (through NAT of the hosts primary interface), but only allows connections inbound (including from other VM’s) if they are port-forwarded.
While we could leave this as the default and configure port-forwards for each inbound connection, this becomes cumbersome when you start to think about  DHCP traffic (to/from the ZTP server) and OpenStack control traffic.

Instead we can change the first network interface for all hosts to be part of a “public network” (a bridged network in VirtualBox).

This comes with the caveat that Vagrant can no longer configure the network interface of the VM.
This instead needs to be completed by one of the following:

  • Use a Vagrant box that comes with a preconfigured network interface
  • Configure the interface manually by logging in through the console
  • Use a DHCP to assign network addressing

We’ll use DHCP in this lab as we’re going to be configuring one for ZTP anyway.

Because Vagrant no longer configures the interfaces itself, we also need to tell it what IP address to connect to in order to finish its’ provisioning.
This is done with the “.ssh.host” option under each host.


The last thing we need to do is configure the network interfaces for the links between the VM’s.
This will be done using “private networks” (internal networks in VirtualBox).

For example, to configure the link between the first spine and leaf switches, we need to add the below to each the config of each VM:


After we add the above to all of or devices, our configuration should look like the below:


With this configuration file we have configured Vagrant to provision all of our VM’s with all of the correct interface-to-adapter mappings.

While we could run “vagrant up” to ask Vagrant to provision and turn on the VM’s, Vagrant won’t finish as it will not be able to reach the hosts.
Before it can do so, we need to configure a DHCP server to hand out addresses, which I’ll cover in the next post.



Vagrant – Getting Started
Vagrant – Official Documentation
VirtualBox Manual – Chapter 6: Virtual Networking
VirtualBox Manual – Chapter 8: VBoxManage
Cumulus VX Documentation – Getting Started on VirtualBox
Cumulus VX Documentation – Vagrant and VirtualBox

Versions used:

Desktop Machine: kubuntu-16.04.3-desktop-amd64
VirtualBox: virtualbox-5.2
Vagrant: 2.0.3
Cumulus VX Vagrant Box: CumulusCommunity/cumulus-vx (virtualbox, 3.6.1)
Ubuntu Server Vagrant Box: ubuntu/xenial64 (virtualbox, 20180615.0.0)

OpenStack Lab Network – Introduction


This post is the first in a series that plans to document my progress through installing and configuring a OpenStack Lab, focusing primarily on the configuration of the network.

The series will roughly look like the following:

  1. Setup VirtualBox and virtual machines using Vagrant.
  2. Install and configure ZTP server and boot switches.
  3. Configure network underlay (eBGP) using Ansible
  4. Configure network overlay (VXLAN with EVPN) using Ansible.
  5. Initial deployment of OpenStack servers using OpenStack-Ansible
  6. Configuration of OpenStack.
  7. Integration of OpenStack Neutron with our Cumulus switches.

If you’re reading this sentence, the above is subject to change as I haven’t yet written all posts yet.


I’ve recently taken an interest in a few different topics.
These are, in no particular order:

  • OpenStack: A cloud computing software platform. Neutron, the networking project, is especially interesting.
  • Cumulus Linux: A network operating system that you can load onto ‘whitebox’ switches.
  • Vagrant: Reproducible Virtual Machine deployment as code.
  • ZTP (Zero Touch Provisioning): Used to provision a network device in the same way that a server can be, using DHCP.
  • VXLAN (Virtual Extensible LAN): A layer 2 overlay/tunnelling network protocol used primarily in data centres.
  • EVPN (Ethernet VPN): A MP-BGP address family used to route layer 2 addresses, commonly used in conjunction with VXLAN for routable layer 2 data centre fabrics.

As I’m a masochist a person who likes to learn as much as possible when I’m not procrastinating, I thought it best to build a lab network that would use all of the above technologies and document my progress.


I’ve drawn some quick diagrams that I’ll be using when setting up the lab.

The management network diagram:

The first network interface (as adapter 1 is Cumulus’ management interface – “eth0”) on all of the switches will be connected to a bridged adapter on my host machine (which also has access out to the internet).
I’ll also be creating three Ubuntu server 16.04 VM’s – one ZTP server and two to build the OpenStack ‘cloud’. Each of the server VM’s will have its’ first network interface connected to the bridge adapter too.

And the production network diagram:

After some initial research into both OpenStack and Cumulus (and using them together) I’ve decided to go with a leaf/spine network design. I’ve mainly chosen this because it is a common data centre network design and so is well documented.
The two OpenStack servers will each have two interfaces connected to two ‘top of rack’ switches, which they will use talk to using BGP to advertise their respective leaf switches.

I’ll go over the more specific choices when we get around to configuring the switches.


This was just a quick post, meant as a foundation for the rest of the series and for me to document before I start configuring anything.

The next post in the series covers the use Vagrant to set up the environment in VirtualBox.

Cisco iBGP vs. eBGP Route Selection

The other day my colleague stumbled upon a question in his studies (towards the CCNP route exam) and posed it to me. Initially I had no clue what the answer was, but figured it would make for a good topic of a first post.

The Question:

Say you administer a network with Cisco routers that uses BGP to peer with two upstream providers (eBGP) who both advertise you the same destination (i.e. a default route, You also use BGP as your internal routing protocol (iBGP) and are re-advertising routes to all internal peers. I’ve included a quick diagram below.


Given the above, you as the admin want to use AS200 as the primary exit point for all traffic. How would you complete this?

The Problem:

Thinking about the problem and looking at the diagram, my first thought (and my mistake, which I’ll elaborate on later) was to compare the AD (administrative distance) values of the routes received, since the length of the route received from both the iBGP and eBGP peers is the same and because I remembered with Cisco they have different values; with eBGP (AD of 20) being preferred over iBGP (AD of 200) as a lower value is preferred over a higher one.

With that thought process, I decided that the routes received via eBGP would always be preferred as the lower AD would take preference, and so the only way to make the router connected to AS300 use its’ iBGP path would be to change the AD of iBGP to be lower (and therefore better) than that of eBGP, right?


Initial Setup:

I’ve set up GNS3 with the topology above and have given the routers serial links between each other. I’m using Cisco 7200’s as my virtual routers and am running 15.2(4)M7 code.


For addressing I’ve gone with a /24 between each device in the format 192.168.x.y/24, where x is the interface number and y is the router number. So R1’s address on the interface to R2 would be

Next we’ll configure BGP on each of the devices. Don’t forget to configure next-hop-self on the routers we control otherwise we won’t be able to reach the routes that are re-advertised by our routers. I’ve again shown the configuration of R1:

And lastly  we’ll confirm that all peers are up before we start advertising anything:

Next we will advertise the default route from R3 and R4. Configuration of R3 below:

We can now check that both routers are receiving the advertisements:

Both routers are preferring the route that they received via their external peer, which is what I expected.

Modifying the AD:

Now that we have everything set up, lets attempt to use AS200 as the preferred path on both routers.

As we already know this is the case on R1, we simply need to change the administrative distance on R2 for routes received from R1. We can do this by using the command distance 19  which sets all routes received from the neighbour with an AD of 19 (and therefore better than the eBGP AD of 20)

But wait, this hasn’t changed the preferred route?

The Solution:

To figure out why this happens, we need to think about how a route is installed in the routing table. There are several parts to this process:

– Routing processes (protocols) determine the best route for each destination (using their metric) and attempt to install these into the routing table.

– The router then receives all routes attempting to be installed by all processes and if there are any conflicting routes (routes to the same destination) then AD is used as a tie breaker before placing one into the routing table to be used for forwarding.

Simply put, AD is only evaluated when there are multiple paths to the same destination attempting to be installed by more than one routing process. So in our situation, because iBGP and eBGP are a part of the same process on a Cisco router, the BGP process chooses only one of the routes to attempt to install into the routing table and therefore AD is not evaluated.

However, if we did have the same route ( being put forward by another routing process, i.e. OSPF, AD would be the tie breaker.


So how does BGP choose the best path then?

Going back to our example; before our routers can install the route into their routing table, it needs to be received by the BGP process which picks the route to install by the attributes that a route has.

In our case, BGP has gone through the route selection process and has reached step 7, where routes learnt by eBGP are preferred over routes learnt via iBGP.

I won’t cover how BGP makes its’ route selection here as that is another topic entirely. All you need to know is that BGP’s metric is actually a lot of different metrics (called attributes) that are compared one by one with other contender routes (other routes to the same destination received from other peers) until there is a difference and the tie is broken for a best route to be chosen.


To solve our problem, we can use the metric called ‘local preference’. This one is number 2 on the list of attributes, meaning it is looked at second (after a Cisco proprietary attribute called weight) to see if it can be used as a tie breaker.

We can check the local preference of each route that we have received using the command show ip bgp:

Here we can see that the local preference of both routes received is 100 (lines 9 and 13), which is also the default local preference value for all BGP routes.

In order to make the local preference for one of our routes higher than the other (a higher number is preferred with local preference), we would apply a route-map to the neighbor we want to change. For this problem I’ll set the local preference of the peering to AS200 higher than 100 to prefer this over all other peers.

We need to clear the neighbor for this to take effect as well:

Now, when we check BGP, we can see that it has chosen the iBGP route instead:

Lastly, AD comes back into play when the route is installed into the routing table. If the installed route was learnt from an eBGP peer, the route would have an AD of 20 and if it was instead learnt by iBGP it would instead have an AD of 200.

If we check the routing table on R2, we can see this has now been installed; with an AD of 19 that we set earlier as well:



Powered by WordPress & Theme by Anders Norén