snapshot

If you have a virtual system, like VMware, and a storage device behind it you might be able to mirror your whole real / live environment and create a complete playground or shadow network to simulate any of your guest VMs like the real system and being able to change, update and adjust the configuration within this lab environment.. This is actually rather easy to accomplish and can save you a lot of headache while using it.

This is just an example, you might be able to accomplish something similar while just cloning single VMs and in theory it wouldn’t even matter if you use VMware vSphere or something like Microsoft Hyper-V. Still, it has advantages to do it this way, but let me explain it.

Assumed scenario:

You work with an VMware environment with one or more host systems and several guest VMs. You need to update software and configurations on those guest VMs but need to test this beforehand to ensure everything runs smooth. The VMs are stored on a central storage device that can do volume-level snapshots.

Prepare the environment

1. you need one NIC per host that you will connect to a switch
2. best use an independent switch that has no other network connections then to the 1x NIC per host
3. create an Shadow-LAN virtual switch in your VMware cluster and use the 1x NIC per host so VMs through the cluster can communicate proper

What you do:

1. create a snapshot and mount it to the VMware host systems as an additional volume
2. go to the clone-volume file system and add the VMs you need to the inventory (you might want to rename to shadow-<servername> so you can easily identify them)
3. re-configure their network connection from your regular LAN virtual switch to the shadow-LAN virtual switch you prepared
4. start the added guest VM
   1. if you are asked if you moved or copied the guest just say you moved it – to avoid hardware / MAC and other changes possibly causing Windows to want to re-activate
   2. VMware might complain about a duplicate MAC address – you can ignore this, cause you are on two different / independent networks

Real usage example:

Let me give you a more detailed example on this with a few more details on what I personally used and did with this. The example you find below should help you understand the whole principle better.

• VMware cluster with e.g. 10x host systems – we had enough RAM and CPU power that we could have 3x hosts go down – you won’t need that much, but of course you would have buffer for RAM and CPU usage
• Nimble All-Flash storage arrays in the background connected to all the VMware hosts and using the Nimble VMware plugins (Note: Nimble was bought by HP / HPE as off today)
  • the Nimbles are configured to do volume level snapshots multiple times per day
• all physical host systems had a dedicated network card (NIC) connected to an independent physical switch that was NOT connected to any other network switch
• a virtual switch SHADOW-LAN was created and those physical NICs of the hosts systems had been assigned to it
  • this allowed any VM connected to this virtual switch to communicate with other VMs connected to the same virtual switch on other hosts

Due to migrations, software updates and quality controlled systems we constantly had the challenge to test and changes and adjustments thoroughly. So I came up with the solution to just clone a snapshot on the Nimble storage array the VM resided and mounting it to the VMware cluster, taking only minutes and then moving forward to add e.g. domain controllers, DHCP servers, necessary file-system servers and the target guest system to the inventory in VMware, adjusting their name so we could quickly identify them (even adding them to resource pools if necessary) and of course most important just changing their virtual switch configuration to the shadow switch.

Advantages and possibilities:

This now allowed to simulate the whole real world system (VMs) and simulate every change that we wanted. In order to get software there we attached if necessary VHDs that did hold what we needed or we even used a secondary internet connection to briefly connect to licensing services or update services that Vendors only provided online. The advantages of the solutions go even further:

• simulate everything you have in your VMware environment available and have it working like the real / live system
• if necessary, provide internet access while connecting a SECONDARY internet connection (router / firewall) to physical shadow network switch
• adding real printers to the shadow switch to be able to test print-outs (we had those cases)
• add physical workstations to simulate whole production environments
• update / refresh the whole system in only a few minutes by using a fresh-snapshot clone
• only minimal to almost none impact on the storage / free space of your storage device
  • this is due to grabbing a Nimble snapshot that was cloned and therefor created a new branch and only the deltas (changes) had an impact on the storage – we talk even for “bigger” simulations only about a few gigabyte changed data – if at all that much – of course depending on your storage and what you do
• we installed VMware console connections on quality testing workstations so they could access the system directly on the console
  • of course only granting them minimal rights to this specific pool of VMs
  • avoiding that their access to the VMware environment had any impact to the real system
• documenting any changes, challenges faced and solutions found
• due to a physical switch, and best practice using a layer-3 switch, we where able to simulate whole VLANs, routing etc. within the environment and even connecting various physical systems like printers, workstations and temporarily an internet-connection to this environment

It is only a small amount of effort to initially prepare for those simulations, cause the virtual shadow switch, the physical shadow switch and the hosts network card connections to this physical switch are a one time effort. After this you just clone and mount snapshots and add the actual VMs you need while adjusting their network connection to the virtual shadow switch.

Once setup, preparing simulations usually takes less then 30 minutes till everything is cloned, mounted, added to the inventory (incl. NIC adjust to the shadow switch) and booted up.

Why not just clone all VMs via VMware?

Good questions – the answer is simple, this would have an impact on your storage capacity, cause it would create an actual clone. And it actually takes longer to clone individual VMs then to just grab a storage-level snapshot and being able to adjust what you want down to the volume level on the storage. Even the clean-up might be more intense or leave some unwanted data back – while a clone on off the volume only needs you to remove the VM guest system from the inventory and then unmount and delete the whole shadow volume.

I did write this all up cause I wanted to share it – the whole idea is not that special in theory, but I thought it is an good example on how you can accomplish having a huge and decent lab environment with only minimal effort. In any case, I hope the idea behind it will help some off you out there 🙂