Nested vSphere 6.0 with VMware tools as standard

Just a quick post on something that I just noticed.  I run a lot of nested ESXi hosts in my homelab for testing as it's quicker and cheaper to stand these up instead of purchasing new hardware.  With vSphere 6.0 when you created a nested ESXi host VMware tools is automatically enabled inside the guest OS without having to install the fling

Performing a cross vCenter vMotion with vSphere 6.0

With the release of vSphere 6.0 it was time to upgrade and completely redesign the homelab.  I wanted to start testing design scenarios around NSX and having multiple vCenter servers but first thing I wanted to try was cross vCenter vMotion which is a new feature in vSphere 6.0.  There are some requirements around cross vCenter vMotion which are required:

• vCenter 6.0 and greater
• SSO Domain
• Same SSO domain to use the UI
• Different SSO domain possible if using API
• 250 Mbps network bandwitdh per vMotion operation
• L2 network connectivity on VM portgroups (IP addresses are not updated)

Within my lab I have two physical ESXi 6 hosts both running 4 nested ESXi 6 hosts each, a vCenter Server Appliance and an NSX Manager.  The four virtual ESXi 6 hosts are added to two clusters (MGMT and Compute) for both PA (Palo Alto) and NY (New York) sites.  Within the New York datacenter I have a TEST01 VM which I will migrate to the Palo Alto Datacenter.  A picture says a thousand words:

All hosts are connected to the same back end storage array so I will not be migrating the storage at the same time and I've already configure VMK ports for vMotion.  Right click on the VM and select migrate:

I'm just going to change the compute resource so ensure it's checked and click Next:

You now need to select the host that you want to migrate the VM to.  You have the option of selecting a specific Host, Cluster, Resource Pool or vApp.  In this example I'm going to move it to the PA-Compute cluster from the NY-Compute cluster.  In order for the clusters to appear I had to enable DRS and EVC on the clusters even though they had exactly the same hardware.  Once the compatibility checks succeed click next:

Select a folder on the remote vCenter to place the virtual machine in and click next:

Select the network on the remote vCenter that you want to connect the VM to.  It doesn't have to have the same portgroup name but obviously it needs to be backed by the same VLAN otherwise you'll loose network connectivity and have to re-ip the VM.  In this example I'm moving a VM from the Compute-VLAN8-NY-Servers portgroup to Compute-VLAN8-PA-Servers.  Once compatibility checks have succeeded click next:

Select the vMotion priority and click next:

Check the destination is correct and click finish and watch the magic happen:

Once the vMotion has completed the VM will now reside in the new cluster attached to the new vCenter:

One ping dropped throughout the entire process:

The task shows completed successfully and the hosts the VM was migrated from and to:

After presenting to numerous customers around what's new with vSphere 6.0 cross vCenter vMotion was definitely a welcomed feature.  So go update your environment and take advantage of all the new features.

Creating a new vMotion VMK adapter using the new vMotion IP Stack

One of the new features of vSphere 6.0 is that you can now route vMotion traffic over layer 3 networks.  You could accomplish this in previous versions by using static routes on a per host basis but it did require approval from VMware before GSS would officially support this.  Now with vSphere 6.0 this is supported out of the box.  This is accomplished by giving vMotion it's own IP stack with it's own default gateway.  I'm going to quickly show you how to create a new vMotion VMK port that utilizes the new vMotion TCP/IP stack.

First, create your VMK port as you normally would and ensure the option for VMkernel Network Adapter is checked and click Next:

Select the portgroup that you can this VMK adapter to be connected to and click Next:

Click on the TCP/IP stack option and select vMotion and click Next:

Give the VMK adapter an IP address and subnet mask.  You'll notice that you don't have the option to enter the default gateway.  We need to configure this option on the actual vMotion TCP/IP stack.  Click next once you've entered the correct IP information:

Verify your settings and click Next:

Your new VMK adapter is created but you still need to configure the default gateway for the IP stack:

Click TCP/IP configuration, select vMotion and then click the pencil to edit it:

Select the DNS option and enter your DNS server and search domain details:

Click Routing and enter your default gateway and then click OK:

The VMK adapter should now have all the correct information to start routing vMotion traffic over layer 3:

Offline upgrade of ESXi hosts to 5.5 U2

This is a quick post to show you how to upgrade your ESXi hosts to vSphere 5.5 U2 without the use of vCenter Update Manager, and also to remind me for later updates.  I currently use vCenter Update Manager to upgrade my production hosts but it's not worth deploying another instance for my nested environment and unfortunately you can only have one vCenter Update Manager per vCenter instance.  So, first thing to do is download the vSphere 5.5 U2 offline bundle:

Next you need to upload the offline bundle .zip to a datastore that's accessible by all ESXi hosts that you wish to upgrade.  In my example I'm going to use IX2-ISCSI-TestLab:

Once the .zip is uploaded place the host you wish to upgrade into maintenance mode and SSH into the host and run the following command to list the image profiles that are in that offline bundle:

esxcli software sources profile list -d /vmfs/volumes/IX2-ISCSI-TestLab/update-from-esxi5.5-5.5_update02-2068190.zip

Select the profile that you want to install and run the following command:

esxcli software profile update -d /vmfs/volumes/IX2-ISCSI-TestLab/update-from-esxi5.5-5.5_update02-2068190.zip -p ESXi-5.5.0-20140902001-standard

Once completed you will be instructed to reboot the host:

Once the host reboots the version should now show as 5.5 U2:

Simply take it out of maintenance mode and upgrade the remaining hosts in the cluster and don't forget about updating VMtools on all your VM's

Happy offline updating

Real World Fault Tolerance Use Cases

I've always been a big fan of VMware's Fault Tolerance feature but could never really find a valid use case to actually implement it.  Fault Tolerance (FT) has some very strict requirements / limitations:

• VM is limited to one vCPU
• VM disks need to be thick-eager zeroed
• VM must be running a supported OS
• Snaphosts are not supported on the VM (Think about how you can backup the VM)
• VM Hot add memory or CPU cannot be utilised
• DRS cannot be utilised
• Physical processors needs to support FT

It is also recommended that you have a dedicated redundant FT network with minimum 1GB pNics for the FT traffic.  In our current infrastructure we are limited to the number of physical nics we can present to the ESXi hosts due to the blade chassis we are currently using.  We were unable to allocate 2 nics to dedicate to FT so we never really utilized FT.

We recently had a requirement to move our DMZ to our ISP and thus reused some decommissioned Dell R610 hosts.  We bumped up the memory and pNic count and now we could comfortably allocate 2 pNics per host for FT.  We use a Citrix Access Gateway (CAG) VPX appliance for remote access to both published apps and desktops and although this is protected with VMware HA, FT offered more protection in the event of a complete host outage.  The only issue that I had was that the CAG was not on the supported list of OS's but it seems to work fine.

We configured the hosts with the required FT network and enabled FT on the VM when it was powered off and 15 minutes later the VM was fully protected with FT

Using Virtualization to migrate between two DMZ’s

We currently have a project ongoing to migrate away from an ASA5510 to two shiny new PaloAlto firewall’s which are located at out ISP.  This part of the project was to migrate all DMZ servers which currently reside on a single ESXi host onto a new DMZ cluster with shared storage at our ISP.  To make things a little complicated, not only were we migrating all our public internet facing services from one firewall to another, but also changing from one public IP address range to another all with minimal downtime.

So, how did virtualization help with this?  Since all public facing servers were already virtualized (Aaprt from our Threat Management Gateway which will be by the end of the project) the actual migration of the VM’s from our current site up to our ISP was very easy using vMotion.  We simple stretched the VLAN up to our ISP and then migrated the servers over night via our 100mb link onto the new infrastructure.  Now this isn’t ideal as traffic was entering the ASA via another link, going across to our second site, going back to our ISP via another link and then hitting the server and then having to travel all the way back down to leave the ASA.  Since this was only a stop gap it was acceptable.  At the same time as stretching the VLAN across to our ISP we created another VLAN for the new DMZ and passed it through to both hosts.  So, both hosts up at our ISP could see the old VLAN hanging off the ASA and the new VLAN hanging of the PaloAlto firewalls.  We then verified and migrated the rule set from the ASA to the new PaloAlto devices and NAT’d the new public IP addresses to private IP addresses in the new VLAN for all the servers we were migrating.

This is where having these servers virtualized saved us potentially 24 to 48 hours downtime (Not including roll back).  We needed to find a way to test the NAT and security rules on the PaloAlto firewalls whilst minimising downtime on the VM’s as some of these servers run public facing websites.  We also needed to change the public IP address for these servers whilst maintaining connectivity.  Luckily all servers were simple web servers with static content.  We arranged downtime to clone each server and then power on the clone which was still connected to the old DMZ and who’s traffic flowed through the ASA.  Once the service was resumed via the clone we could change the network for the original server to be the new VLAN, who’s traffic flowed via the PaloAlto, re-ip it to fall in line with the new DMZ’s IP range and then test both internally and externally by changing the hosts file on our client pc’s to point the hostname to ether the public or private IP address.  This allowed us to test both the NAT and security rules whilst not interrupting any public facing services.  Once we were happy that the rule set was correct and signed off we simply contacted our ISP and requested them to change the public IP addresses for the hosts to point to the new addresses and within 24 to 48 hours users would start connecting to the original server rather than the clone.  After DNS had updated globally we could destroy the clone.  If all these servers were physical we would either have to look into migrating the services to virtual or another physical host which would involve significant work to build new servers and deploy applications or manually move them to our ISP, test firewall rules and then request the DNS changes which could have taken potentially a few days to fully complete and document.

DCUI via SSH

It’s amazing how long you can use a product and then find out something you never knew.  I just found out recently that you can actually open the DCUI interface via an SSH session into an ESXi host.  Simply open an SSH session into your ESXi host and then type DCUI:

Once in the DCUI via SSH you can then perform the same functions as per the actual console session.

Ghost VMkernel Adapter

I came across a strange issue recently when configuring a new vDS in vSphere 5.  I had already configured two VMkernel adapters but for some reason one of them half created.  Only one was visible when viewing the vDS settings on the host:

When attempting to create a new one I’d receive a message stating that one already existed (apologies I don’t have the screen shot as I fixed it before creating this post).  I was unable to locate this within vCenter or even when connecting directly to the host.  Only way I could resolve it was to enable and connect to the host via SSH and list all VMkernel adapters with the following command:

esxcli network ip interface list

Now I could see the offending adapter.  Next option was to remove it:

esxcli network ip interface remove –i <Adapter name>

Now when listing the adapters again it was successfully remove and could be re-added again within vCenter.