vRealize Network Insight 3.5 NTP Changes

With the release of vRealize Network Insight 3.5 the install and boot now has a mandatory requirement on NTP.  If your installing it and don't point it to a valid NTP source (I used my Domain Controller which technically isn't a valid NTP source) you might see the following when issuing the show-service-status command via the Platform CLI (Default credentials can be found here):

Simply issue the setup command and change the NTP source, I used uk.pool.ntp.org.  Once this change has been made the services will restart and you should see all green when issuing the show-service-status command again:

p.s.  Check out the new PCI Compliance dashboard in version 3.5.

Thanks to Hirdesh Gupta from the Customer Support and Operations team for pointing this out

New Home Lab 2017 Edition

Over the last few months I've really been struggling for resources with my Home Lab so figured it was time for an upgrade.  My previous lab comprised of the following:

3 x HP ML310 G8's with 32GB Ram

1 x RouterBOARD RB751

1 x Linksys SRW2016 16 Gigabit Switch

1 x Synology DS412+ (4 x Crucial m4 256GB Sata 3 SSD)

http://www.m80arm.co.uk/2013/08/home-lab-upgrade-more-power.html

I normally leave all three hosts powered on but during holidays and quite periods I often power off two of them and just keep one permanently on which runs my Remote Desktop Gateway, Domain Controller and Jumpbox VM's.  I wanted a new host that could handle at least 128GB Ram with sufficient CPU to run multiple nested ESXi hosts and NSX Managers / Controllers / Edges etc.

I opted for a Supermicro SYS-5028D-TN4T (https://www.supermicro.com/products/system/midtower/5028/sys-5028d-tn4t.cfm) maxed out with 128GB RAM and only running a few local SSD's for storage as the bulk of my critical VM's still run on my Synology DS412+.  This is what the bulk of my testing and demo VM's will be running on.  I also purchased an Intel NUC (NUC6i5SYH) with 16GB Ram for my infrastructure VM's that need to be on 24x7.

Both boxes are currently running ESXi 6.5 and so far they are all performing as expected.  The NUC was a great choice as it's draws very little power and silent.  It wasn't worth the extra money to max it out with 32GB Ram as the bulk of my resource intensive VM's will run on my Supermicro.

Big thanks go out to Tom from Server Factory who sorted me out with a great deal on the Supermicro.

Resolving Error "Failed to clear disk on...." when clearing a partition table

I'm starting a new project and I needed to spin up a single nested ESXi host and deploy a VCSA to manage and integrate with NSX.  I used to run a VSAN cluster a while back to get more familiar with the product but disabled it when I was finished.  When starting this new project I decided to utilise the SSD in my host as a datastore for my new nested ESXi host to improve performance.  I upgrade the host to ESXi 6.0 U2 to utilise the new HTML5 host Web Interface and started to create the new SSD Datastore.  I first had to clear the existing partition table which is a lot easier in the web client now rather than having to use the CLI but I encountered the "Failed to clear disk Local HP Disk" error:

I then tried to do this via partedUtil but encountered a "Read-only file system during write on" error:

I verified the device wasn't being used for any existing datastore's, DRM's or diagnostic logs:

I remembered that this device was previously used for VSAN so checked that it still wasn't a member of the cluster, which it wasn't, but the devices were still associated with the disk group somehow:

Problem was resolved by using the "esxcli vsan storage remove -u <VSAN UUID>" to disassociate the device from the VSAN Cluster:

Once this was completed the devices were no longer in use and I could clear the partitions and create a local datastore.

Communication Channel Health not showing correct health

I recently had an issue in my homelab whereby the communications channel health was showing that communication from NSX Manager to the firewall agent was down but NSX Manager to Control Plane Agent and Control Plane Agent to Controllers were both fine:

This seemed a bit strange to me because in order for NSX Manager to communicate with the Control Plane Agent (NETCPA) it does this via the vShields Firewall Deamon (VSFWD) as per my pervious blog article on NSX 6.2 Communication Channel Health.

The environment was working as expected so I started troubleshooting by verifying that both deamons were in fact running on the host, as expected they were:

I didn't have much time to troubleshoot as I was travelling home so I picked this up the following day, when opening my RDP connection to my jumpbox I noticed that I was not automatically logged out of the web client.  I then logged out and refreshed the browser but the login page would not load.  I tested this on another browser which displayed the login screen as expected, once logged in the communications channel health with the host in question was showing that everything was OK.  A simple clear of the chrome cache on my jumpbox VM resolved the inconsistencies with the communications channel health.

Configuring BGP between RouterOS and an NSX Edge

I use a RouterBOARD RB715G in my homelab as my router and layer 3 switch which has been absolutely rock solid since I purchased it.  I'm probably not even using 10% of it's functionality but it's definitely worth the money.  When testing NSX and creating logical networks I always end up creating static routes into the NSX environment so I can test connectivity from my physical workstation so I decided to configure BGP from the Edge to the MikroTik.  The diagram below shows my current network layout.  I have OSPF configured from the distributed logical router (PA-DLR-01) to the Edge (PA-Edge-01)

Below are the results from the show ip route command on both the Edge and the DLR:

PA-Edge-01:

PA-DLR-01:

Below is the route information from my Mikrotik router:

As you can see OSPF is populating the routes from PA-DLR-01 into PA-Edge-01.  Now it's time to configure BGP between PA-Edge-01 and Mikrotik.  I'm not going to show you how to configure BGP on the Edge as there are numerous blog articles out there that document this process.  To configure BGP on the Mikrotik router log in and navigate to Routing and then BGP and edit the default entry in the Instances tab:

Ensure you have entered a router ID and Redistribute Connected and Redistribute Static options are ticked (Or whatever you want to redistribute via BGP).  Navigate to the Peers tab and add a new entry:

Give the new entry a name and then enter the IP address of the Edge, in my case it's 10.201.1.41. Enter the Remote AS number and then set the Default Originate option to Always and click OK.  If everything is configured correctly all directly connected routes to the PA-WebTier, PA-AppTier and PA-DBTier should appear in the Mikrotik's route list and any new networks directly connected to the PA-DLR-01 should appear as well:

PA-Edge-01

PA-DLR-01

Mikrotik:

Thanks to Geordy Korte (Blog | Twitter) for the assistance in configuring the Mikrotik router.

Decommission old vCenter Server 6.0 Appliance from Platform Services Controller

I deployed a temporary vCenter Server 6.0 Appliance to my existing lab to test the cross vCenter vMotion functionality across three vCenter appliances.  Once I validated this I no longer had the need for the VCSA and didn't want it taking up valuable resources in my lab.  My OCD also kicked in and I wanted to get rid of the associated error message that displayed within the web client:

As a good techie I searched for the solution but not couldnt find anything.  I posted the issue to twitter and legend William Lam (Blog | Twitter) was kind enough to post a link to a VMware KB article that explained the command:

If you've done what I've done and forgotten which VCSA was the PSC when you SSH in it should tell you at the logon screen:

After following the KB article I was able to successfully remove the decommissioned VCSA from the PSC and I no longer receive the warning within the Web Client.  I did have a slight hiccup in that the name of the VCSA in the error above was in capitals but inside the component manager database it was all lower case so the command had to be in lower case

Nested vSphere 6.0 with VMware tools as standard

Just a quick post on something that I just noticed.  I run a lot of nested ESXi hosts in my homelab for testing as it's quicker and cheaper to stand these up instead of purchasing new hardware.  With vSphere 6.0 when you created a nested ESXi host VMware tools is automatically enabled inside the guest OS without having to install the fling

Replacing Sky's SR102 modem / router

This post is mainly for my own knowledge as with age and NSX my brain doesn't seem to retain information as good as it use to.

** Please note it is against Sky's policies to use any other router / modem apart from the one supplied **

I recently moved house and was no longer in a Virgin Media area and since I was ordering Sky television I also opted for their Sky Fibre unlimited package as well.  I'm a big fan of the MikroTik routers and switches and use a RB751G in my lab as my default router and L3 gateway and obviously wanted to continue using this in my new house.

Sky Fibre unlimited now comes with the combined modem / router called the SR102 which is adequate for home use, but not for my lab:

When researching I found that I simply couldn't replace the SR102 with a standard BT OpenReach modem as Sky use DHCP option 61 as a client authentication as per ITEF:

"
9.14. Client-identifier

This option is used by DHCP clients to specify their unique identifier.  DHCP servers use this value to index their database of address bindings.  This value is expected to be unique for all clients in an administrative domain.

Identifiers SHOULD be treated as opaque objects by DHCP servers.

The client identifier MAY consist of type-value pairs similar to the 'htype'/'chaddr' fields defined in [3]. For instance, it MAY consist of a hardware type and hardware address. In this case the type  field SHOULD be one of the ARP hardware types defined in STD2 [22].  A hardware type of 0 (zero) should be used when the value field contains an identifier other than a hardware address (e.g. a fully qualified domain name).  

For correct identification of clients, each client's client-identifier MUST be unique among the client-identifiers used on the subnet to which the client is attached.  Vendors and system administrators are responsible for choosing client-identifiers that meet this requirement for uniqueness.

The code for this option is 61, and its minimum length is 2.

Code   Len   Type  Client-Identifier
+-----+-----+-----+-----+-----+---
|  61 |  n  |  t1 |  i1 |  i2 | ...
+-----+-----+-----+-----+-----+---
"

So, in order to replace the SR102 with an OpenReach modem and MikroTik router you first need to obtain the username and password used to authenticate with Sky's servers.  This can be achieved by using wireshark and sniffing the DHCP discover packets when powering on the SR102.  This video shows you how to extract the username and password:

Once you have it extracted in the form of MACADDRESS@skydsl|PASSWORD i.e. (00:00:00:00:00:00@skydsl|a34sdre6) you need to convert it from ASCII to HEX using an online converter (This worked for me - http://www.asciitohex.com/)

Once you have the client identifier in HEX format you need to configure option 61 on the DHCP client of your MikroTik router.  Log into WinBox and browse to IP and then DHCP Client and in the Option 61 option enter the value as 0xHEXVALUE (You need to put 0x at the front of the value)

Once finished ensure your MikroTik is set to DHCP and not PPPoE or static and you should recieve and IP address from Sky's DHCP servers:

I also configured the WAN port of my MikroTik with the same MAC address as my SR102 as I've read a few articles that say it can potentially take a few hours for the MAC address to time out which means if you need to swap back to the SR102 you potentially have wait before connectivity will be restored.

Upgrading to NSX 6.1

In this article I'm going to go through the steps of upgrading my nested NSX test lab from version 6.0.4 to 6.1  Chris Wahl (Blog | Twitter) has a great article with what's new in NSX 6.1.  First step is to download the NSX 6.1 upgrade bundle from the VMware site:

Once it's downloaded log into the NSX Manager admin page and browse to the Upgrade section:

Click on the Upgrade button and choose the upgrade bundle that you just downloaded and click Continue:

The upgrade bundle should start uploading.  This will take a few minutes:

Once the bundle has uploaded click Continue.  The upgrade process warns you to ensure you have taken a snapshot prior to the upgrade.  Once you are ready, click Upgrade:

Once the process starts you can monitor its progress:

Once the upgrade process has completed the NSX Manager VM will automatically reboot and take you back to the login screen.  Once you log back in and go to the Upgrade section you should see it completed successfully and that it's now running at version 6.1:

Now that the NSX Manager is running at version 6.1 it's time to upgrade all the controllers and Edges.  Since this is a test lab I only have one controller VM so as long as there are no vMotions within my lab I should not have any connectivity issues.  To upgrade the controller VM log into the vSphere Web Client and navigate to the NSX Manager plugin and then click on the Installation menu item:

Click on the Upgrade Available link and then Yes to upgrade the NSX controllers:

If you refresh the vSphere Web Client you should see the installation in progress and then eventually the controller rebooting:

Once the controller has rebooted it should also be at version 6.1.  

Next were going to upgrade the network virtualization components on the hosts.  Browse to the Installation tab and click update:

The VIBS on each ESXi host should update to 6.1 and show as Ready:

The final step is to upgrade any Edges that you have within your environment.  In my lab I only have two so this should be nice and easy.  As you can see, they are both at version 6.0.4:

To upgrade the Edges simply select one, click on the Actions menu and then Upgrade Version

Finally click Yes to confirm:

Once all Edges are upgraded they should show version 6.1:

The steps above shows you how to upgrade to NSX 6.1 without any issues.  During my upgrade I did have issues upgrading the Edges but I think this was because I hadn't upgraded the hosts prior to the Edges.  Ultimately I ended up re-creating the Edges but if I had RTFM I don't think I would have encountered any issues.  Also, remember to disable any firewall rules that block L2 connectivity between two VM's before testing :)

That's the upgrade complete, have yourself a well deserved cup of tea and go explore some of the new features.

Offline upgrade of ESXi hosts to 5.5 U2

This is a quick post to show you how to upgrade your ESXi hosts to vSphere 5.5 U2 without the use of vCenter Update Manager, and also to remind me for later updates.  I currently use vCenter Update Manager to upgrade my production hosts but it's not worth deploying another instance for my nested environment and unfortunately you can only have one vCenter Update Manager per vCenter instance.  So, first thing to do is download the vSphere 5.5 U2 offline bundle:

Next you need to upload the offline bundle .zip to a datastore that's accessible by all ESXi hosts that you wish to upgrade.  In my example I'm going to use IX2-ISCSI-TestLab:

Once the .zip is uploaded place the host you wish to upgrade into maintenance mode and SSH into the host and run the following command to list the image profiles that are in that offline bundle:

esxcli software sources profile list -d /vmfs/volumes/IX2-ISCSI-TestLab/update-from-esxi5.5-5.5_update02-2068190.zip

Select the profile that you want to install and run the following command:

esxcli software profile update -d /vmfs/volumes/IX2-ISCSI-TestLab/update-from-esxi5.5-5.5_update02-2068190.zip -p ESXi-5.5.0-20140902001-standard

Once completed you will be instructed to reboot the host:

Once the host reboots the version should now show as 5.5 U2:

Simply take it out of maintenance mode and upgrade the remaining hosts in the cluster and don't forget about updating VMtools on all your VM's

Happy offline updating