Creating a Universal Logical Switch in NSX 6.2 and vMotioning a VM from one vCenter to another

Now that we have our Universal Transport Zone that spans clusters across multiple vCenter servers it's time to create a Universal Logical Switch, attach two virtual machines to it and then vMotion one from the Palo Alto vCenter to the New York vCenter.  One thing to remember when creating any universal objects is that they need to be created on the NSX Manager that has the primary role.  If you try to do it whilst connected to an NSX Manager with the secondary role you will not see any Universal Transport Zones.  Log into the vSphere Web Client and navigate to Logical Switches within the Network & Security tab and create a new Logical Switch whilst ensuring you are connected to the NSX Manager with the primary role:

Once the switch has been created you will notice that it's VNI number falls inline with the range we specified within the Universal Segment ID Pool:

If you switch across to the NSX Manager with the secondary role you will also see the same Universal Logical Switch created with the same VNI number:

If we check the VDS portgroups you should see the newly created portgroup in both vCenters:

I have the following two VM's which I am going to attach to the Universal-Web-Tier-01 Logical Switch and power on ensuring that are both in the same vCenter:

WEB01 - 192.168.1.11

WEB02 - 192.168.1.12

As you can see we have layer 2 connectivity between the two virtual machines:

Now time to vMotion WEB02 from the Palo Alto vCenter (PA-VCSA-01) to the New York vCenter (NY-VCSA-01).  Right click on the VM and select Migrate:

Select the option to Change Compute Resources only and click Next (The same storage is available to both vCenters / hosts in my lab):

Click on the clusters tab and select the cluster you want to migrate the VM to and click Next.  In my example I am moving WEB02 from the PA-Compute cluster in PA-VCSA-01 to the NY-Compute cluster in NY-VCSA-01:

Select the folder to place the VM into and click Next:

Select the network you want to attach the VM to in the other vCenter.  In my example I'm connecting it to the same Universal Logical Network.  It would be nice for this to have been automatically selected for me:

Set the appropriate priority and click Next:

Verify your setting, click Next and watch the magic happen:

WEB02 has now been vMotioned from the Palo Alto vCenter across to the New York vCenters with no drop in connectivity:

The possibilities for this are endless especially around Disaster Recovery with Site Recovery Manager.

Creating a Universal Transport Zone in NSX 6.2

One of the new features in NSX 6.2 is the ability to span logical networks across multiple vCenters. In order to do this we now have the ability to create a Universal Transport Zone than can contain clusters from different vCenters. In my lab I'm going to create three new Transport Zones:

PA-TransportZone - A dedicated transport zone for the Palo Alto DataCenter that will contain the PA-Compute and PA-MGMT clusters.

NY-TransportZone - A dedicated transport zone from the New York DataCenter that will contain the NY-Compute and NY-MGMT clusters.

Universal-TransportZone - A dedicated transport zone that will contain all four clusters that span over two vCenters.

In order to achieve this we first need to create local and universal segment ID pools.  Segment ID pools are used to allocate VNI's (VXLAN Network Identifiers) to logical networks. I'm going to use the following VNI pools:

Palo Alto Segment ID Pool - 5000-5999
New York Segment ID Pool - 6000-6999
Universal Segment ID Pool - 10000-10999

This will allow me to create 1000 logical networks in Palo Alto and New York but then also another 1000 than can span both vCenters.

To start log into the vSphere Web Client and navigate to Installation -> Logical Network Preparation and then click on the Segment ID tab.  Ensure that you are connected to the NSX Manager that has the Primary Role:

Edit the setting and add the segment ID pools for the primary NSX Manager:

Click OK and then change to the secondary NSX Manager.  You should already notice that the Universal Segment ID's have already been populated.  Add the segment ID's for the secondary ensuring that they don't overlap with the Universal segment ID's:

Click on the Transport Zone tab and create a new Transport Zone.  The first Transport Zone we are going to create is the PA-TransportZone and add the two Palo Alto DataCenter clusters:

Now create a new Transport Zone but ensure you tick the option to Mark this object for Universal Synchronization and add the two Palo Alto DataCenter Clusters again:

We now have two Transport Zones:

Switch to the secondary NSX Manager and you will notice that we already have the Universal Transport Zone available.  Add the two New York DataCenter clusters to it:

Finally create the second local transport zone and add the two New York DataCenter clusters to it:

We should end up with a Local and Universal Transport Zone in both the Palo Alto and New York sites:

Installing NSX 6.2 within a multi vCenter Environment

With the release of NSX 6.2 we now have the ability to span Transport Zones, Logical Networks, Distributed Logical Routers (DLR's) and even firewall rules across multiple vCenters.  In this post I'm going to show you how to install NSX 6.2 in a multi vCenter environment.  In my lab at home I have two physical ESXi hosts each running four nested ESXi hosts to simulate a Palo Alto and New York Datacenter.  All hosts are running vSphere 6.0 and there are two vCenters in linked mode with the Platform Services Controller running on the Palo Alto vCenter. The screen shot below shows the configuration via the vSphere Web Client:

I'm not going to show you how to deploy NSX Manager as this has been very well documented by numerous bloggers. I'd recommend Giuliano Bertello's (Blog | Twitter) NSX for newbies series.  So I've deployed two NSX managers called PA-NSX-01 and NY-NSX-01 and associated them with the corresponding vCenters:

PA-NSX-01 is configured to use PA-VCSA-01 for it's lookup service and also it's vCenter server:

NY-NSX-01 is configured to use PA-VCSA-01 for it's lookup service and NY-VCSA-01 as it's vCenter server:

Remember when using vCenter 6 the lookup service port is now 443 instead of 7444.

After logging into the Web Client you will now see that you can manage multiple NSX Managers:

I've deployed a single controller in the Palo Alto management cluster as this is just a test lab:

I'm also going to prepare all 4 clusters and configure the host VTEP's:

Palo Alto Management

Palo Alto Compute

New York Management

New York Compute

You will notice that you now have the ability to select the NSX Manager that you want to perform tasks on via the NSX Manager menu:

I've now prepared all four clusters:

I'm now going to configure PA-NSX-01 to be the primary NSX Manager and NY-NSX-01 to be a secondary.  Navigate back to Installation and then the Management tab and highlight the NSX Manager that you want to assign the primary role to, click Action and then Assign Primary Role

Click Yes to confirm the operation:

Keep the primary NSX manager highlighted and click Actions and then click Add Secondary NSX Manager:

Ensure the IP address is correct for the secondary NSX Manager and enter the Admin credentials for the secondary NSX Manager:

Once finished we should have Primary and Secondary NSX Manager instances and you should see the secondary instance using the NSX Controller from the primary site:

Within NSX 6.2 there are four NSX Manager Roles:

Standalone - Default option for new installation or upgrade. All objects in a Standalone NSX Manager are local.

Primary - One NSX Manager per Cross VC NSX deployment is assigned the Primary role. Deploys and manages the Universal Controller Cluster and any Universal objects.

Secondary - The Secondary role is assigned when adding a Standalone NSX Manager to a Primary. Universal objects from the primary are synchronised to the Secondary where they are read-only.

Transit - Temporary role used when an NSX Manager has a Primary or Secondary Role removed. If universal objects still exist the NSX Manager is assigned a Transit role until these objects are deleted then it can move to a Standalone role.

NSX 6.2 for vSphere Released

Just a quick post to let you know that NSX 6.2 for vSphere has GA'd and available to download on vmware.com site.  The release notes can be found here. So, what's new in 6.2

NSX vSphere 6.2 includes the following new and changed features:

Cross vCenter Networking and Security

• NSX 6.2 with vSphere 6.0 supports Cross vCenter NSX where logical switches (LS), distributed logical routers (DLR) and distributed firewalls (DFW) can be deployed across multiple vCenters, thereby enabling logical networking and security for applications with workloads (VMs) that span multiple vCenters or multiple physical locations.
• Consistent firewall policy across multiple vCenters: Firewall Rule Sections in NSX can now be marked as "Universal" whereby the rules defined in these sections get replicated across multiple NSX managers. This simplifies the workflows involving defining consistent firewall policy spanning multiple NSX installations
• Cross vCenter vMotion with DFW: Virtual Machines that have policies defined in the "Universal" sections can be moved across hosts that belong to different vCenters with consistent security policy enforcement.
• Universal Security Groups: Security Groups in NSX 6.2 that are based on IP Address, IP Set, MAC Address and MAC Set can now be used in Universal rules whereby the groups and group memberships are synced up across multiple NSX managers. This improves the consistency in object group definitions across multiple NSX managers, and enables consistent policy enforcement
• Universal Logical Switch (ULS) - This new functionality introduced in NSX 6.2 as a part of Cross vCenter NSX allows creation of logical switches that can span multiple vCenters, allowing the network administrator to create a contiguous L2 domain for an application or tenant.
• Universal Distributed Logical Router (UDLR) - This new functionality introduced in NSX 6.2 as a part of Cross vCenter NSX allows creation of distributed logical routers that can span multiple vCenters. The universal distributed logical routers enable routing across the universal logical switches described earlier. In addition, NSX UDLR is capable of localized north-south routing based on the physical location of the workloads.

Operations and Troubleshooting Enhancements

• New traceflow troubleshooting tool: Traceflow is a troubleshooting tool that helps identify if the problem is in the virtual or physical network. It provides the ability to trace a packet from source to destination and helps observe how that packet passes through the various network functions in the virtual network.
• Flow monitoring and IPFIX separation: In NSX 6.1.x, NSX supported IPFIX reporting, but IPFIX reporting could be enabled only if flow reporting to NSX Manager was also enabled. Flow monitoring with IPFIX functionality are included in NSX Manager in layer deployment. Starting in NSX 6.2.0, these features are decoupled. In NSX 6.2.0 and later, you can enable IPFIX independent of flow monitoring on NSX Manager.
• New CLI monitoring and troubleshooting commands in 6.2: See the knowledge base article for more information.
• Central CLI: Central CLI reduces troubleshooting time for distributed network functions. Commands are run from the NSX Edge command line and retrieve information from controllers, hosts, and the NSX Manager. This allows you to quickly access and compare information from multiple sources. The central CLI provides information about logical switches, logical routers, distributed firewall and edges.
• CLI ping command adds configurable packet size and do-not-fragment flag: Starting in NSX 6.2.0, the NSX CLI 'ping' command offers options to specify the data packet size (not including the ICMP header) and to set the do-not-fragment flag. See the NSX CLI Reference for details.
• Show health of the communication channels: NSX 6.2.0 adds the ability to create communication channel health. The channel health status between NSX Manager and the firewall agent, between NSX Manager and the control plane agent, between host and the NSX Controller can be seen from the NSX Manager UI. In addition, this feature detects when configuration messages from the NSX Manager have been lost before being applied to a host, and it instructs the host to reload its NSX configuration when such message failures occur.
• Standalone Edge L2 VPN client CLI: Prior to NSX 6.2, a standalone NSX Edge L2 VPN client could be configured only through OVF parameters. Commands specific to standalone NSX Edge have been added to allow configuration using the command line interface. The OVF is now used for initial configuration only.

Logical Networking and Routing

• L2 Bridging Interoperability with Distributed Logical Router: With VMware NSX for vSphere 6.2, L2 bridging can now participate in distributed logical routing. The VXLAN network to which the bridge instance is connected, will be used to connect the routing instance and the bridge instance together.
• Support of /31 prefixes on ESG and DLR interfaces per RFC 3021
• Support of relayed DHCP request on the ESG DHCP server
• Ability to keep VLAN tags over VXLAN
• Exact Match for Redistribution Filters: The redistribution filter has same matching algorithm as ACL, so exact prefix match by default (except if le or ge options are used).
• Support of administrative distance for static route
• Ability to enable/disable uRPF check per interface on Edge
• Display AS path in CLI command show ip bgp
• HA interface exclusion from redistribution into routing protocols on the DLR control VM
• Distributed logical router (DLR) force-sync avoids data loss for east-west routing traffic across the DLR.
• View active edge in HA pair: In the NSX 6.2 web client, you can find out if an NSX Edge appliance is the active or backup in an HA pair.
• REST API supports reverse path filter(rp_filter) on Edge: Using the system control REST API, rp_filter sysctl can be configured, and is not exposed on REST API for vnic interfaces and sub-interfaces. See the NSX API Guide for more information.
• Behaviour of the IP prefix GE and IP prefix LE BGP route filters: In NSX Release 6.2, the following enhancements are made to BGP route filters:
• For the null route network address (defined as ANY or in CIDR format 0.0.0.0/0), less-than-or-equal-to (LE) and greater-than-or-equal-to (GE) keywords are no longer allowed. In previous releases, these keywords were allowed.
• LE and GE values in the range 0-7 are now treated as valid. In previous releases, this range was not valid.
• For a given route prefix, you can no longer specify a GE value that is greater than the specified LE value.

Networking and Edge Services

• The management interface of the DLR has been renamed to HA interface. This has been done to highlight the fact that the HA keepalives are going through this interface and issues on this interface can result in split-brain situation.
• LB Health Monitoring Improvements: Delivers granular health monitoring, that reports information on failure, keeps track of last health check and status change, and reports failure reasons.
• Support VIP and Pool port range: Enables LB support for applications that require a range of ports.
• Increased number of VIP: VIP support increased up to 1024.

Security Service Enhancements

• New IP address discovery mechanisms for VMs: Authoritative enforcement of security policy based on VM names or other vCenter-based attributes requires that NSX know the IP address of the VM. In NSX 6.1 and earlier, IP address discovery for each VM relied on the presence of VMware Tools (vmtools) on that VM or the manual authorization of the IP address for that VM. NSX 6.2 introduces the option to discover the VM's IP address using DHCP snooping or ARP snooping. These new discovery mechanisms enable NSX to enforce IP address-based security rules on VMs that do not have VMware Tools installed.

Solution Interoperatibility

• Support for vSphere 6.0 Platform Services Controller topologies: NSX now supports external Platform Services Controllers (PSC), in addition to the already supported embedded PSC configurations.
• Support for vRealize Orchestrator Plug-in for NSX 1.0.2: With NSX 6.2 release, NSX-vRO plug-in v1.0.2 is introduced in vRealize Automation (vRA).

Migrating Workloads into NSX

In this post I'm going to show you how to migrate virtual workloads from a VLAN backed portgroup into NSX and onto a VXLAN backed portgroup (Logical Network).  I've setup two VM's connected to a VLAN backed portgroup called Compute-VLAN14-Test and my router (RouterOS) has an L3 interface in this VLAN (10.1.14.1).  The VM's have an IP address on the 10.1.14.0/24 network and their default gateway is 10.1.14.1:

The plan is to create a new Logical Network (VXLAN backed portgroup) called Test-Network within NSX and then bridge this logical network and the VLAN and seamlessly migrate workloads across. The default gateway will remain on the physical router for external connectivity:

Once all workloads have been migrated the L3 interface the on physical router will be disabled and the logical network will be connected to the Logical Distributed Router (LDR) and an IP address of 10.1.14.1 will be assigned to that interface. Since I have OSPF configured between the LDR and the EDGE and BGP between the EDGE and the physical router (RouterOS) the 10.1.14.0/24 network will be advertised to the physical router and L3 connectivity will be restored:

This will allow us to migrate workloads into NSX and onto a logical network without having to re-ip any VM's and minimize outages to the application.  With that, lets get started.  We can see that the two VM are connected to the Compute-VLAN14-Test portgroup

We are now going to create the logical network by logging into the vSphere Web Client, navigating to the Networking and Security Plugin and then selecting logical switches.  I've created a new logical network called Test-Network and assigned it to my transport zone:

Now, I'm going to create the bridge between the Compute-VLAN14-Test portgroup and the logical network Test-Network. Bridging is configured on an LDR so I'm going to navigate to NSX Edges, modify my LDR and go to the Bridging tab and add a new Bridge.  In order to add a new bridge you need to supply a name and then the logical network and portgroup that you want to bridge together and then publish the changes:

You might have noticed that I'm bridging MGMT-VLAN14-Test instead of Compute-VLAN14-Test.  This is because I have a separate vSphere Distributed Switch for Management and Compute and my LDR resides in the management cluster.  I'm now going to move TEST01 onto the new logical network and show that we still have L2 and L3 connectivity:

We now have the following topology:

We are now going to change the network for TEST02 and attach it to the Test-Network, disable the L3 interface on the physical router and attach the Test-Network to the LDR and give it an IP address of 10.1.14.1.  After this the network will converge and the 10.1.14.0/24 network will be advertised to the EDGE and then back to the physical router.  First thing to do is disable the L3 interface on the physical router.  As you can see the interface has been disabled, the route has been removed from the routing table and I've lost connectivity to TEST01 and TEST02 (My client IP is in the 10.1.9.0/24 range):

We now go back into the LDR and add an interface into the Test-Network and assign it an IP address of 10.1.14.1:

As soon as this is completed L3 connectivity has been restored and the route has been advertised back up to the physical router:

Final task is to remove the bridge on the LDR and decommission the Compute-VLAN14-Test portgroup and VLAN from the physical infrastructure.  The default gateway for the VM's is the LDR and we have migrated our workloads with minimal downtime and no IP address changes. We are now left with the following topology:

How awesome is NSX

NSX Fundamentals Online - Thursday 6th August 16:00 - 19:00 UK Time

VMware is hosting an NSX Fundamentals Online training course on Thursday 6th August from 16:00 to 19:00 UK time.  You can register for the event here and they are normally available to view on demand after the event so if you can't make it you wont miss out.  The agenda is as follows:

VMware NSX Fundamentals Online is a three-hour virtual event designed for IT professionals where you will discover how to bring the operational model of a virtual machine to your data center network, so you can transform the economics of network and security operations.

Configuring OSPF / BGP Authentication within NSX and RouterOS

A little tip Andy Kennedy (Twitter) gave in one of his internal presentations was to use authentication when configuring a routing protocol during a Proof of Concept to avoid redistributing routes into a production environment in the event of a misconfiguration.  This is just a quick post to show you how to configure authentication when using OSPF and BGP within NSX and RouterOS.  My lab environment currently looks like this:

I have OSPF configured between my DLR (PA-DLR-01) and my Edge (PA-EDGE-01) and BGP between my Edge (PA-EDGE-01) and my physical router (Mikrotik) which runs RouterOS.  To configure OSPF authentication on the DLR simply fire up the vSphere web client and navigate to Networking & Security -> NSX Edges -> DLR you want to configure -> Routing -> OSPF and then modify the area definition.  Change the authentication type to Password and then enter a password, click OK and then publish the changes:

At this point routes should stop distributing into the Edge as we need to configure the same password on the Edge (PA-EDGE-01).  Once configured on Edge routes should start populating again.  We can also run a show ip ospf interface on the edge and see that authentication is enabled:

To configure authentication for BGP on the edge navigate to Networking & Security -> NSX Edges -> Edge you want to configure -> Routing -> BGP and edit the neighbour you want to configure for authentication.  In the password field enter the required password, click OK and then publish the changes:

Route distribution will stop from the Edge into the physical router and vice-versa until we configure authentication within the physical router.  To do this is RouterOS simply log in and navigate to Routing -> BGP -> Peers and modify the Edge peer.  Within the TCP MD5 Key field enter the same password you used on the Edge and click OK:

Once you do this routes should start redistributing again.  Now if the Edge uplink interface is accidentally connected to the wrong VLAN backed portgroup with a physical router with the same neighbour IP address we will not accidentally redistribute routes into the production environment.