VMware Cloud on AWS and native AWS route tables

During my time working with customers who use VMware Cloud on AWS and wish to integrate with native AWS services, I frequently see issues with network connectivity when multiple route tables are in use. Some customers have an automated way of deploying VPCs that use multiple route tables, perhaps for public and private subnet architectures. A prerequisite of deploying a VMware Cloud on AWS SDDC is to connect it to a customer-owned VPC, sometimes referred to as the connected VPC:

https://docs.vmware.com/en/VMware-Cloud-on-AWS/services/com.vmware.vmc-aws-operations/GUID-BC0EC6C5-9283-4679-91F8-87AADFB9E116.html

VMware Cloud on AWS Cluster Rename

Just a quick blog to let you know that a new feature customers have been requesting for a long time, the ability to rename clusters, is now available with clusters that have two or more hosts.

If you navigate to the SDDC within the Cloud Services Portal you can see that I have an SDDC with a 2-node cluster with the default name of Cluster-1:

Within vCenter you can also see the default

VPN termination on a NAT'ed Tier-1 within VMware Cloud on AWS for DMZ traffic segregation

A colleague's customer recently had a requirement to host both DMZ and Production workloads in VMware Cloud on AWS while ensuring that traffic is segregated during transit. Currently, if the customer was to deploy DMZ and Production networks attached to the default Compute Gateway (CGW) Tier-1 then that traffic would be routed by the Tier-1 and thus violate the segregation required as per below:

In SDDC version 1.18 we introduced the ability to deploy multiple compute gateways (Routed, NAT'ed or Isolated) More information

Registering VMware Cloud Disaster Recovery to vCenter with a restricted user account

A customer who is currently looking to deploy VMware Cloud Disaster Recovery (VCDR) globally recently asked about using a single active directory account with the minimum required permissions within vCenter as the account used to register their VCDR connectors to vCenter. For those who are new to VCDR, this is VMware's Disaster Recovery as a Service solution that offers on-demand disaster recovery with a very compelling total cost of ownership in comparison to on-premises.

The customer in question wanted to use a single active directory account across all their vCenters globally and didn't want to add the active directory

Routing to a connected VPC when attached to VMware Transit Connect

Recently there was an internal discussion around a customer request to access an Amazon FSx for Windows File Servers that was currently running in a connected VPC from another SDDC. The topology that the customer was looking at was as follows:

The customer had two SDDC's with one of the SDDC's (SDDC 01) being connected to a VPC that was running Amazon FSx for Windows File Servers. They also wanted

VMware Transit Connect default route and the impact on VPN and HCX Connectivity

I recently had a query from a customer who was implementing intra-region peering between a VMware Transit Gateway and a native AWS Transit Gateway which would then be attached to a security VPC. Their requirement was to ensure that all VM connectivity from the SDDC would traverse the security VPC before egressing out to the internet or back to on-premises. This would require them to add a static route into the vTGW to point all traffic (0.0.0.0/0) to the peering attachment which connected the vTGW to the TGW. From there they

HCX Service Mesh and Route Based VPN with default route advertised

I've been asked a few times around whether or not HCX can be used if the customer has a route-based VPN into VMware Cloud on AWS and is advertising the default route 0.0.0.0/0 into the SDDC. The short answer is Yes, this is supported and works.

The long answer as to why this question comes up is that when we advertise the default route of 0.0.0.0/0 into the SDDC then all traffic from the SDDC will flow via on-premises, this includes traffic destined for the internet. Some customers prefer to do this to ensure all outbound internet traffic routes via their perimeter firewall so they can ensure that all security and logging policies are applied. The confusion comes around HCX. Since HCX is unable to use an existing IPSEC VPN tunnel to send traffic from on-premises into the SDDC as per the KB article it needs to establish its own. The HCX-IX Interconnect and HCX Network Extension Appliances both establish an IPSEC VPN tunnel from on-premises to their peer appliances in the SDDC using UDP/4500. So the question is, if we are advertising 0.0.0.0/0 into the SDDC so all traffic traverses the IPSEC VPN tunnel back to on-premises, then how can the HCX-IX Interconnect and HCX Network Extension Appliances communicate?

HCX Mobility Optimized Networking Policy Routes

With the R145 release of HCX on 30th October 2020, VMware Cloud on AWS customers were treated to some great new functionality at no additional cost. New features such as Replication Assisted vMotion, Application Path Resiliency, TCP Flow Conditioning, Mobility Groups and my personal favourite, Mobility Optimized Networking. I'm not going to go into too much detail around Mobility Optimized Network (MON) since Patrick Kremer (Blog | Twitter) has covered it extensively here.

On an internal slack channel the following question was asked:

When we migrate a VM from on-premises into VMC that resides on a stretched layer 2 network without enabling MON, any traffic that needs to egress that network either destined to VMs on-premises, within VMC or out to the internet

Migrating back from Amazon FSx to on-premises file servers

In a previous article, I showed how AWS DataSync could be used to copy data from an on-premises file server (Either physical or virtual) into an Amazon FSx Windows file server. Amazon FSx is a fully managed, highly reliable, and scalable file storage that is accessible over the industry-standard Service Message Block (SMB) protocol. This is ideal for customers who are looking to migrate workloads to VMware Cloud on AWS and start taking advantage of native cloud services to improve resiliency and cost. When it comes to customers who want to utilise VMware Cloud Disaster Recovery to failover their environment into VMware Cloud on AWS but then failback to on-premises once the on-premises environment has been recovered things start to get a little tricky. AWS DataSync does not support copying data back from AWS (FSx, EFS or S3) to on-premises so this has to be performed manually. This post will showcase the steps required to copy the data back from Amazon FSx to on-premises which can be incorporated into your failback process to ensure that there is no data loss.

We first need to ensure we change the existing AWS DataSync task to either run manually or delete it completely to avoid having old data copied (If available) back into FSx which could potentially cause some inconsistencies. Navigate to the AWS DataSync service and either delete the task:

Or change the task schedule to Not Scheduled which means you have to start it manually:

Typically I would recommend stopping the share to 100% confirm

AWS DataSync via VPC Endpoints instead of Public Endpoints

In my previous article, I walked through the steps of replicating an on-premises file server into Amazon FSx using the AWS DataSync service. Since I didn't have a VPN between my on-premises and the VPC where the FSx service was deployed to I had to use the public endpoint whereby all communication from the DataSync agent to AWS occurs over the public internet. Within this article, I'm just going to quickly show you the process of setting this up using VPC Endpoints so communication goes over a VPN or Direct Connect directly into the VPC. This will allow for reverse replication which will be a topic for a future article. 

I currently the AWS DataSync agent deployed with a routable static IP address on-premises and a VPN established into my VPC. I first need to create a VPC endpoint in my VPC for the AWS DataSync Service. Ensure you are in the correct region and navigate to the VPC service. From within there, you will see the option to add Endpoints:

Search for the AWS DataSync service and ensure you select the correct VPC it needs to be created in. I've left the default options to

Utilising AWS DataSync and AWS Cloud Native Storage to migrate file servers to support workloads running in VMware Cloud on AWS

Whilst working with customers on the technical aspects of migrating their applications from on-premises into VMware Cloud on AWS most often the topic of file servers comes up. If the customer currently has their file servers running as virtual machines on top of ESXi then it's very straight forward, they can use HCX to migrate the workload. Depending on the size of the file server the customer may not want to do this due to potentially increasing the host count of the cluster and instead use some native AWS storage services such as FSx, EFS or even S3. Another scenario I have come across is customers having physical file servers running on storage arrays such as NetApp etc. 

During my investigation, I can across AWS DataSync which automates and accelerates the moving of data from on-premises into AWS storage services as well as between AWS storage services.

As of writing this article, the following services/protocols are supported:

• Amazon EFS file system (Source and Target)
• Amazon FSx for Windows File Server (Source and Target)
• Amazon S3 (Source and Target)
• Network File System (NFS) (Source Only)
• Object storage (Source Only)
• Server Message Block (SMB) (Source Only)

As an example, DataSync can be used to move data from an on-premises Windows file server into AWS FSx for Windows or from Amazon EFS into...

VMware Cloud on AWS Online vExperience Days

Are you interested in some free technical on-demand VMware Cloud on AWS training videos? If so, then the EMEA Solution Architects team have produced nine videos to guide ranging from introducing you to the service all the way through to automation and cloud economics:

1. Introduction to VMware Cloud on AWS and Use Cases

2. Deploy Your First VMware Cloud on AWS SDDC

3. Accelerate Your Cloud Application Migration with VMware HCX

4. Protect Your Data, Minimize Downtime and Reduce Costs with Optimized Disaster Recovery

5. Create Hybrid Applications with VMware Cloud on AWS and Native AWS Services

6. Day-2 Operations: Managing Your VMware Cloud on AWS Platform

7. Deploy and Configure Your Entire VMware Cloud on AWS with Automation

8. Application Modernization with VMware Cloud on AWS and Kubernetes

9. Cloud Economics: What is it and How it Can Help You to Accelerate Your Journey to the Cloud

Click on the link below to register and get started. 

Feel free to provide any feedback or if there is any additional content you would like to see.

HCX Central CLI Mind Map

In a previous article I introduced the HCX Central CLI (CCLI) as a tool to help troubleshoot HCX connectivity and performance issues. I've created a quick Mind Map of all the various commands that are available through CCLI to ease the troubleshooting process.

You can download a .pdf version of the full Mind Map here

This is correct as of HCX Version R144 (Build 16989452)

Troubleshooting HCX Connectivity and performance issue into VMware Cloud on AWS

When working with customers on VMware Cloud on AWS POC's or Pilots a lot of the success criteria typically includes using Hybrid Cloud Extension (HCX) to migrate workloads from on-premises into VMware Cloud on AWS either using bulk migration or live vMotion. This sometimes involves troubleshooting connectivity and performance issue so I figured I would show the typical process that I follow to try and narrow down and identify the issue.

HCX Service Mesh Appliances Tunnel Status

VMC Sizer now accepts RVTools and LiveOptics Inputs

When it comes to sizing customer environments who are looking to migrate from on-premises into VMware Cloud on AWS we always use the VMC Sizer website. The tool allows us to input and modify various parameters to correctly size the environment. Basic values that we need to gather from customers to size the environment are:

Number of VMs

Storage per VM

vCPU per VM

vRAM per VM

North East VMUG - Thursday 6th February 2020

The first North East VMUG of 2020 is ready for registration and boy is it going to be a good one. The event will take place on Thursday 6th February at the Royal Station Hotel which is situated right next to Newcastle Central Station. Once again, the team have put on a spectacular agenda with some great VMware speakers as well as community sessions. Special thanks to all the sponsors who fund these events and without them these events would not happen so during the breaks go spend some time with them and see what they have to offer:

Datrium
TruStack
10Zig
Zerto
Runecast

The current agenda is as follows:

VMware Keynote - Frank Denneman (Blog | Twitter) - Chief Technologist at VMware
Title - VMware’s Hybrid Cloud Vision and Strategy

In this session, Frank will discuss VMware’s 3-year Hybrid Cloud Vision and Strategy. He’ll discuss industry trends driving our thinking and then lay out our vision for how we can evolve our core platform to support where customers and the industry is going. re provides a lot of value, especially in large-scale Kubernetes deployments.

VMware Keynote - Duncan Epping (Blog | Twitter) - Chief Technologist at VMware 
Title - How HCI is revolutionizing the datacenter today and tomorrow! 

A year has passed and a lot has changed since then. Not just for you, but also for HCI and vSAN in particular. In this session, Duncan will discuss where we are coming from, but more importantly where we are going. Be warned, this session will include forward-looking statements and demos of to be released features.

Community Session - Kyle Jenner (Blog | Twitter) - AWS Architect at Rackspace
Title - VI admin to Cloud Architect

A little northerners journey to the cloud - tips and tricks learned along the way and thoughts on the future.

Community Session - Craig Dalrymple (Blog | Twitter) - Senior Systems Engineer at Brightsolid
Title - ...And now for something completely different

A frank and open presentation about mental health in IT, it is real and it really can happen to anyone ...did I mention there will be Irn Bru?

Community Session - Michael Sweeney (Twitter) - Virtualistion Specialist at DXC.Technology
Title - Modern Day Infrastructure Management using vRealize Ops

Overview and lessons learnt from a vROps deployment and what advantages this brought to make more informed business decisions.

VMware Session - Andy Watson (Twitter) - Systems Engineer at VMware
Title - Virtual Cloud Networking

Andy Watson, a Solution Engineer in VMware’s Network and Security Business Unit, gives an update on the Virtual Cloud Network. In the last 3 years, VMware has gone from a single networking product to providing complete consistent networking and security from the Datacentre to the Edge and the Cloud.

This is a great event to network with peers and experts from VMware as well as have a few sneaky beers since its no longer dry January :)

See you there

Michael

Protecting a 3-Tier App in VMC with the NSX Distributed Firewall

VMware Cloud on AWS SDDC v1.9 was officially released on the 16th January 2020 and includes a wealth of enhancements with regards to the NSX Distributed Firewall and Inventory Groups. The release notes for SDDC v1.9 can be found below if you want to check out all the new features that are available with this release:

https://docs.vmware.com/en/VMware-Cloud-on-AWS/0/rn/vmc-on-aws-relnotes.html

In this article, I am going to show you how to protect a simple 3-Tier Application using the NSX Distributed Firewall and some of the enhancements around Inventory Groups. A policy-based VPN has already been established from on-premises into VMC and the application has already been deployed into a single network segment as per the diagram below:

A policy-based VPN has been configured to allow the following traffic:

On-Premises to VMC
10.1.9.0/24 - 10.101.1.0/24
10.1.9.0/24 - 10.2.0.0/16

VMC to On-Premises
10.101.1.0/24 - 10.1.9.0/24
10.2.0.0/16 - 10.1.9.0/24

The on-premises firewall has been configured to allow all traffic between the network segments and an ANY - ANY - Allow rule has been configured on the Compute Gateway Firewall to allow all traffic into the 3-Tier App Semgemt. I could lock this down but I want to focus on the Distributed Firewall security aspects for this particular blog post:

Prior to configuring the distributed firewall, I check the application is working as expected by accessing the webserver from the on-premises management segment (10.1.9.0/24) to verify connectivity:

Since this is the only application that will currently be running in VMC and I want to ensure maximum security, I switch from a Blacklist security approach to Whitelist with logging. This a new feature of VMC and set's the default ANY - ANY rule to deny with logging rather than allow. When setting the security approach you have the following options:

Blacklist - This option creates a default ANY-ANY rule to allow all traffic. This is the default option for the distributed firewall.
Blacklist with logging - This option creates a default ANY-ANY rule to allow all traffic with logging enabled.
Whitelist - This option creates a default ANY-ANY rule to block all traffic. All communication is denied access including DHCP traffic.
Whitelist with logging - This option creates a default ANY-ANY rule to block all traffic with logging enabled. All communication is denied access including DHCP traffic.

The option can be changed in the Cloud Services Portal (CSP) under Networking & Security then Distributed Firewall:

I verify connectivity has been dropped by pinging my WEB, APP and DB servers as well as try to access the application via http://10.101.1.11

The Distributed Firewall rules that I want to implement are as follows:

I only want to allow SSH and ICMP from my on-premises management segment to the 3-Tier App for support/troubleshooting purposes. I then want to allow anything to talk to my webserver but only over HTTP (TCP port 80). My web server needs to talk to my app server also over HTTP and finally, the application servers communicates with the database server using the MySQL service (TCP port 3306). Since we have a whitelist security approach all other communication will be blocked thus preventing any lateral movement by any malicious hacker. An example of this would be a hacker compromising my web server and laterally trying to move, say over a known MySQL exploit, into the database server to extract any data.

I’m going to use NSX Inventory Groups as the basis for my DFW ruleset and dynamically add members to the groups based on tags, which is new with v1.9 of the SDDC. Using tags has the added benefit that if I need to scale out any portion of my application i.e. add more web servers then I can simply deploy the server, assign a tag to the server an automatically ensure the correct security posture is applied to the workload without having to modify any firewall rules. I’m going to start by creating the following Inventory Groups with the following membership criteria:

Web Servers - Dynamically add workloads to the group when a Web tag is applied to the VM
App Servers - Dynamically add workloads to the group when an App tag is applied to the VM
DB Servers - Dynamically add workloads to the group when a DB tag is applied to the VM
3-Tier App - This group will have static membership and I will add the Web Servers, App Servers and DB Servers group into it. Nested groups are also a new feature released as part of SDDC v1.9.
On-Premises Management - This group will have static membership which will include the CIDR range of the on-premises management segment (10.1.9.0.24)

To create an Inventory Group simply log into the Cloud Services Portal (CSP) and navigate to your SDDC. From here click on the Networking & Security, Groups and then Add Group:

I'm going to call this group Web Servers and give it a suitable description. Click on the Set Members link to set the membership criteria:

I now want to click on Add Criteria and specify Virtual Machine Tag Equals Web and click Apply:

Finally, I can click Save to commit the changes:

I now have my first Infrastructure Group which will dynamically add VMs whenever a Web tag is applied to them. I will now continue and create the App Servers and DB Servers groups based on the App and DB tags.

I'm now going to create my 3-Tier App group with the membership being the Web Servers, App Servers and DB Servers groups:

To nest groups select Members and check the groups that you want to be nested:

Finally, the last group that I need to create is my On-Premises Management group:

This group is going to be based off the CIDR range 10.1.9.0/24 so select the IP/MAC Addresses option and enter the range:

I now have all the required Infrastructure Groups created and can start building out my Distributed Firewall ruleset:

The Distributed Firewall has been completely overhauled as part of SDDC v1.9 with a ton of great new features. The interface has been updated and now includes rule categories which are evaluated based on priority precedence i.e. Emergency rules will be evaluated before Environment rules:

1 - Ethernet - Applied to all SDDC network traffic
2 - Emergency - Used for quarantine and allow rules
3 - Infrastructure - Define access to shared services. Global rules - AD, DNS, NTP, DHCP, Backup, Management Servers
4 - Environment - Rules between zones - production vs development, inter-business unit rules
5 - Application - Rules between applications, application tiers, or the rules between microservices

Configurations can now be saved and published at a later date/time (Maybe within a change window) and we can also fully exclude VMs from having DFW policies applied. These are just a few of the improvements, remember to check the release notes for more.

One of the huge usability improvements in SDDC v1.9 is the ability to dynamically re-prioritize rules by dragging and dropping them into place. You can also filter rules by name, source, destination or service and can edit rules inline on the UI. This makes creating and modifying DFW rules much more efficient.

We are going to create the required rules for our 3-Tier App and the first thing we need to do is create a new policy within the Applications section. The name of the policy can be modified inline:

Click on the three dots to the right of the rule and click Add Rule:

Give the rule a suitable name, in our example, we want this rule to allow Admin access to the 3-Tier App from the On-Premises Management Network for troubleshooting and support purposes:

Within the Source set it to the On-Premises Management Infrastructure Group:

Set the Destination to be the 3-Tier App group:

Now the services we want to allow are SSH and ICMP (If the service you require is not available you can add custom services):

We can see that our admin access rule is in place but since we have not published the ruleset this is currently not live:

I'm now going to create the complete ruleset for the application based on the following requirements:

Once the ruleset has been created remember to publish it otherwise it will not be realised within NSX.

The final task is to tag our VMs with the correct tags which will ensure they are added to the corresponding infrastructure groups and the correct security policy should be applied. Navigate to the Virtual Machines section and edit one of the VMs:

Tag the VM with the required tag. In our example, this would be either Web, App or DB:

Ensure all VMs in scope are tagged accordingly:

You can check to ensure the VMs have been tagged correctly and they appear in the security group by viewing the group members:

My Web Servers group now has WEB01 as a member:

Using groups and tags has the operation benefit of not having to make any changed to rules if I scale up the application. If I add a new Web Server all I simply have to do it tag it with the Web tag (Ideally through automation) and the correct security policies will be applied to the workload from day one.

Now the Distributed Firewall policies have been created and the VMs tagged let's check to ensure my application is working as expected. If I browse to the Web Server I can see that I can successfully connect to the 3-Tier Application:

From my desktop which is in the On-Premises Management CIDR range, I can successfully ping all workloads:

I can also successfully SSH into the Web Server but notice that I cannot SSH or ping from the Web Server to the Database Server:

VMware Cloud on AWS SDDC v1.9 is an update packed with new features and functionality and just shows the power of the platform in helping customers migrate and secure their workloads with familiar tooling.

Monitoring VMware Cloud on AWS vCenter alarms within vRealize Log Insight Cloud

vRealize Log Insight Cloud (vRLIC) gives us unified visibility across public and private clouds through robust log aggregation, analytics and faster root cause determination. The great news is that it is also included as part of your subscriptions to VMware Cloud on AWS (VMC) and as of VMworld Europe 2019 also now includes additional features and functionality:

Check out the official blog article here

With the core version, VMC customers now get access to real-time reporting which is what we are going to look at today. In a previous post, I talked using creating VMC vCenter alarms and setting notifications for specific events. The event we were particularly interested in was if the VSAN datastore reaches 70% utilisation because at 75% a new host will be added to ensure we stay within SLA. In the example, we used an alert that would trigger if the datastore was less than 100% utilised as this would ensure the alert would always trigger. We are now going to use vRLIC to query for the alert and then send us a notification once it has been triggered.

vRLIC is automatically configured to ingest logs for VMC and can be accessed via the Cloud Services Portal so there is nothing that you need to do to start using it, simply launch the application:

The initial landing page gives us a great overview of recent alerts and event observations over the last hour. It is definitely worth spending some time with vRLIC to see the level of information and default alerts that are available to VMC customers:

I've re-created the vCenter alert that triggers when VSAN Datastore Usage is below 100 percent just to ensure the log is sent instantly to vRLIC:

If we explore the logs and query for the alert name VSAN Datastore Usage is below 100 percent with a timeframe of the last ten minutes then we can see the triggered alert. We know this is the alert because we can see it change its state from Gray to Red:

Now that we have the query needed we can click on the save icon:

Give the query a suitable name and description and click Save:

Once we save the query we can click on the alert icon to create an alert based on the query:

Give the alert a suitable name and description and click Save:

The Alert Definition screen will appear which will allow you to customise the alert. Remember to add the Email address where you would like the alert to be sent, set the trigger to evaluate on every match and enable it before clicking on the save icon:

It's also worth sending a test alert to ensure you receive the notification:

Hopefully, if everything is set up correctly next time the alert is triggered you should receive a notification via email and also see it in the Recent Alerts:

With the example above we are triggering an email notification when we see any log that has been ingested and contains the text, VSAN Datastore Usage is below 100 percent. This is not ideal because it will also trigger the notification when any changes to the alarm are made i.e. reset to green or disabling and re-enabling. I tried testing this on alarm name as well as the text gray to red which is sent when the state of the alarm changes but during testing, I noticed that this was not always sent on certain alarm configuration changes which I have fed back to the BU and will be addressed in the future. I don't envisage these changes being made regularly in customer environments so it should not cause an influx of emails.

A point that I would like to highlight is that vRLIC currently runs out of one of the US AWS regions so if there are issues with logs residing outside of the UK/EU then please get in touch and I will continue to raise this internally.