Monitoring VMware Cloud on AWS vCenter alarms within vRealize Log Insight Cloud

vRealize Log Insight Cloud (vRLIC) gives us unified visibility across public and private clouds through robust log aggregation, analytics and faster root cause determination. The great news is that it is also included as part of your subscriptions to VMware Cloud on AWS (VMC) and as of VMworld Europe 2019 also now includes additional features and functionality:

Check out the official blog article here

With the core version, VMC customers now get access to real-time reporting which is what we are going to look at today. In a previous post, I talked using creating VMC vCenter alarms and setting notifications for specific events. The event we were particularly interested in was if the VSAN datastore reaches 70% utilisation because at 75% a new host will be added to ensure we stay within SLA. In the example, we used an alert that would trigger if the datastore was less than 100% utilised as this would ensure the alert would always trigger. We are now going to use vRLIC to query for the alert and then send us a notification once it has been triggered.

vRLIC is automatically configured to ingest logs for VMC and can be accessed via the Cloud Services Portal so there is nothing that you need to do to start using it, simply launch the application:

The initial landing page gives us a great overview of recent alerts and event observations over the last hour. It is definitely worth spending some time with vRLIC to see the level of information and default alerts that are available to VMC customers:

I've re-created the vCenter alert that triggers when VSAN Datastore Usage is below 100 percent just to ensure the log is sent instantly to vRLIC:

If we explore the logs and query for the alert name VSAN Datastore Usage is below 100 percent with a timeframe of the last ten minutes then we can see the triggered alert. We know this is the alert because we can see it change its state from Gray to Red:

Now that we have the query needed we can click on the save icon:

Give the query a suitable name and description and click Save:

Once we save the query we can click on the alert icon to create an alert based on the query:

Give the alert a suitable name and description and click Save:

The Alert Definition screen will appear which will allow you to customise the alert. Remember to add the Email address where you would like the alert to be sent, set the trigger to evaluate on every match and enable it before clicking on the save icon:

It's also worth sending a test alert to ensure you receive the notification:

Hopefully, if everything is set up correctly next time the alert is triggered you should receive a notification via email and also see it in the Recent Alerts:

With the example above we are triggering an email notification when we see any log that has been ingested and contains the text, VSAN Datastore Usage is below 100 percent. This is not ideal because it will also trigger the notification when any changes to the alarm are made i.e. reset to green or disabling and re-enabling. I tried testing this on alarm name as well as the text gray to red which is sent when the state of the alarm changes but during testing, I noticed that this was not always sent on certain alarm configuration changes which I have fed back to the BU and will be addressed in the future. I don't envisage these changes being made regularly in customer environments so it should not cause an influx of emails.

A point that I would like to highlight is that vRLIC currently runs out of one of the US AWS regions so if there are issues with logs residing outside of the UK/EU then please get in touch and I will continue to raise this internally.

VMware Cloud on AWS vCenter Alarms

A lot of VMware customers use vCenter alarms and notifications for monitoring their on-premises environment and the same goes for when they move to VMware Cloud on AWS. I was recently asked by a customer on how they can receive a notification when the VSAN storage capacity is getting close to 75% full. For those who are not aware we need 25% slack space for VSAN and will automatically add a host once storage utilisation reaches 25%, which is documented in the Service Level Agreement for VMware Cloud on AWS.

Creating an alarm can either be completed directly in the vCenter client or via the Cloud Gateway Appliance. Simply browse to the WorkloadDatastore, select Configure and then Alarm Definitions. From here you need to Add a new alarm:

Give the alarm a suitable Name and Description and click Next:

In the example, I am setting the alarm to be triggered if the utilization is less than 100% (Which will always be the case) to ensure that I receive a notification. Typically you would set this to is above 70% or whatever threshold you feel comfortable with. Once you have the correct parameters enter the email address to whom you would like the notification sent to and click Next:

Set the email notification if you want to be notified once the condition clears otherwise click Next:

Review your settings and click Next when ready:

Since our alarm was set to trigger if storage utilisation was less than 100% we can see that it triggered straight away within vCenter:

We also received an email notification:

Since currently there is no sender address there is a chance this might get picked up by your email spam filtering software so you might have to create a rule to allow it through. Due to this annoyance in the next article, I will show you how to trigger an alert from vRealize Log Intelligence Cloud.

Creating a Universal Logical Switch in NSX 6.2 and vMotioning a VM from one vCenter to another

Now that we have our Universal Transport Zone that spans clusters across multiple vCenter servers it's time to create a Universal Logical Switch, attach two virtual machines to it and then vMotion one from the Palo Alto vCenter to the New York vCenter.  One thing to remember when creating any universal objects is that they need to be created on the NSX Manager that has the primary role.  If you try to do it whilst connected to an NSX Manager with the secondary role you will not see any Universal Transport Zones.  Log into the vSphere Web Client and navigate to Logical Switches within the Network & Security tab and create a new Logical Switch whilst ensuring you are connected to the NSX Manager with the primary role:

Once the switch has been created you will notice that it's VNI number falls inline with the range we specified within the Universal Segment ID Pool:

If you switch across to the NSX Manager with the secondary role you will also see the same Universal Logical Switch created with the same VNI number:

If we check the VDS portgroups you should see the newly created portgroup in both vCenters:

I have the following two VM's which I am going to attach to the Universal-Web-Tier-01 Logical Switch and power on ensuring that are both in the same vCenter:

WEB01 - 192.168.1.11

WEB02 - 192.168.1.12

As you can see we have layer 2 connectivity between the two virtual machines:

Now time to vMotion WEB02 from the Palo Alto vCenter (PA-VCSA-01) to the New York vCenter (NY-VCSA-01).  Right click on the VM and select Migrate:

Select the option to Change Compute Resources only and click Next (The same storage is available to both vCenters / hosts in my lab):

Click on the clusters tab and select the cluster you want to migrate the VM to and click Next.  In my example I am moving WEB02 from the PA-Compute cluster in PA-VCSA-01 to the NY-Compute cluster in NY-VCSA-01:

Select the folder to place the VM into and click Next:

Select the network you want to attach the VM to in the other vCenter.  In my example I'm connecting it to the same Universal Logical Network.  It would be nice for this to have been automatically selected for me:

Set the appropriate priority and click Next:

Verify your setting, click Next and watch the magic happen:

WEB02 has now been vMotioned from the Palo Alto vCenter across to the New York vCenters with no drop in connectivity:

The possibilities for this are endless especially around Disaster Recovery with Site Recovery Manager.

Installing NSX 6.2 within a multi vCenter Environment

With the release of NSX 6.2 we now have the ability to span Transport Zones, Logical Networks, Distributed Logical Routers (DLR's) and even firewall rules across multiple vCenters.  In this post I'm going to show you how to install NSX 6.2 in a multi vCenter environment.  In my lab at home I have two physical ESXi hosts each running four nested ESXi hosts to simulate a Palo Alto and New York Datacenter.  All hosts are running vSphere 6.0 and there are two vCenters in linked mode with the Platform Services Controller running on the Palo Alto vCenter. The screen shot below shows the configuration via the vSphere Web Client:

I'm not going to show you how to deploy NSX Manager as this has been very well documented by numerous bloggers. I'd recommend Giuliano Bertello's (Blog | Twitter) NSX for newbies series.  So I've deployed two NSX managers called PA-NSX-01 and NY-NSX-01 and associated them with the corresponding vCenters:

PA-NSX-01 is configured to use PA-VCSA-01 for it's lookup service and also it's vCenter server:

NY-NSX-01 is configured to use PA-VCSA-01 for it's lookup service and NY-VCSA-01 as it's vCenter server:

Remember when using vCenter 6 the lookup service port is now 443 instead of 7444.

After logging into the Web Client you will now see that you can manage multiple NSX Managers:

I've deployed a single controller in the Palo Alto management cluster as this is just a test lab:

I'm also going to prepare all 4 clusters and configure the host VTEP's:

Palo Alto Management

Palo Alto Compute

New York Management

New York Compute

You will notice that you now have the ability to select the NSX Manager that you want to perform tasks on via the NSX Manager menu:

I've now prepared all four clusters:

I'm now going to configure PA-NSX-01 to be the primary NSX Manager and NY-NSX-01 to be a secondary.  Navigate back to Installation and then the Management tab and highlight the NSX Manager that you want to assign the primary role to, click Action and then Assign Primary Role

Click Yes to confirm the operation:

Keep the primary NSX manager highlighted and click Actions and then click Add Secondary NSX Manager:

Ensure the IP address is correct for the secondary NSX Manager and enter the Admin credentials for the secondary NSX Manager:

Once finished we should have Primary and Secondary NSX Manager instances and you should see the secondary instance using the NSX Controller from the primary site:

Within NSX 6.2 there are four NSX Manager Roles:

Standalone - Default option for new installation or upgrade. All objects in a Standalone NSX Manager are local.

Primary - One NSX Manager per Cross VC NSX deployment is assigned the Primary role. Deploys and manages the Universal Controller Cluster and any Universal objects.

Secondary - The Secondary role is assigned when adding a Standalone NSX Manager to a Primary. Universal objects from the primary are synchronised to the Secondary where they are read-only.

Transit - Temporary role used when an NSX Manager has a Primary or Secondary Role removed. If universal objects still exist the NSX Manager is assigned a Transit role until these objects are deleted then it can move to a Standalone role.