AWS Simple Email Service (SES) Domain Verification in BIND zone file

Recently I had to verify a domain for use with AWS SES.  Creating the initial Domain Verification request is simple in the AWS Console and if your domain is in Route53 then Domain Verification is automatic.  However, the domain I needed to verify was running on DNS servers we manage ourselves (I work for a Service Provider) and these run BIND.  So here is a quick copy / paste format for the entry in your zone file once you create the Domain Verification.

Create the AWS SES Domain Verification request:-

aws_ses_bind_1Grab the TXT verification details:-

aws_ses_bind_2Now go to your DNS server, edit the zone file and insert the new record in the following format (note the ” marks around the record):-

_amazonses      IN      TXT     "n5vae4N9fHBcIpdoSwFaxJx85Rmbv9L8unt27xd8pEM="

To check your domain you can either run nslookup or dig (my personal preference is for dig). I also like to check DNS propagation and so bounce the query off an external DNS server (usually Google public DNS servers).

Dig method:-

dig @8.8.8.8 -t TXT _amazonses.mydomain.com

NSlookup method

nslookup -type=TXT _amazonses.mydomain.com 8.8.8.8

Google DIG Tool Method (https://toolbox.googleapps.com/apps/dig/):-

https:[email protected]8.8.8

After some time some when DNS has propagated a background AWS automated process will check your domain and then verify.  At which point it should look like this:-

aws_ses_bind_4

vSphere 6 Setting up vFlash Read Cache

This example is being performed on a Dell R630 with the Perc RAID controller running ESXi 6.  I didn’t do anything special with the Flash disk in the RAID controller, I simply created a RAID 0 volume group with the single drive.

A lot of the inspiration for this fix is from this Yellow Bricks post.

Once the Host has ESXi installed, network configured, booted up, added to vCenter (and all the extra gubbins done), you will first need to mark the disk as a Flash disk …… ESXi won’t pick this up automatically.

vsphere_6_vflash_read_cache_1Manage > Storage > Storage Devices > Highlight disk and click the F button to mark as Flash.

vsphere_6_vflash_read_cache_2Now right-click the ESXi host > Storage > Add Virtual Flash Resource Capacity.

At this point I ran into a problem.  As far as I can see you should simply be able to add the Flash drive as Virtual Flash Resource Capacity, however, by default it was greyed out and isn’t selectable……

vsphere_6_vflash_read_cache_3…… and it only becomes selectable if you click the “Enable remote flash selection” tickbox.

vsphere_6_vflash_read_cache_4When you click OK, vCenter tells you it has completed, however, there are no Flash Datastores available for use and I was getting the following error under the ESXi Host > Monitor > Events:-

Configuration on disk /vmfs/devices/disks/naa.614187704de041001e622fd406f5f19e failed. Reason : A specified parameter was not correct: naa.614187704de041001e622fd406f5f19e

This fix for this is to tag the disk from esxcli, so next SSH into your Host and run

esxcli storage vflash device list

You will see that ESXi has detected this disk as a remote SAS SSD.

vsphere_6_vflash_read_cache_5If you try and add the disk as local (as per the comments in the Yellow Bricks post) you will get the following error:-

esxcli storage nmp satp rule add -s VMW_SATP_LOCAL -d naa.614187704de041001e622fd406f5f19e -o enable_local

Error adding SATP user rule: Duplicate user rule found for SATP VMW_SATP_LOCAL matching device naa.614187704de041001e622fd406f5f19e PSP and PSP Options

vsphere_6_vflash_read_cache_6So to resolve this you will need to remove the rules, readd them correctly, run reclaim and then (back in the Web Client) retag as flash.

esxcli storage nmp satp rule remove -s VMW_SATP_LOCAL -d naa.614187704de041001e622fd406f5f19e
esxcli storage nmp satp rule add -s VMW_SATP_LOCAL -d naa.614187704de041001e622fd406f5f19e -o enable_local
esxcli storage core claiming reclaim -d naa.614187704de041001e622fd406f5f19e

Once this is done you should be able to add flash correctly.

vsphere_6_vflash_read_cache_7

vCenter 6 Appliance (vCSA) ESXi Net Dump Collector configuration

Login to your vCenter as [email protected] (or use an account which has been put into the “Administrators” group from Home > Administration > Single Sign-On > Users and Groups > Groups).

Now go to Home > Administration > Deployment > System Configuration > Services > VMWare vSphere ESXi Dump Collector

vCSA_6_net_dump_1Now right click the service and tick Automatic so the services comes up at boot time.

vCSA_6_net_dump_2Now right click the service and hit start.

Now to set your hosts to use the Dump Collector on the vCSA its probably easier to do via PowerCLi.  The post below has a simple and quick way to do this:-

vSphere Dump / Syslog Collector: PowerCLI Script

So in powershell, you can grab the setting for all hosts in a datacenter like this:-

Foreach ($ESXi_host in (get-vmhost -Location My-DC)) {
 $esxcli = Get-EsxCli -VMHost $ESXi_host
 Write-Host "Getting Net Dump config for $ESXi_host"
 $esxcli.system.coredump.network.get()
}

And to set the Dump Collector, you can do something like this:-

Foreach ($ESXi_host in (get-vmhost -Location My-DC)) {
  $esxcli = Get-EsxCli -VMHost $ESXi_host
  Write-Host "Setting Net Dump config for $ESXi_host"
  $esxcli.system.coredump.network.set($null, “vmk0”,

One thing I did notice was that if you go to to the vCenter > Manage > Settings and click on ESXi Dump Collector I got an “ESXi Dump Collector Service is not running. Enable ESXi Dump Collector and refresh”.

vCSA_6_net_dump_3

I double checked that the service is running in the GUI and via the appliance shell:-

vCSA_6_net_dump_4Even a reboot didn’t fix it!

If / when I find a fix for this I will update this post …..

 

 

vRealize Automation Windows Blueprint Administrator password set (via gugent)

Here is a very simple way (i.e. non Orchestrator workflow route) for allowing your users to define their own passwords for a Windows VM when they request it via the catalogue.  This method uses the vRA Guest Agent.

An excellent walk through on vCAC 6.2 of how to do this for Linux VMs (and the main inspiration for this post) can be found here:-

http://www.virtualvcp.com/vmware-vrealize-automation-vcac/207-vrealize-automation-6-2-specifying-a-new-root-password-using-custom-properties

A good post on how to prepare your Linux and Windows Templates for the vRA Guest Agent (gugent) operations can be found here:-

vRA7 – Gugent on Linux

vRA7 – Gugent on Windows

So assuming you have all the pre-reqs in place you then need to create a small batch file on your Windows template in say c:\scripts.

My .bat file is:-

c:\scripts\vCACSetAdministratorPassword.bat

…. and the contents of the script (where %1 is the argument you pass from the Blueprint Request form):-

echo off
net user administrator %1

You now need to create the Property Group, in vRA 7 this is in Administration > Property Dictionary > Property Groups.

Create a new group with the following script actions:-

Name Value Encrypted Overridable Show in Request
VMware.VirtualCenter.OperatingSystem windows7Server64Guest No Yes No
VirtualMachine.Customize.WaitComplete true No Yes No
VirtualMachine.Software0.Name Generate new root password No Yes No
InitialRootPassword No Yes No
VirtualMachine.Admin.UseGuestAgent true No Yes No
VirtualMachine.Software0.ScriptPath c:\scripts\vCACSetAdministratorPassword.bat {InitialRootPassword} No Yes No

When you’re done it should look like this:-

vra7_windows_gugentNow add your new Property Group to the Blueprint:-

vra7_windows_gugent_1

Once you have saved the Blueprint, try a deploy and see if it works.  If you are having problems then the first place to look is the gugent log files which by default are in:-

C:\VRMGuestAgent\GuestAgent.log

post upgrade vRA 7 orchestrator endpoint won’t connect

I upgraded vCAC 6.2 to vRA 7 the other day. Everything looked good until I tested some blueprints which call custom workflows configured in the embedded Orchestrator.

I had my endpoint configured as per the documentation, listed here:-

http://pubs.vmware.com/vra-70/topic/com.vmware.vrealize.automation.doc/GUID-9E9E2393-A268-432C-B9EA-2CBEFA25361B.html#GUID-9E9E2393-A268-432C-B9EA-2CBEFA25361B

You can test if your Endpoint is working or not by either running a refresh or checking the logs. To force the endpoint to run a data collection login to vCAC as a Infrastructure Admin and go to Infrastructure > Endpoints > Endpoints, then select Data Collection and click Start to force a data collection run.vRA_7_Orchestrator_Problems_1

To check the logs go Infrastructure > Monitoring > Log and filter on Message for “endpoint”:-vRA_7_Orchestrator_Problems_2

As you can see I am getting the following error:-

Endpoint Data Collection failed for endpoint vCO-Embedded [Workflow Instance Id=588727]
Unable to connect to the remote server
Inner Exception: Unable to connect to the remote server

I checked the DEM log on my IaaS server. By default the logs can be found here:-

C:\Program Files (x86)\VMware\vCAC\Distributed Execution Manager\<DEM Worker>\Logs

And found the following errors:-

Endpoint Data Collection failed for endpoint vCO-Embedded [Workflow Instance Id=588727]
System.Net.WebException: Unable to connect to the remote server —> System.Net.WebException: Unable to connect to the remote server —> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 172.22.4.61:8281

I tried a telnet from the IaaS machine and there is no connection available on port 8281.  So a quick check with the client via:-

https://<vcac app fqdn>/vco/client/client.jnlp

and you are presented with the auto filled java client session, which points at 443 rather than 8281:-

vRA_7_Orchestrator_Problems_4

So I edited the Endpoint, removed 8281 and ran the Data Collection again and it now works fine!

Conclusion

At the time of writing it looks like the VMware documentation is out of date but in short remove 8281 from your endpoint for the Embedded vCO and it should work fine.

I’ve left all the working out gumpf in this post on the off chance it helps others…..

vSphere 6 Web Client TIPs and TRICKs

This post is just a collection of various TIPs and tricks to make the VMware vSphere web client more bearable.

To be fair to VMware it sounds like they are working hard to make the web client better and as of 6.0 U1 and 5.5 U3 there are some big improvements.  This video from last years VMworld runs through the various changes they have made and are planning (INF5093):-

However, until the HTML5 version comes out this page will be here as a collection of as many tips as possible…….

Adobe Flash bitching about storage?

http://kb.vmware.com/kb/2109770

Want to see all users tasks in the web client (be warned this can have a knock on performance wise on busy setups).

http://kb.vmware.com/kb/2104914

Client integration plugin not working in FF (i.e. can’t upload files etc)

http://kb.vmware.com/kb/2125623

If you have more feel free to comment and I will update this page……

vRA 7 Migration Tool

If you’re about to upgrade (or already have) to vRA 7.x and want to make sure your logins all work post upgrade then you will need to use the Migration Tool to shift your ID stores from the old SSO setup (whether this was a dedicated ID appliance or integrated with vCenter SSO etc) to the new vIDM system. If you don’t migrate the logins then you will have to create the SSO ID Stores manually which also requires creating a local vIDM user in the Tenant(s).

A good explanation of why VMware have moved from SSO to vIDM can be found here:-

vRealize Automation 7 – Part 5, Identity Management

How you perform the migration relies on which SSO method you were using for your 6.x system.  For example if you were running of a Windows based 5.5 vCenter SSO service you will need to run the migration tool on the Windows server that is hosting it.  Similarly if you were running a dedicated vCAC 6.x ID appliance you will need to run the linux version from the appliance.

In this example we will be running this migration from a dedicated appliance and as always make sure you have backups / snapshots etc before you go any further!!!

First off you will need to obtain the migration tool zip file URL:-

https://vra-app1.domain.local:5480 > vRA Settings > SSO and copy the migration tools URL.

SSH to your old SSO ID appliance (SSH can be enabled from https://<SSO ID Appliance IP>:5480 > Admin), cd /tmp, download and extract the zip:-

ssh root@172.22.4.60
cd /tmp
wget https://vra-app1.domain.local:5480/service/cafe/download/vra-sso-migration.zip --no-check-certificate
unzip vra-sso-migration.zip
unzip vra-sso-migration.zip -d vra-sso-migration

vRA_7_vIDM_Migration_1

Now cd to the new dir and set your environment:-

cd vra-sso-migration/bin/
. setenv

Now migrate your ID Stores (you will be prompted at various points for passwords and confirmations):-

./migrate-identity-stores

vRA_7_vIDM_Migration_2

The script claimed to add everything but it wasn’t 100% clean. For some reasons it failed to sync my groups and when I got in to take a look it turns out the Group DN was missing.  I didn’t look too deep into this as the previous step gave me enough access to start putting my AD logins back in the right place!

SSO [email protected] account expiry

The other day I hit the “Associated user’s password is expired” when trying to login to my SSO as the [email protected] account.

You can just reset the password for the account as per VMware KB 2035864. However, on vSphere 5.1 this causes some confusion over the SSO user password and the so called master password (which never changes) – see this communities post for more info.

A quick and dirty fix for this (if you are running the vCenter with a SQL DB) follows.

  1. Take a backup of your RSA DB (if you don’t and you trash your DB then don’t complain to me :)).
  2. Open SQL Server Management Studio, expand the RSA DB, expand the Tables folder and find the dbo.IMS_AUTHN_PASSWORD_POLICY table.
  3. Right click and select Edit Top 200 Rows.
  4. Now edit the MAX_LIFE_SEC column (this is in seconds), so for example if you want to set it to 5 years it would be 157680000 (apparently you can set this to 0 for never expire).  I’m setting mine to 90000000 (1014 days).

  1. Restart the SSO service.
  2. Log back into the Web Client as [email protected]
  3. Go to Administration, Configuration, Policies tab.  It should now look like this:-

vCenter Operations Manager – Monitoring vAPP Health Status

Overview

Need to monitor a vCOPS vAPP’s general Health so you are aware of problems with vCOPS itself?  Well, hopfully this post will help.

There are a number of options for monitoring vCOPS, self-monitoring, using vCenter or using an external monitoring system (like Xymon or Nagios).

Continue reading “vCenter Operations Manager – Monitoring vAPP Health Status”