vHutch PowerCLI Daily Check Script – The usual suspects plus a few bells and whistles…

Another daily check script?

I’ve recently been heavily involved in a vSphere Metro Storage Cluster (vMSC) design / implementation / migration project for a customer, which has presented quite a few automation opportunities.  I’ve written 4 or 5 scripts during this project, the most recent being a “Daily Check” script for the environment.  I know there are lots of these already knocking around on the internet but I wanted to pull something together of my own.  I had to find a way to add value while progressing workload migrations somehow!  There are a few unique aspects, nothing too crazy and hopefully it will be of some use – regardless of whether you use the script as is or take elements for your own creations.

 

The script generates a simple tabular HTML file output which optionally can be emailed via SMTP which is fairly standard stuff.  What I have done is added some conditional formatting, which was interesting to say the least.  Many thanks to my buddy Dean Emerson of ComputerCrashers.co.uk fame for suggesting this, which encouraged me to go back to something I’d deemed unachievable (with CSS at least) earlier in the week.  In the end this was achieved using the ConvertTo-HTMLConditionalFormat function which is published on the MS Script Centre.  The function was originally written by Joel Bennett (@Jaykul) on a StackOverflow post so thanks very much to Joel for this.  I had to hack about and get creative with RegEx to get the function to pass me a HTML “fragment” back rather than saving a file, but I got there.

 

The script has been written with PowerCLI 6.3 R1.  While I don’t believe I’m using many of the latest and greatest features I’d recommend using this version (or 6.5) if you choose to run the script as is.  Similarly it was written targeting a vSphere 6.0 U2 environment, I’ve also tested it against a vCenter 5.5 U2 deployment successfully.  Just bear this in mind before you start.  I’ll hopefully get chance to test against vSphere 6.5 at weekend and will edit the post to confirm once I do.

 

EDIT: 10/04/17 – Successfully tested with vSphere 6.5b and ESXi 6.5a.

Summary of content

The script output includes data for ESXi Hosts, Datastores, VMs and Snapshots.  I’ll go on to break this down in more detail but first a few screenshots….

NOTE:  The screenshots were taken after running the script in a lab.  That’s just me justifying the 18.4:1 vCPU:pCPU ratio 😉

Screenshot 1 - Showing the initial sections

Screenshot 2 - Showing the VM section
Screenshot 3 - Showing the final sections

Screenshot 4 - Showing the output when running the script within a PowerShell console window

So a fair amount of data is included I’m sure you’ll agree, all generally pretty useful as proactive notifications go.  The list below summarises the highlights:

  • ESXi Hosts
    • Core host data
    • Powered On VM Quantity
    • vCPU Count
    • vCPU:pCPU Ratio – useful for determining how subscribed or indeed oversubscribed a host’s CPU resources are
  • ESXi Host Alarms
    • Basic data for any triggered host alarms
  • Datastores
    • Core datastore data
    • Used (%)
    • VM Count
    • Number of connected hosts
  • Virtual Machines
    • Core VM Data
    • VMFS “Master” Datastore – essentially the location of the VMX file which in most “ordinary” cases will also be the datastore where the VM’s VMDK’s live
    • Snap Count
    • Total size of a VM’s snaps in GB
    • Whether Orphaned Snaps are present – this is quite useful and works based on a little string manipulation and comparison with the depth of the snap chain as known by vCenter, vs those actually reflected in each VMDK’s name
    • An average of the Highest Latency performance metric over the last 24hr
    • Whether a CD Drive is connected
    • What type of CD Drive is connected, if an ISO include the path
  • Virtual Machine Snapshots
    • Core Data
  • Virtual Machine Guest Partition Usage Exceptions
    • Guest disk partition utilisation status where this exceeds a certain used percentage, specified via an input parameter
    • VM, Disk Path (i.e. drive letter or mount point on Linux), Capacity/Free/Used (GB) and of course Used (%)

Just before we get to the “big reveal”…

I’m not going to talk through all 640 lines but before I post the code itself a few notes on the script’s inputs and bits you may wish to tweak.

Input Parameters

  • OutFolder – The folder where the output .html file will be written.  Filename is auto-generated and includes a timestamp.
  • VIServer – The vCenter Server IP/FQDN
  • OutRetention – How many .html files will be retained, once this number is reached (if specified) the oldest file will be removed by the script  (optional)
  • GuestPartUtilThreshold – The percentage that should trigger the inclusion of Guest disk partitions as exceptions
  • CustomisationText – A descriptor (e.g. company name) that will be included in the output header and generated email
  • SMTPServer – SMTP Server IP/FQDN  (optional)
  • SMTPTo – SMTP address to email the file to  (optional)
  • SMTPFrom – SMTP sender address to send the email as (optional)

 

NOTE:  All SMTP parameters must be provided for the script to generate an email.

 

Conditional formatting tweaks

The script has a couple of aspects that I expect will need to be modified for some environments.  These are centred on thresholds for highlighting cells with conditional formatting.  I’ve built-in formatting for the following metrics and summarise the default values, along with what you need to do to amend them if required:

  • Host CPU/RAM Usage (%)
    • Defaults at greater than or equal to 80% = Yellow, 90% =Red
    • Modify the value of $CF_VMH_CPU_Y / $CF_VMH_CPU_R for CPU, for example
  • Host vCPU:pCPU Ratio
    • Defaults at greater than or equal to 4:1 = Yellow
    • Modify the value of $CF_VMH_vCPUpCPU_Y
  • Datastore Usage (%)
    • Defaults at greater than or equal to 75% = Yellow, 85% = Red
    • Modify the value of $CF_DS_UTIL_Y / $CF_DS_UTIL_R
  • VM Snapshot Count
    • Defaults at greater than or equal to 1.0 = Yellow, 3.0 = Red
    • Modify the value of $CF_VM_SNAPCOUNT_Y / $CF_VM_SNAPCOUNT_R
  • VM Snapshot Size (GB)
    • Defaults at greater than or equal to 2GB = Yellow, 5GB = Red
    • Modify the value of $CF_VM_SNAPSIZE_Y / $CF_VM_SNAPSIZE_R
  • VM Has Orphaned Snaps
    • Defaults to Red if snaps are present – these guys are BAD so don’t change this one!
  • VM Latency
    • Defaults at greater than or equal to 3.1ms = Yellow, 10ms = Red (you can tell the vMSC is on top of an All-Flash array.  These numbers were being conservative too!)
    • Modify the value of $CF_VM_LATENCY_Y / $CF_VM_LATENCY_R
  • VM CD Drive Connection
    • Default is to highlight Yellow if connected
    • Update within the code, due to there being no foreseeable need to amend this behaviour easily
  • VM Tools
    • Default to Yellow if old, Red if not running/not installed
    • Modify the value of $CF_VM_VMTOOLS_Y / $CF_VM_VMTOOLS_R to change when the Tools Status results are formatted
    • NOTE:  Don’t lose the triple apostrophes!

 

A word about the decimals…

So after getting the HTML Conditional formatting function mentioned earlier doing what I needed to, I hit a problem when using the same threshold value for multiple metrics when generating formatted HTML.  This was because the function was written to use a hashtable to pass the formatting “rules” in/out.  Of course hashtables cannot have multiple entries with the same name (duh!)  As the threshold value and criteria was being passed via this attribute I had to find a way to workaround this duplication.

One simple way was to make a slight differentiation with the values, which was easily achievable by using decimals.  This is why you see things like 80.0 and 80.1 values in the script below.  This could have been fixed by re-engineering how values were passed in/out of the function but this simply didn’t seem worth the time effort at the point where I hit the issue.  If any readers have the inclination to resolve this properly feel free to do so and post in the comments for everyone (myself included) to benefit from 🙂

The main event…

Finally he’s got to the script itself I hear you say!  After me knocking out around 1200 words of pre-text here we are…  Feel free to copy/download/butcher, whatever you like.

 

I really hope some (or all) of this script is useful to somebody.  If you have any questions please write in the comments or get in touch via Twitter.

 

Bye for now

 

Chris

Leave a Reply

Your email address will not be published. Required fields are marked *

*