#vcops

vRealize Operations (formerly known as vCenter Operations Manager) is an indispensable tool for monitoring a VMware environment. The upgrade process between recent versions is riddled with pitfalls, however. Here, I will cover the process and make notes of all the gotchas so that you can have as smooth of an experience as possible.

The upgrade process goes from 5.x -> 5.8.1+ -> 6.0.x -> 6.1.

Upgrading from 5.x to 5.8.5

Prerequisites and Common Pitfalls:

You should change your admin password before attempting the upgrade. New security policies that come with this upgrade will expire older passwords for admin. It must be a complex password that you have not used before. http://kb.vmware.com/kb/2013358

You should ensure that you have a good amount of free disk space on the UI and Analytics VMs. You can check this with “df -h” on the command line.

Appliances that were originally deployed as 5.6 and earlier had a smaller root partition that may get filled by the update. If you run into any issues caused by this, you can follow http://kb.vmware.com/kb/2074688

If the /data partition is >85% full on either VM, you should add a disk and reboot. Boot-time scripts will handle data volume expansion onto new disks for you.

Steps to Upgrade:

Take a snapshot of both VMs.

Follow the instructions in the release notes for the upgrade at https://www.vmware.com/support/vcops/doc/vcops-585-vapp-release-notes.html

After ensuring that you can get in to the UI, delete the snapshots.

Migrating data from 5.8.1+ to 6.0.x

Prerequisites and Common Pitfalls:

You must have version 5.8.1 or higher. The data migration will be more reliable with version 5.8.5, but is known to work with all versions from 5.8.1+. The best instructions for upgrading are in the release notes.

You must have forward and reverse DNS entries in place for your source 5.8.1+ VMs, as well as for every node in your new 6.0.x cluster.

Please size your nodes accordingly per the handy Excel spreadsheet attached to the bottom of http://kb.vmware.com/kb/2130551

When naming and addressing your version 6 nodes, please note:

All nodes of a vROps 6 cluster (with the exception of remote collectors) must be on the same physical LAN (>1MS latency will cause problems) and must not be separated by a firewall.

You should change your admin password on 5.8.1+ before attempting the upgrade. An expired password is common, sometimes difficult to identify, and will cause vague errors. It must be a complex password that you have not used before. http://kb.vmware.com/kb/2013358

If you suspect that performance may be an issue, stopping DT calculation and new data collection on the 5.8.1+ appliance will improve the speed of the data migration greatly. http://kb.vmware.com/kb/2040008

Although it is not supposed to be an issue anymore, there are some cases where DNS resolution does not work properly and this KB may still be necessary: http://kb.vmware.com/kb/2100944

Node and host names can not have underscores

Node names and IP addresses cannot be changed after installation

The roles of a node are subject to change, so naming a node according to a role may get confusing in the future.

Steps to upgrade:

Install the latest version of 6.0.x per the documentation at http://pubs.vmware.com/vrealizeoperationsmanager-6/index.jsp

After you bring your cluster online, you will be presented with a wizard. Don’t select the option for importing data or entering a license key yet. Even though you are going to import data and probably have a license key, it is more reliable to do these things after the setup wizard in my experience.

Go to Administration>Solutions and select the tab called “Import Data”

Follow the prompts for importing data.

Upgrading from 6.0 to 6.1

Prerequisites and Common Pitfalls:

There may be a malformed upgrade file from previous versions. You should remove this before upgrading. http://kb.vmware.com/kb/2120616

Previous updates, patches and add-on Management Packs may cause issues if their files still linger in your update queue. To avoid issues with this, you should log in to each node via console or SSH and move the files in “$STORAGE/db/casa/pak/dist_pak_files/” somewhere else.

Large database blobs from a previous data migration may choke the update process. http://kb.vmware.com/kb/2132479

If you have remote collectors that are separated by high-latency links, the update may time out when the master node pushes the update out to them. You can pre-stage the update pak files to work around this: http://kb.vmware.com/kb/2127895

The cluster state files may have some malformed data in them. I have a script that can fix this and many more issues athttps://github.com/nakedhitman/vrops-cluster-repair

Steps to upgrade:

Take a snapshot of all your nodes.

Install the OS Update pak file

Install the Application Update pak file.

Log in to the product UI as admin and ensure that your dashboards, adapter instances, and other data are present and working as expected.

Once you’re sure that everything is running OK, delete the snapshots.

Ensure that you check both the boxes so that the “Reset out of the box content” option is selected. If you do not do this, some parts of the system may not be upgraded. This will not affect custom dashboards or other content, only the ones that ship with the system that you oughtn’t to have changed anyway ;) If you have customizations to the built-in content, you can clone them to preserve your changes.

If it makes you go through the first run setup wizard again: don’t panic, its probably fine. Just choose evaluation mode and complete the wizard. Everything should be there when it’s complete. http://kb.vmware.com/kb/2132452

If you end up with vCenter adapter instances that cannot be configured, missing dashboards or licenses, or end up loosing historical data, you should revert your snapshot and tell VMware about it immediately. If you don’t have a support contract, then make some noise on the communities site. VMware pays attention to this stuff and wants to prevent it from happening.

Conclusion

VMware vCenter Operations data mining

*Warming – This is most likely totally unsupported and not how the product is intended to be used!

I’ve been working on a couple projects with vCOps recently (NO SPOILERS YET!) and came across something that was totally new to me that I wanted to share.   No internal secret sauce here either.  I found this via KB 2011714.

FYI – I’m not an expert on licensing, but I assume only the highest level…

You will get free self paced training from VMware , when you opt for VMware vSphere with Operations Management promotion

VMware has given a new dimension to the hypervisor world by introducing Operations management.  With VMware vSphere with Operations…

A couple of issues I ran into this week with vCOPs that are worth mentioning. One has to do with the infamous grey question mark. This replaces a badge when data collection is broke. The other has to do with pruning data collection in an environment where you are licensed for less per VM than what is being managed by a single vCenter instance. Yes a bit long winded, but let me explain…

First off the most noticeable tip off to vCOPs not collecting data is the grey question mark. In times of normalcy, each badge should be colored with a number indicating, visually and numerically, that it's reading and making sense of the data. If it's not there are a couple things that could contribute to this. First, disk space on the analytics VM, lack of..breaks ActiveMQ. VMware uses conservative guidelines as follows in regards to disk sizing, but keep in mind you can add disk space to the Analytics VM, albeit with a shutdown, at anytime.

A bit of background, ActiveMQ is an open source message broker that writes to a database called kahadb. Here it stores and forwards to other services. If the service can't accept the messages, it writes it to this database. If the disk underflowth ActiveMQ is unable to write a complete file, there in potentially corrupting the database. To resolve this you can follow the steps in VMware KB 2013266. The process is pretty straight forward which haves you rename the kahadb and create a new one. However, it doesn't state which default account to use to do so. I mistakenly used root (as opposed to admin) to make the changes which prevented the ActiveMQ service (which uses admin) from writing to the kahadb. Simply changing the permissions ( chown -R admin:admin /data/activemq/kahadb/)on the database resolved the issue.

Secondarily, unexpected shutdowns. As with any vApp make sure to shutdown, or restart the VM's within the vApp at the vApp level. This will save you a lot of trouble down the line and keep that kahadb in working order.

Understanding data collection in vCOPs tends to be a mind bender. There are multiple associated terms that are used to illustrate this process. Terms like attributes, metrics, super metrics, thresholds, and KPI's is enough to make you question your own self worth. But with a few helpful explanations maybe we can break new ground together…

First off we all know what resources are in a data center environment. These are items such as datastores, VM's, vSS's, vDS's, vCPU's, pCPU's, etc. They are items we consume or that have value. Data that is ingested within vCOPs has an associated attribute(s) for individual resources. By default vCOPs groups together multiple attributes for resources into attribute packages. By doing so you are telling vCOPs to collect only these attributes for this resource.

A metric is a point in time instance of an attribute. If a single metric doesn't clearly convey what you are looking for, then perhaps a super metric would. A super metric takes this metric, that metric and that other metric applies a mathematical operation (formula) to give you a broader more scalable metric. For instance, if you wanted to capture the average CPU utilization across 50 web servers you would use a super metric to do so.

Thresholds mark the boundary between what’s normal and what’s abnormal for a single metric or super metric. When either boundary is crossed an anomaly is logged. Both hard and dynamic thresholds exist, with the latter being the default. Dynamic thresholds are new to vCOPS and are based on incoming and historical data. It formulates a pattern of normalcy, there in decreasing the number of alerts that are generated. Hard thresholds model themselves after vCenter thresholds. Static in nature, changing only when you change it.

Finally, KPI's or Key Performance Indicators are attributes you deem important. By defining an upper and/or lower threshold violation on a attribute or super metric package, you are in essence establishing a KPI for those attributes. Once an attribute is defined as KPI, new rules are set forth in motion. For instance, alerting is treated differently than non-KPI attributes. But more importantly a feature called predictive alerting is utilized. Once a violation occurs on a KPI for a application or tier, vCOPs examines the events prior to the violation and marks that as a fingerprint. If it finds similar events in the future it can alert you prior to the thresholds being broken. How sick is that!

Here is an evolving graphic to illustrate the relationships. It’s not pretty, sorry…More to come… 

I'm recalling that it was tricky to find documentation on this, so for those people who are running out of space on vCenter Operations Manager's Analytics VM, it's pretty easy (though not intuitive) to add more space:

Shut it down the entire vCenter Operations Manager vApp

Add a new (as in separate, additional) VMDK to the Analytics VM

Boot the vApp back up

The Alanytics VM will discover the new disk and append it to its current working volume.  If you extend the existing VMDK, there is no way inside the Analytics VM to extend the volume manually (prove me wrong, VMware!), so you'll just strand that extra disk.