This has been a busy week starting with installing of the newly released vRealize Operations 6.0 the other day. Read my blog about the sizing guidelines here. The last two days have been used lifting a vCloud Automation Center (vCAC) environment from 6.0.1 release to vRealize Automation (vRA) 6.2 release. As this gave us a couple bumps on the way, I wanted to share this info with you all. First, I’m sorry for the long blog article, but I would recommend reading it, to avoid any of these problems I ran into.

I will not cover the exact step-by-step procedure in this article, but just share the lessons learned and some recommendations.
Prerequisites:
Before upgrading your environment I would recommend you to read the release notes of each product upgrade and also the installation and upgrade documentation. For your convenience I have links to the documentation here below:

vCloud Automation Center 6.1 documentation

vCloud Automation Center 6.2 documentation

First thing to do, is to determine the update sequence. In this environment the vCAC environment had dependencies to the following other VMware products:

  • vSphere ESXi and vCenter Server
  • vShield Networking and Security (vCNS)
  • vCenter Orchestrator appliances

The update sequence I used was then:

  1. Upgrade vShield Manager to latest release
  2. Install/Upgrade or Configure vCenter Orchestrator to latest release
  3. Install the NSX plugin for vCenter Orchestrator (as its needed for vCNS)
  4. Optional: Upgrade vCenter Server Appliance to latest release
  5. Optional: Upgrade ESXi hosts
  6. Upgrade vCAC Café appliance to 6.1
  7. Upgrade the vCAC Database to 6.1 using the dbupgrade script
  8. Upgrade the vCAC IAAS server to 6.1
  9. Test and verify everything works

If you are using vShield Networking and Security within your vCAC environment – vCAC 6.1 requires vCenter Orchestrator configured and installed with the VMware vCenter Orchestrator Plugin for NSX 1.0.0. If you do not install this, the data collection towards the vCNS Manager will fail. This is unfortunately not very well documented. At least I couldn’t find any information about it.

Before starting the upgrade procedure, take a backup / snapshot or similar. I did a full shutdown of all the components to do an offline snapshot and also performed a copy-backup of the vCAC Database. This provides me with a fallback procedure.

Important note: There is no-direct upgrade path from 6.0.x to 6.2. So you have to perform an upgrade to 6.1. After you verify that the vCAC 6.1 is fully functional you can then perform the upgrade to vRA 6.2

If you have been installing vCAC 6.0 you will probably remember that .Net 4.5.1 was unsupported. But for the upgrade to vCAC 6.1 to work you have to install .Net 4.5.1 before performing the upgrade of the IAAS component. And again before upgrading to vRA 6.2 you have to do another upgrade to .Net 4.5.2. (PS. I have NOT verified if .Net 4.5.2 will work with vCAC 6.1 and I would just recommend you to stick with 4.5.1 for that upgrade).

Secondly according to the documentation, its required to install 64bit Java JRE 1.7 or later. I installed the latest release 1.8u25 on the IAAS server.

Finally after installing Java software, you have to create an environment variable for the path to the bin directory. Follow the documentation:

Before running the upgrade script make sure you are running 64-bit Java version 1.7 or later.
Environment variable JAVA_HOME is set to the Java install folder. (The %JAVA_HOME%\bin\java.exe file is available.

Almost done with the prerequisites, however one final thing. Make sure the MSDTC is configured accordingly, both on the IAAS server – but also on the SQL server hosting the vCAC database. Follow this KB article. If you do not follow this you may see the following error after upgrade when doing a data collection in vCAC:

Error processing workflow creation Error executing query usp_SearchInitializingRequestVirtualMachines Inner Exception: Error executing query usp_SelectGroup
DataBaseStatsService: ignoring exception: Error executing query usp_SelectAgent Inner Exception: Error executing query usp_SelectAgentCapabilities
stack trace at DynamicOps.VMPS.DataAccess.ConnectionFactory`1.ExecuteReader(RecordReader`1 recordReader, String storedProcedure, String whereClause, Boolean firstRecordOnly, SqlParameter[] collection) at DynamicOps.VMPS.DataAccess.DataAccessLayer.SearchInitializingRequestVirtualMachines(Int32 resultCount) at DynamicOps.VMPS.DataAccess.DataAccessLayer.<>c__DisplayClass181.

Before upgrading the vCAC Appliance, follow the procedures in these two links:

Shut Down Services on the vCloud Automation Center Appliance
Shut Down vCloud Automation Center Services on Your IaaS Windows Server

The above links is for vCAC 6.1 but also applies to 6.2 upgrade

 

Issues found after the vCAC 6.1
The upgrade procedure for vCAC 6.1 went very well, but we found some issues with MSDTC not configured properly as described earlier.

I also found that the data collection on the vCNS Inventory failed with the following error

Workflow ‘vSphereVCNSInventory’ failed with the following exception: Could not reach vCenter Orchestrator endpoint {0}. Trying the next highest priority endpoint

The first time I saw this error was because I didn’t install the VMware vCenter Orchestrator Plugin for NSX 1.0.0. Today I saw the same error, but this time it was the credentials I used, that couldn’t be authorised in vCO. Make sure the account you use, is authorised. Verify it by connecting to the vCenter Orchestrator using the Orchestrator client and the account specified in vCAC.

After fixing these errors, data collection ran without problems and I could both provisioning multi-machine blueprint fenced using vCNS Edges and decommission them again to verify functionality.

Now it was time to upgrade to vRA 6.2 and I thought that now we had the process figured out and documented, so I was not expecting any issues with this final upgrade.

 

Issues found during the upgrade to vRA 6.2
Following the same procedure as above, except some of the component had been lifted to latest release, so these steps were of course skipped here.

The upgrade of the vCAC Café 6.1 appliance to vRA Café appliance 6.2 took a little bit of time, but went successful without problems.

After this I had to upgrade the database, exactly the same way we did the 6.1 upgrade, just using a new dbupdate script downloaded from the upgraded vRA Café appliance. However this kept failing with the following error:

Build started 18-12-2014 13:17:58.

Project “C:\Temp\DBUpgrade\Liquibase.xml” on node 1 (default targets).

RunLiquibase:

“C:\Program Files\Java\jre1.8.0_25\\bin\java.exe” -Djsse.enableCBCProtection=false -Djava.library.path=”C:\Temp\DBUpgrade” -cp “C:\Temp\DBUpgrade\liquibase.jar;C:\Temp\DBUpgrade\jtds-1.3.1.jar” liquibase.integration.commandline.Main –driver=net.sourceforge.jtds.jdbc.Driver –url=jdbc:jtds:sqlserver://servername/”dbname”;instance=instancename;ssl=request releaseLocks

Liquibase releaseLocks Failed: liquibase.exception.DatabaseException: java.sql.SQLException: Network error IOException: Connection reset

C:\Temp\DBUpgrade\Liquibase.xml(28,5): error MSB3073: The command “”C:\Program Files\Java\jre1.8.0_25\\bin\java.exe” -Djsse.enableCBCProtection=false -Djava.library.path=”C:\Temp\DBUpgrade” -cp “C:\Temp\DBUpgrade\liquibase.jar;C:\Temp\DBUpgrade\jtds-1.3.1.jar” liquibase.integration.commandline.Main –driver=net.sourceforge.jtds.jdbc.Driver –url=jdbc: jtds:sqlserver://servername/”dbname”;instance=instancename;ssl=request releaseLocks” exited with code -1.

Done Building Project “C:\Temp\DBUpgrade\Liquibase.xml” (default targets) — FAILED.

Build FAILED.

“C:\Temp\DBUpgrade\Liquibase.xml” (default target) (1) ->

(RunLiquibase target) ->

C:\Temp\DBUpgrade\Liquibase.xml(28,5): error MSB3073: The command “”C:\Program Files\Java\jre1.8.0_25\\bin\java.exe” -Djsse.enableCBCProtection=false -Djava.library.path=”C:\Temp\DBUpgrade” -cp “C:\Temp\DBUpgrade\liquibase.jar;C:\Temp\DBUpgrade\jtds-1.3.1.jar” liquibase.integration.commandline.Main –driver=net.sourceforge.jtds.jdbc.Driver –url=jdbc: jtds:sqlserver://servername/”dbname”;instance=instancename;ssl=request releaseLocks” exited with code -1.

0 Warning(s)

1 Error(s)

Time Elapsed 00:01:46.79

Searching Google did get alot of hits about connectivity issues to the SQL server, either the SQL port was not right, a firewall was blocking or other network connectivity errors. None of which helped. And considering I used the same command as I did during the vCAC 6.1 upgrade – it was pretty confusing that this part was not working.

A community forum post led us on the right track again, where some had reported problems using Java JRE 1.8u25. So we uninstalled that version and installed a version 1.7.0.71-b14 and reattempted the same dbupgrade command again. And this time it went successful!

Update: I got a message from Ingemar Fällman, that he removed ssl=request from the Liquibase.xml to get Java JRE 1.8u25 to work.

Final part was to upgrade the IAAS component to vRA 6.2 and this part also went without any problems and now we had a running vRealize Automation 6.2 environment. I verified data collection and provisioning after upgrade.

 

Summary of lessons learned

  • Take backup/snapshots of all virtual machines and databases
  • Read the release notes for every upgrade
  • Read and follow the Installation and upgrade guide, but do NOT use Java version 1.8u25. Stick to version 1.7.0.71-b14.
  • Set the Environment variable JAVA_HOME according to documentation
  • Install the VMware vCenter Orchestrator Plugin for NSX 1.0.0, if using vShield Networking and Security in vCAC / vRA.
  • Correct any MSDTC issue on IAAS / SQL servers following this KB article
  • Verify that credentials is working and authorised within vCenter Orchestrator
  • Install the correct .Net version 4.5.1 for vCAC 6.1 and .Net 4.5.2 for vRA 6.2

I hope this will help you avoiding upgrading your environment. Feel free to reach out if you found the above information not correct or needs updated.

Happy upgrading!