I ran into a rather unpleasant problem today where we had a service window to do a vCAC 6.0.1.1 upgrade first to 6.1 and then to 6.2. We did an upgrade just before Christmas in the test environment, which also had some challenges along the way. You can read more about this here.

So in the process of upgrading the vCAC IAAS Component (last component) to 6.1 version, we experienced this error in installer.

Distributed Execution Manager <name of DEM> cannot be upgraded because it points to a Management Model Web host <IaaS Web site>:443 which cannot be validated. You must resolve this error before running the upgrade again: Model Manager Web Service is installed on host <IaaS Web Server>:443 but it is not up and running. HTTP Web response status code: InternalServerError.

I did a quick search on Google and I found this KB article on exactly this error.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2091053

However it was unfortunately not too much help ☹

After verifying all credentials used for service account and access to the databases we opened a ticket for VMware support. I was lucky to get Frank Büchsel on the line to help troubleshooting this error.

After troubleshooting different log files, we found that the issue was regarding the repository service not running and that it actually has stopped working 2 days prior to we started this upgrade. This was not so nice, as our fallback plan was to restore just before we started upgrading in case of issues. So this would not help us now, unless we wanted to roll back to before the issue started. This was the errors we found in the Model Manager Web log files:

[UTC:2015-01-10 14:16:47 Local:2015-01-10 15:16] [Error]: Failed to start repository service. Reason: One or more errors occurred.
[UTC:2015-01-10 14:16:59 Local:2015-01-10 15:16] [Error]: Failed to start repository service. Reason: One or more errors occurred.
[UTC:2015-01-10 14:17:11 Local:2015-01-10 15:17] [Error]: Failed to start repository service. Reason: One or more errors occurred.
[UTC:2015-01-10 14:17:23 Local:2015-01-10 15:17] [Error]: Failed to start repository service. Reason: One or more errors occurred.
[UTC:2015-01-10 14:17:35 Local:2015-01-10 15:17] [Error]: Failed to start repository service. Reason: One or more errors occurred.
[UTC:2015-01-10 14:17:47 Local:2015-01-10 15:17] [Error]: Failed to start repository service. Reason: One or more errors occurred.
[UTC:2015-01-10 14:17:59 Local:2015-01-10 15:17] [Error]: Failed to start repository service. Reason: One or more errors occurred.

As you can see, the reason in the above log was not very descriptive and didn’t help much.

As the error message during the installer indicated that it was a problem for the DEM to validate that the repository service was not running, Frank suggested uninstalling the DEM components manually. Frank had the idea if we removed the DEM service the installer would continue. After an uninstall of both the DEM and DEO – the upgrade installer continued and caused a much more descriptive error in the vCACInstaller log that led Frank to the root cause of the problem.

[12-01-2015 15:19:45]EXEC : <title>Internal Server error (500)<br>Request: <br>POST https://servername:7444/sts/STSService/vsphere.local<br>Response:<br>&lt;?xml version=&quot;1.0&quot; ?&gt;&lt;S:Envelope xmlns:S=&quot;http://schemas.xmlsoap.org/soap/envelope/&quot;&gt;&lt;S:Body&gt;&lt;S:Fault xmlns:ns4=&quot;http://www.w3.org/2003/05/soap-envelope&quot;&gt;&lt;faultcode xmlns:ns0=&quot;http://docs.oasis-open.org/ws-sx/ws-trust/200512&quot;&gt;ns0:FailedAuthentication&lt;/faultcode&gt;&lt;faultstring&gt;Invalid credentials&lt;/faultstring&gt;&lt;/S:Fault&gt;&lt;/S:Body&gt;&lt;/S:Envelope&gt;</title> [C:\Program Files (x86)\VMware\vCAC\Server\Model Manager Data\DeployRepository.xml]

In the vSphere Web Client, we found that the application user (com.vmware.csp.iaas.user-unique-id) created by vCAC had a certificate that was expired. A time bomb issue unfortunately. Btw – the vCAC installer creates this application user automatically when it registers vCAC to SSO. And that certificate expires after 1 year!

Unfortunately I did not screendump the error we found on the certificate, but this is how it should be with all application users valid and green.

application_user

Now we found the problem, time to fix the issue.

Again Frank was quick to find the procedure to fix this. Follow the steps in this article

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2101390

A huge thanks to Frank Büchsel from VMware support. You are in good hands if you get him on the phone.