vCAC 6.x Error: "HTTP Error 503. The service is unavailable" under the Infrastructure tab

Recently after a network outage in my lab, all my machines were restarted. I started getting the service not available messages everywhere under the Infrastructure tab, when I brought my vCAC infrastructure online. I am documenting the steps I followed to fix this issue, you may jump to the final resolution part. I hope these troubleshooting steps would certainly help for other similar issues.

Issue: Any selection under Infrastructure tab show error- HTTP Error 503. The service is unavailable.

Catalina.out log @ /var/log/vcac/ on vCAC Server shows below errors

2014-12-28 01:36:00,036 vcac: [component="cafe:component-registry" priority="WARN" thread="mainTaskExecutor-48" tenant=""] com.vmware.vcac.platform.rest.RestTemplateIpv6.handleResponseError:581 - GET request for "https://lab-vcac-iaas.ps.local/WAPI/api/status" resulted in 503 (Service Unavailable); invoking error handler

2014-12-28 01:38:00,044 vcac: [component="cafe:component-registry" priority="WARN" thread="mainTaskExecutor-44" tenant=""] com.vmware.vcac.platform.rest.RestTemplateIpv6.handleResponseError:581 - GET request for "https://lab-vcac-iaas.ps.local/WAPI/api/status" resulted in 503 (Service Unavailable); invoking error handler

2014-12-28 01:40:00,046 vcac: [component="cafe:component-registry" priority="WARN" thread="mainTaskExecutor-35" tenant=""] com.vmware.vcac.platform.rest.RestTemplateIpv6.handleResponseError:581 - GET request for "https://lab-vcac-iaas.ps.local/WAPI/api/status" resulted in 503 (Service Unavailable); invoking error handler

2014-12-28 01:42:00,045 vcac: [component="cafe:component-registry" priority="WARN" thread="mainTaskExecutor-31" tenant=""] com.vmware.vcac.platform.rest.RestTemplateIpv6.handleResponseError:581 - GET request for "https://lab-vcac-iaas.ps.local/WAPI/api/status" resulted in 503 (Service Unavailable); invoking error handler

2014-12-28 01:44:00,109 vcac: [component="cafe:component-registry" priority="WARN" thread="mainTaskExecutor-50" tenant=""] com.vmware.vcac.platform.rest.RestTemplateIpv6.handleResponseError:581 - GET request for "https://lab-vcac-iaas.ps.local/WAPI/api/status" resulted in 503 (Service Unavailable); invoking error handler

2014-12-28 01:46:00,029 vcac: [component="cafe:component-registry" priority="WARN" thread="mainTaskExecutor-38" tenant=""] com.vmware.vcac.platform.rest.RestTemplateIpv6.handleResponseError:581 - GET request for "https://lab-vcac-iaas.ps.local/WAPI/api/status" resulted in 503 (Service Unavailable); invoking error handler

2014-12-28 01:48:00,044 vcac: [component="cafe:component-registry" priority="WARN" thread="mainTaskExecutor-47" tenant=""] com.vmware.vcac.platform.rest.RestTemplateIpv6.handleResponseError:581 - GET request for "https://lab-vcac-iaas.ps.local/WAPI/api/status" resulted in 503 (Service Unavailable); invoking error handler

Possible Resolutions:

I am listing out the steps I followed to fix this issue, maybe any of the below step might your issue as well.

  

1. Make sure all the vCAC services are up on IaaS Server by going to services.msc.
2. Make sure the vcac server service is running on vCAC appliance. Confirm it by running

service vcac-server status

If it is stopped, then start or restart it by below command

service vcac-server start

3. Microsoft Hotfix for Server 2008 R2 which could help in case if vCenter server is throwing the above error (refer Vmware KB - kb.vmware.com/kb/2033822)

http://support.microsoft.com/kb/2577795

4. Make sure all the services in IaaS are running with the same account or maybe local system account if appropriate.

Below is the step which worked in my case

1. Open - Internet Information Services (IIS) Manager and Select the Application Pool under the root node.

2. Make sure the DefaultAppPool service is running. Try restarting if it is still running.

3. If this doesn’t resolve the issue then make sure these services associated with vCAC are running with the same account as in services.msc
   

1. RepositoryAppPool
2. vCACAppPool
3. WapiAppPool

In my case the above 3 services were stopped. After starting these services, everything was back to normal.