Issues faced while upgrading vCloud Automation Center 6.1 to vRealize Automation 6.2

The complete upgrade process seems simple  but I am sure many would not agree with me. I was a bumpy ride to upgrade 3 components, below the flow diagram for upgrade of vCloud Automation Center 6.1 to vRealize Automation 6.2. The upgrade for Identity Appliance went well without a single error. Mine was a minimal installation, there are some additional steps depending on your distributed install. The actual issues started when I started the upgrade of vCAC Server Appliance.

Check the step by step upgrade here.

Upgrading vCAC Server Appliance:

The procedure is pretty simple, where you just need to back up some configuration files, stop the services, check for updates and then install it. I was using the CD-ROM option to install the updates. The very first unusual issue I encountered was the "Check Updates" option, which remained in "Checking for available updates…" forever. There was no option to stop this process from looking for new any updates. Restarting the vCAC services, un-mounting the CD-ROM or even restarting of vCAC appliance was of no help. After reboot the status would still show "Checking for available updates…"

Solution:

This is what you could do to come out of this loop. Click Here

Well, this wasn’t a good symptom for upgrade, as it would hardly take few seconds to find any new Updates(like vRA 6.2) when you click on Check Updates. Now whenever I tried the above workaround to come out of loop and check again, it would again go into the same loop. Luckily after 3-4 hours it did detected the vRA 6.2 and when clicked on Install Updates it again into the same loop and failed. There wasn’t much in the vami logs.

Solution:

This is what you could do to install the updates(click here), make sure you backup config file and "ovf-env.xml" file. During my troubleshoot I happened to reboot vCAC appliance couple of times and then I Found that the "ovf-env.xml" file is empty. This was obviously failing the upgrade process while running the post-install scripts and ultimately failing the Install Updates process.

After the above solution and a reboot, the vCAC appliance was upgraded to vRA 6.2 (vRealize Appliance). I had spent more than 2 days to fix these unusual issues and thought  it would certainly help others save time if they encounter these issues.

Upgrading IaaS Windows Server:

01: Upgrade Database

 Before you start with the upgrade process, make sure you meet below two requirements

1. Java version 1.7 or higher (64-bit) must be installed, the environment variable JAVA_HOME must be set to the Java install folder, and %JAVA_HOME%\bin\java.exe must exist.

2. Micosoft .NET Framwork is installed.

This is the first step of IaaS Server upgrade process and it kept failing everytime from local and remote machines. Dbupgrade logs shows below error

INFO: Current release database version: 6.2.0.9574

Executing script >>> C:\Users\administrator.PS\Desktop\db\vCAC.ChangeLog-Master.xml

Microsoft (R) Build Engine version 4.0.30319.18408

[Microsoft .NET Framework, version 4.0.30319.18408]

Copyright (C) Microsoft Corporation. All rights reserved.

Build started 1/14/2015 11:25:52 AM.

Project "C:\Users\administrator.PS\Desktop\db\Liquibase.xml" on node 1 (default targets).

RunLiquibase:

  "C:\Progra~1\Java\jdk1.8.0_20\bin\java.exe"  -Djsse.enableCBCProtection=false -Djava.library.path="C:\Users\administrator.PS\Desktop\db" -cp "C:\Users\administrator.PS\Desktop\db\liquibase.jar;C:\Users\administrator.PS\Desktop\db\jtds-1.3.1.jar" liquibase.integration.commandline.Main --driver=net.sourceforge.jtds.jdbc.Driver --url=jdbc:jtds:sqlserver://localhost/"vcacdb";ssl=request releaseLocks

  Liquibase releaseLocks Failed: liquibase.exception.DatabaseException: java.sql.SQLException: Network error IOException: Connection reset

 

C:\Users\administrator.PS\Desktop\db\Liquibase.xml(28,5): error MSB3073: The command ""C:\Progra~1\Java\jdk1.8.0_20\bin\java.exe"  -Djsse.enableCBCProtection=false -Djava.library.path="C:\Users\administrator.PS\Desktop\db" -cp "C:\Users\administrator.PS\Desktop\db\liquibase.jar;C:\Users\administrator.PS\Desktop\db\jtds-1.3.1.jar" liquibase.integration.commandline.Main --driver=net.sourceforge.jtds.jdbc.Driver --url=jdbc:jtds:sqlserver://localhost/"vcacdb";ssl=request releaseLocks" exited with code -1.

Done Building Project "C:\Users\administrator.PS\Desktop\db\Liquibase.xml" (default targets) -- FAILED.

Build FAILED.

"C:\Users\administrator.PS\Desktop\db\Liquibase.xml" (default target) (1) ->

(RunLiquibase target) ->

  C:\Users\administrator.PS\Desktop\db\Liquibase.xml(28,5): error MSB3073: The command ""C:\Progra~1\Java\jdk1.8.0_20\bin\java.exe"  -Djsse.enableCBCProtection=false -Djava.library.path="C:\Users\administrator.PS\Desktop\db" -cp "C:\Users\administrator.PS\Desktop\db\liquibase.jar;C:\Users\administrator.PS\Desktop\db\jtds-1.3.1.jar" liquibase.integration.commandline.Main --driver=net.sourceforge.jtds.jdbc.Driver --url=jdbc:jtds:sqlserver://localhost/"vcacdb";ssl=request releaseLocks" exited with code -1.

    0 Warning(s)

    1 Error(s)

Solution/Workaround:

Check the complete solution here.

  

After this rest of the installation went well and I hope these notes would help you guys while you are running upgrade in your environment and save some time.  I hope this was informative!! 

Dbupgrade fail during vCAC IaaS Server upgrade - Error MSB3073. Upgrade from vCAC 6.1 to vRA 6.2

Check the complete step by step upgrade procedure here.

Before you start with the upgrade process, make sure you meet below two requirements

1. Java version 1.7 or higher (64-bit) must be installed, the environment variable JAVA_HOME must be set to the Java install folder, and %JAVA_HOME%\bin\java.exe must exist.

2. Micosoft .NET Framwork is installed.

This is the first step of IaaS Server upgrade process and it kept failing everytime from local and remote machines. Dbupgrade logs shows below error

INFO: Current release database version: 6.2.0.9574

Executing script >>> C:\Users\administrator.PS\Desktop\db\vCAC.ChangeLog-Master.xml

Microsoft (R) Build Engine version 4.0.30319.18408

[Microsoft .NET Framework, version 4.0.30319.18408]

Copyright (C) Microsoft Corporation. All rights reserved.

Build started 1/14/2015 11:25:52 AM.

Project "C:\Users\administrator.PS\Desktop\db\Liquibase.xml" on node 1 (default targets).

RunLiquibase:

  "C:\Progra~1\Java\jdk1.8.0_20\bin\java.exe"  -Djsse.enableCBCProtection=false -Djava.library.path="C:\Users\administrator.PS\Desktop\db" -cp "C:\Users\administrator.PS\Desktop\db\liquibase.jar;C:\Users\administrator.PS\Desktop\db\jtds-1.3.1.jar" liquibase.integration.commandline.Main --driver=net.sourceforge.jtds.jdbc.Driver --url=jdbc:jtds:sqlserver://localhost/"vcacdb";ssl=request releaseLocks

  Liquibase releaseLocks Failed: liquibase.exception.DatabaseException: java.sql.SQLException: Network error IOException: Connection reset

 

C:\Users\administrator.PS\Desktop\db\Liquibase.xml(28,5): error MSB3073: The command ""C:\Progra~1\Java\jdk1.8.0_20\bin\java.exe"  -Djsse.enableCBCProtection=false -Djava.library.path="C:\Users\administrator.PS\Desktop\db" -cp "C:\Users\administrator.PS\Desktop\db\liquibase.jar;C:\Users\administrator.PS\Desktop\db\jtds-1.3.1.jar" liquibase.integration.commandline.Main --driver=net.sourceforge.jtds.jdbc.Driver --url=jdbc:jtds:sqlserver://localhost/"vcacdb";ssl=request releaseLocks" exited with code -1.

Done Building Project "C:\Users\administrator.PS\Desktop\db\Liquibase.xml" (default targets) -- FAILED.

Build FAILED.

"C:\Users\administrator.PS\Desktop\db\Liquibase.xml" (default target) (1) ->

(RunLiquibase target) ->

  C:\Users\administrator.PS\Desktop\db\Liquibase.xml(28,5): error MSB3073: The command ""C:\Progra~1\Java\jdk1.8.0_20\bin\java.exe"  -Djsse.enableCBCProtection=false -Djava.library.path="C:\Users\administrator.PS\Desktop\db" -cp "C:\Users\administrator.PS\Desktop\db\liquibase.jar;C:\Users\administrator.PS\Desktop\db\jtds-1.3.1.jar" liquibase.integration.commandline.Main --driver=net.sourceforge.jtds.jdbc.Driver --url=jdbc:jtds:sqlserver://localhost/"vcacdb";ssl=request releaseLocks" exited with code -1.

    0 Warning(s)

    1 Error(s)

Time Elapsed 00:01:41.27

Solutions/Workaround:

Solution01:

1. Open Liquibase XML file from extracted files of "Database Upgrade Scripts".

2. Locate the ssl=request and delete it. Save the file.

3. Run the DB upgrade command again, and it works like a charm.

Personally I would not go with this workaround of modifying the configuration file.

Solution02:

Download and install JRE 1.7 and Run the DB upgrade command again.

Note: If the DB upgrade fail even after following Solution02 or Solution02, then make sure the environmental variable are set in the below format

C:\Progra~1\Java\jre7

Not C:\Program Files\Java\jre7. Looks like DB upgrade doesn’t handle the space!!

I hope this was informative. 

Upgrade vCAC 6.1 to vRealize Automation 6.2 [Step by Step]

Upgrade vCloud Automation Center components is the below order: virtual appliances, IaaS databases, and IaaS services. There is a different sequence and procedure for upgrading vCAC in distributed environment (Load balancers and multiple vCAC Appliances). This post is about upgrading minimal vCAC deployment (which has single Identity Appliance, vCAC Appliance and IaaS Server). I am following the steps provided in the upgrade guide.

Prerequisites 

1. Verify that all appliances and servers that are part of your deployment meet 6.2 system requirements for installation. See the vRealize Automation Support Matrix at

https://www.vmware.com/pdf/vrealize-automation-62-support-matrix.pdf

2. Verify that Microsoft Distributed Transaction Coordinator (MSDTC) is enabled on all vRealize Automation and associated SQL servers.

3. Back up vRealize Appliance configuration files in the following directories on each appliance.

/etc/vcac/

/etc/vco/

/etc/apache2/

/etc/rabitmq/

4. Back up all databases. Create a snapshot of your virtual appliances and IaaS servers.

5. Shut Down vCloud Automation Center Services on Your IaaS Windows Server.

6. Backup "ovf-env.xml" located at "/opt/vmware/etc/vami"

Important Note:

1. If you are using vCloud Automation Center Designer, then uninstall it and install the 6.2 version after complete upgrade is done successfully.

2. If you are using external vCO or vRealize Orchestrator appliance and if it is connected to the Identity Appliance, then upgrade vRealize Orchestrator before you upgrade vRealize Automation.

Shut Down Services:

Shut down services on  vCAC Appliance and IaaS Services.

vCAC Server Appliance:

Disable the vco-server service by below command

service  vco-server stop

chkconfig  vco-server off

Stop vcac-server, apache2, and rabbitmq-server services

service  vcac-server stop

service apache2 stop

service rabbitmq-server stop

IaaS Windows Server

Shut down services in the following order

a. All VMware vCloud Automation Center agents

b. All VMware DEM workers

c. VMware DEM orchestrator

d. VMware vCloud Automation Center Manager Service

Installing the Virtual Appliance Updates

Getting the  Virtual Appliance Updates

There are 3 options to get the updates

1. Online VMware Repository - Requires internet connection to public Vmware repository.

2. Internal Repository - Zipped bundles needs to be manually downloaded from Vmware repository to your internal repository.

3. CD-ROM Drive - ISO is read from Virtual CD-ROM

In my case, I am using the option # 3. So, download the vRealize Automation Appliance 6.2.0 Update Repository Archive and VMware Identity Appliance for VMware vRealize Automation 6.2.0 Update Repository Archive for vCAC Server and ID Appliance respectively.

a. Connect the CD-ROM drive for the virtual appliance you are updating to the ISO file you downloaded.

b. Connect to vCAC ID Appliance @ port 5480  https://va-hostname.domain.name:5480.

c. Go to Update >> Settings, Under Update Repository, select Use CDROM Updates. Click save Settings.

d. Now click on Status Tab and then click Check Updates to verify that if the mounted update is detected.

e. Click Install Updates. Click OK to confirm that you want the update to proceed.

d. After the update is installed, reboot the virtual appliance.

***Follow the same procedure for vCAC Server Appliance and make sure after reboot all the services are running. After upgrade you would notice couple of new tabs and Vmware vRealize Appliance instead of vCloud Automation center.

Upgrading the IaaS Server

1. Upgrade the Database

2. Download the IaaS Installer

3. Upgrade the IaaS Components

1. Upgrade the Database

a. Download the "Database Upgrade Scripts" from the vCAC Appliance download page by going to https://vcac-va-hostname.domain.name:5480/i

b. Extract the contents of the DBUpgrade.zip archive file. With the elevated Windows command prompt, go to the respective file location.

c. Enter a DBUpgrade command string in the following format

DBUpgrade [-S dbserver[,port number][\SQLinstance]] [-d dbname] [{-E | -U username}] [-l logfilepath]

I downloaded these files on the same IaaS server and I am running DB locally, so I used the below command

DBUpgrade -S localhost        -d vCAC01 -E

Note: .net 4.5.2 is supported for vRA 6.2, the download is available at https://vcac-va-hostname.domain.name:5480/i otherwise you are likely to get the error - MSB3073 while upgrading the DB.

2. Download the IaaS Installer

a. Open the browser and go to https://vcac-va-hostname.domain.name:5480/i and download "IaaS Installer"

Note:  Do not change the file name. It is used to connect the installation to the vRealize Appliance.

3. Upgrade the IaaS Components

a. Start the IaaS setup installer and enter vCAC login details.

b. Click on the Upgrade option and continue.

c. Now the next page detects all the components which needs to be upgraded. Enter the Service Account and DB details.

d. Now click on upgrade. It would take some time for replacing and upgrading all the components.

e. Click on Finish after successful completion.

This concludes the upgrade to vRealize Automation 6.2.

vRealize Automation Branding

vCloud Automation Center branding remains on the console login page after you upgrade from vCloud Automation Center 6.1 to vRealize Automation 6.2. You can update your login console to use vRealize Automation branding.

a. Connect to vRealize Server Appliance @ port 5480 https://vra-va-hostname.domain.name:5480

b. Click the vRA Settings >> SSO

c. Enter the settings for your SSO Server. These settings must match the settings you entered when you configured your SSO appliance.

d. Select Apply Branding and click on Save Settings. After couple of minutes, the branding is changed to vRealize Automation.

Check the issues faced during this upgrade here.

vCAC Server Appliance 6.x upgrade fails, the Update / Install runs for long time and fails

In my previous post, I discussed how to come out of the checking for updates loop. Sometimes after 3-4 hours the appliance was able to detect the updates and again the installation of updates would take a very long time and fail after couple of hour. Good amount of space was available on all VM partitions.

Here is what I did to fix this issue,

 * Before jumping on to the solution, make sure you backup below files

1.  /etc/vcac/
2. /etc/vco/
3. /etc/apache2/
4. /etc/rabitmq/

 Important: Backup "ovf-env.xml" located at "/opt/vmware/etc/vami", in my case this file was empty after couple of reboots. This file is required while the upgrade process runs post-install scripts.

Don’t forget to take snapshot of VM before starting the upgrade!!!

Time(NTP) is synchronized on all machines.

Solution:

Uninstall the Vmware tools. As its an Appliance the procedure is different.

1. Take ssh connection to vCAC Server Appliance.

2. Make note of the vmware tools RPMs installed by running the below command

lab-vcac-server:~ # rpm -qa | grep -i vmware-tools

vmware-tools-libraries-nox-9.4.6-1.sles11

vmware-tools-core-9.4.6-1.sles11

vmware-tools-vmxnet-common-9.4.6-5.sles11

vmware-tools-vsock-kmp-default-9.5.6.0_3.0.76_0.11-5.sles11

vmware-tools-plugins-deployPkg-9.4.6-1.sles11

vmware-tools-plugins-powerOps-9.4.6-1.sles11

vmware-tools-vmxnet-kmp-trace-2.0.15.0_3.0.76_0.11-5.sles11

vmware-tools-guestlib-9.4.6-1.sles11

vmware-tools-services-9.4.6-1.sles11

vmware-tools-vmci-kmp-default-9.5.13.0_3.0.76_0.11-5.sles11

vmware-tools-vsock-common-9.4.6-5.sles11

vmware-tools-vmxnet-kmp-default-2.0.15.0_3.0.76_0.11-5.sles11

vmware-tools-plugins-autoUpgrade-9.4.6-1.sles11

vmware-tools-plugins-grabbitmqProxy-9.4.6-1.sles11

vmware-tools-plugins-hgfsServer-9.4.6-1.sles11

vmware-tools-plugins-timeSync-9.4.6-1.sles11

vmware-tools-plugins-vmbackup-9.4.6-1.sles11

vmware-tools-esx-nox-9.4.6-1.sles11

vmware-tools-esx-kmods-default-9.4.6-1.sles11

vmware-tools-foundation-9.4.6-1.sles11

vmware-tools-vmci-common-9.4.6-5.sles11

vmware-tools-vmci-kmp-trace-9.5.13.0_3.0.76_0.11-5.sles11

vmware-tools-plugins-guestInfo-9.4.6-1.sles11

vmware-tools-plugins-vix-9.4.6-1.sles11

vmware-tools-vsock-kmp-trace-9.5.6.0_3.0.76_0.11-5.sles11

3. Now run the below command to uninstall all vmware-tools RPMs

# zypper remove 'vmware-tools-*'

This command works for all SLES 11 editions.

4. Confirm all the RPM are gone by running the command rpm -qa | grep -i vmware-tools

This will uninstall the vmware tools.

5. Now go ahead and Run the "Check Updates" it would take 20-30 seconds and click on "Install Updates" which would take couple of minutes.

At the final stage of upgrade, the appliance runs the post-upgrade scripts which would look for "ovf-env.xml" which has vCAC Server Appliance's network information, vm description, SSH settings and vsphere port group information. Sometimes after reboot this file goes blank, if that happens it would again fail the upgrade. Make sure you backup this file.

After the successful upgrade, the vmware tools are installed automatically. I hope this post was informative.

vCAC Server Appliance 6.x: Check Updates status stuck on "Checking for available updates" forever

I encountered this issue while upgrading vCAC 6.1 to vRealize Automation 6.2. As per the procedure, I mounted the upgrade updates ISO image to the vCAC Server Appliance and clicked on Check Updates and it remained there forever.

These were steps I tried initially to stop this with no luck!

Restarting the vCAC server services

Rebooting the Appliance

Migrating Appliance on different host

Even after reboot, the status was "Checking for available updates". So basically there was no option on console or command to come out of this loop.

Solution:

The solution is to fool the appliance to stop looking for updates in never ending loop. Follow the below steps to stop this

1. Click on the Update tab and Settings tab.

2. Under Update Settings Select "Automatic Check for Updates".

3. Set the next coming time slot, the options are available in hourly gap and click on save settings.

4. Wait till the appliance passes the selected time slot and it will automatically come out of the loop.

Important: Make sure you choose the option which you are not using to update, i.e., if I am using the CD-ROM option then I would select "Use Default Repository" or "Use Specified Repository" to break the loop.

This was tricky and simple. Hope this was informative.