Application Services 6.x service blueprint deployment error - ERROR: C:\opt\vmware-appdirector\agent\appd.properties not found. Abort

Issue:

This error is very common error and very irritating, the application deployment fails after very long wait on Node Setup stage and when you see the Deployed VM you would find the below error at C:\opt\vmware-appdirector\agent\logs

INFO 2015-11-10 16:56:17,468 - C:\opt\vmware-appdirector\agent\appd.properties not detected.

INFO 2015-11-10 16:56:17,468 - Wait for 2 seconds before retry ...

And after 150150 seconds, retrying every 2 seconds it aborts

INFO 2015-11-10 16:57:03,578 - Wait limit 150150 seconds over. Giving up.

INFO 2015-11-10 16:57:03,578 - ERROR: C:\opt\vmware-appdirector\agent\appd.properties not found. Abort.

INFO 2015-11-10 16:57:03,734 -

INFO 2015-11-10 16:57:03,734 -

INFO 2015-11-10 16:57:03,749 - Process exited with code 1

And on Application Services execution plan of Deployments will remain at Node Setup for several minutes and fails

Resolution:

There could be multiple reasons for this, before trying below steps make sure you have prepared the Windows Template as recommended in the below VMware document

 http://pubs.vmware.com/vra-62/topic/com.vmware.ICbase/PDF/vrealize-automation-62-using-application-services.pdf page 52

If correct JRE version, Vmware Cloud Application Director Agent Bootstrap service and VCACGuestAgentService (vCAC agent service) is installed and could be restarted without errors in the Template then stop both the services and proceed further.

Check: If the windows machine deployed is getting the proper IP and networking, sometime the VM doesn’t pickup the IP from Customization Specification. If that is the case then create a new customization specification and run again or try the blow steps to reconfigure the networking configuration again

You can find customization specification in vCenter Server at Home>Customization Specifications Manager

01: In the networking section, select Custom settings instead of Typical settings

02: If it was already on Custom settings then delete the default network interface and Click on New

03: By default it is set on DHCP, click on the setting button available on top right of network interface button

04: Now Select the DNS tab and update the DNS information, Save the settings and complete the Customization Specification Wizard

Now request Application Service from vRealize Automation service Catalog and see if get through that stage, if it still continues then follow the below steps

Ideally in the application services blueprint (Template), the vCAC guest agent is installed automatically by  VMware Cloud Application Director Agent Bootstrap service. So as per the Vmware documents, you would just  place the unzipped files to their designated location and run the Bootstrap service. In this method we would manually install the services. For that first lets uninstall all the services

Uninstalling the Services (Cleanup)

• Connect to your Template (Convert to VM if Template or Power On the VM is using Linked Clone), and run the Command Prompt with Admin Privilege

• Go to Temp directory where you extracted the Vmware Cloud Application Director Agent Bootstrap service  and run the below Batch file

appd_bootstrap_removal

This batch file  will remove the files and folder from "C:\opt\". Check the respective service has been removed, sometimes it stays on disabled mode, you may wait or restart the Windows if it stays at disabled mode.

• Now go to uninstall the vCAC guest agent service by going to its extracted location and running the below command, i.e.,

winService.exe -u

At this point, I would also delete all the extracted files and start over again, delete the files from

C:\VRMGuestAgent

C:\opt\vmware-jre

Lets start from the beginning, reboot the Windows  and download the required files

Find all the agents on Application Services Appliance @ /home/darwin/tcserver/darwin/webapps/darwin/agent

1. JRE 1.7 http://ip of Application Services appliance/agent/jre-1.7.0_72-win64.zip

2. VMware Cloud Application Director Agent Bootstrap service http://ip of Application Services appliance/agent/vmware-appdirector-agent-bootstrap-windows_6.0.0.0.zip

3. Windows vCAC Guest Agent  https://ip_of_vRealize_Automation_appliance:5480/installer

Important: Before extracting the downloaded files, disable the security by going to the properties of the zip file and click on "UNBLOCK", if you miss this step then certainly you will encounter above error message.

1. Extract the JRE 1.7 files to "C:\opt\vmware-jre"

Confirm the Java version by below command

2. Extract the VMware Cloud Application Director Agent Bootstrap service to Temporary folder  "C:\Temp", also place the "NTRights.exe" file in the same location. If you do not have NTRights.exe file then download "Windows 2003 Resource Kit", then copy NTRights.exe from this kit. If you do not use NTRights.exe for bootstrap service delegation then you are instructed to manually delegate .\darwin user to the Bootstrap service.

Now install this service by running below command

install.bat password=your_desired_password cloudProvider=vcac vcacServer=fqdn_of_IaaS_Server httpsMode=true

There shouldn’t be any error after the above command runs

Check if bootstrap service is installed and delegated to  .\darwin

You can reset the .\darwin user's password by going to Start>>RUN>>lusrmgr.msc, use this if the service doesn't start with the given password

3. Extract Windows vCAC Guest Agent to "C:\" and

Cd C:\VRMGuestAgent

Run the below command

WinService -I -h fqdn_of_IaaS:443 -p ssl

Make sure the above services are installed and confirm if you can start the services. Stop both services now and power off the VM.

Create a new snapshot for linked clone or convert the VM to template for full clone and proceed deploying the application now. Hope these steps are helpful, as I was lost and spent much more time troubleshooting. 

Installing and Configuring vRealize Operations Manager 6.0 (vROM) AKA vCenter Operations Manager

I am deploying vRealize Operations Manager 6.0 which is completely different from all its previous versions. The configuration is completely changed, looks very simple and light. I am setting up single vROM, meaning the master node must also perform data collection and analysis because it is the sole node, and the only place where vRealize Operations Manager adapters are installed. For distributed or multi-node  installation those roles could designated to different nodes.

1. Master Node: Required node in the cluster which performs all the actions.
2. Data Node: (Optional) - which has adapters installed to perform collection and analysis. Suitable for larger Deployments.
3. Replica Node: (Optional) - Replica of Master Node for High Availability. Suitable for larger Deployments.

4. Remote Collector Node: (Optional) - Distributed deployments might require a remote collector node that can navigate firewalls, interface with a remote data source. Only gathers objects for inventory and doesn’t store or perform any analysis. Suitable for Distributed Installation.

vROM sizing calculator could be found here

kb.vmware.com/kb/2093783

vROM could be installed and configured on

1. As an Vmware Appliance (From vCenter not through ESXi Host)

2. Windows Operating System

3. Linux Operation System.

Check the below guide for installing on Windows or Linux Operating System, I am deploying as an Appliance using .ova file.

http://pubs.vmware.com/vrealizeoperationsmanager-6/topic/com.vmware.ICbase/PDF/vrealize-operations-manager-60-linux-windows-install-guide.pdf

Deploy the vROM Appliance:

1. Connect to vCenter Server, click on File and select deploy from OVF Template and locate the ova file and select next.
   
2.  Accept the License and give the name for vRealize Operations Manager Appliance.  Select the appropriate Cluster, Host and Datastore.
3. Select the appropriate deployment Configuration and complete the wizard to start the deployment.
   
4. Select appropriate Network Mapping and complete the network configuration details. It is always static IP for Master and Replica Node.
   

Additional nodes as mentioned above could be deployed following the same above steps. I am setting up this solution for smaller environment so I would only use single Master Node.

Master Node Configuration:

1. Enter the FQDN or IP of the appliance in browser. You are presented with 3 options as mentioned below, the getting started tab for each of the option has very nice flow diagram which gives you a high level idea how its going to work

1. Express Installation
   

2. New Installation
   
3. Expand Existing Installation
   

I am going with the New Installation, so I would be creating a new vRealize Operations Manager cluster with this node as the master node. 

2. Set the Administrator Password and click on Next.
   
3. I am using the default certificates which came with vROM. If you have server certificate of your own to be used for secure communication, you may upload here.
   
4. Enter the name for Master Node and NTP server details. Click on Next and then Finish to complete the configuration.
   
5. Now you can add more nodes or enable services for data collection on master node.
   

Import Data from Existing vCenter Operations Manager or Configure to collect data from environment

Existing vCenter Operations Manager can be imported to new deployment and when you turn off the earlier deployment, monitoring of the inventory and data is done solely by the new version of vRealize Operations Manager.

1. I would choose "New Environment" option to monitor my other solutions.
   

This how it goes if you choose "Import Environment" option.

2. Accept the License Agreement, enter the License Key and Click on Finish to complete the configuration.
   

Add Solutions to be managed by vRealize Operations Manager

1. Click on Solutions, and then Configure. I'm adding 2 vCenter details here.
   
2. IN the next step you define what all needs to monitored, I am going with all default options.
   
3. Click on Finish to complete the configuration.

It would take some time to collect the performance Data from added vCenter Server. This concludes the configuration, the new version of Operations Manager is smooth and works like a charm. You just need one 1 IP to setup the Single Node environment, I hope this post was informative. 

Prepare Linux 6.5 (RHEL) Template for vRealize Automation Application Services (Application Director)

Regular vSphere Template cannot be directly used in vRealize Application Services, we need to install the agent and prepare it for to be used with Application Services. In this post, I am using RHEL 6.5, the steps would remain same for other flavours of linux

1. Prepare a Linux VM, I am using RHEL 6.5. Login with root.

2. Download the installer script http://Application_Services_Server_IP/tools/preparevCACTemplate.sh to the Linux machine. You can use the wget or scp commands to download the file.

3. Ensure that the installer script has permission to execute.

chmod +x ./preparevCACTemplate.sh

4. Run the installer script.

./preparevCACTemplate.sh

1. Enter the vRealize Automation Appliance Host Name or IP
2. Enter the IaaS  Manager Services Server Hostname or IP

3. Enter the Application Services Server Hostname or IP
   

4. Next prompt is to Check Certificate and Download & Install Java Runtime Environment 1.7.0_72

Make sure you get the confirmation

#######################################

# Installation Completed Successfully                  #

# Ready to capture as a template                         #

#######################################

5. Make sure if the Application Director agent service is available

 rpm -qa | grep -i appd

vmware-appdirector-agent-service-vcac-6.0.0.0-0.x86_64

5. Shut down the Linux virtual machine.

For this VM to be added as Template on Application Services (Application Director), we need to publish this as Blueprint from vRealize Automation (vCAC). You could use the Full Clone or Linked Clone method in the Blueprint. I am using Linked Clone to save storage and time.

1. Take a snapshot of this VM and give the name & description appropriate.
   
2. Connect to your vRealize Automation console and got to Infrastructure>>Blueprints and enter respective information.
3. Under Build Information, Select "Linked Clone" for Action and select the snapshot of respective VM from "Clone From" option. Rest of the options are self explainatory.
   
4. Now hover the mouse over Blueprint name and click on Publish.
   
5. Now go to Infrastructure>>Compute Resources, Hover the mouse on respective vCenter(Endpoint) and click on Data Collection. Click Request Now for Inventory, State and Performance.

This will make this template available for vRealize Automation Services (Application Director)

Install and Configure vRealize Application Services 6.2

Deploy vRealize Application Services 6.2 (Application Director)

Before you deploy the Application Services appliance, make sure you have created the IP Pool at the Datacenter level and associated the right Network Pool. The deployment of Application Services is the same OVA deployment, input all the required information and your Application Services appliance deployed in few minutes. Once the Appliance is successfully deployed follow the below steps to complete the configuration

Configuring vRealize Application Services after Deployment

1. While it is booting, it would prompt for the Serial Number (license). Enter the Serial Number and press Enter.
2. Prompted for "root" paasword.

3. Next prompt is for "Darwin_User". Remember, When you are prompted for this password again, you have only three attempts to type the correct password.
   

4. Here you can enable ssh services by typing "yes" to the prompt.

5. Now you are prompted to enter the vRA or vCAC url.
6. Now enter System Administrator Username, i.e., administrator@vsphere.local and enter the password when prompted. It would start the registering process.
   
7. Wait till you get the "vCAC server registered successfully" message.
   
8. Next prompt is to set up Out-Of-Box sample content , select "y" and press enter.
9. Now enter the Tenant details, make sure the tenant user mentioned has the application services role in order to import the sample contents.

10. Enter the Business Group details, the Business group has space in it then enclose the entire name in double quotes.

11. At this stage it would import the out-Of-Box sample contents to added tenant. Wait till you get the "import successful" message.  Press any key to continue

12. In the last step you are logged in as admin and prompted to enter the new password. Enter the password and that concludes deployment of vCloud Application Services 6.2.

Configurations on vRealize Application Services 6.2

1. Connect the Application Services portal by going to https://ip_or_fqdn_of_appd:8443/darwin/org/tenant_name
   

2. Enter the Tenant username and password to login, the user who has application services privileges.
   

3. Click on Application drop down and select "Cloud Providers"
   

4. New Cloud Provider: Application Services registers a cloud provider by connecting to a vRealize Automation business group.  Click on the Green Plus and Fill in the details. I am using vCAC as the Cloud Provider here, vCloud and EC2 are the other supported options. The username and password is the Business Group Manager credentials for the respective Business Group selected. Click on "Validate Connection" and then save after successful validation.

Add Template: Template would be the IaaS VM, required to deploy application and services through Application Services. I have created a Linux Blueprint with Linked Clone with vCAC guest agent and AppD agent installed. See Prepare RHEL Template for Application Services here.

Click on the Green Plus next to Template and select the Blueprint published for this purpose.

5. Create a Deployment Environment: You can map a deployment environment to a vRealize Automation reservation policy before you deploy applications to that cloud environment. If you do not select a reservation policy, vRealize Automation picks any suitable reservation policy at deployment time.

Click on the Green Plus and fill in the details. Select the Reservation policy, (in vCAC >> Infrastructure>>Reservation>>Reservation Policy) and click on Save.

Now you are ready for the Application Deployments, I would share post on Application Deployment soon. Hope this post was informative.

Installing and configuring vCenter Log Insight 2.0

vCenter Log Insight or now known as "vRealize Log Insight" is a real-time log management solution for Vmware environment. It has "machine learning -based intelligence grouping" which will group the related data together, consolidates information and shows in customizable graphs. It is also a Syslog Server which can consolidate logs and analyze it.

Log Insight includes a built-in syslog server, it can analyze log events from any source which can forward syslog feeds. So you may configure ESXi, vCenter or any Cloud Suite product to forward log events to Log Insight.

Integration Options:

Currently it integrates with multiple vCenter Servers and vCenter Operations Managers.

Log Insight can collect two types of data from vCenter Server instances and the ESXi hosts that they manage.

* Events, tasks, and alerts are structured data with specific meaning. If configured, Log Insight pulls events, tasks, and alerts from the registered vCenter Server instances.

* Logs contain unstructured data that can be analyzed in Log Insight. ESXi hosts or vCenter Server Appliance instances can push their logs to Log Insight through syslog

Deployment Configuration:

Small - upto 100 ESXi servers

Medium - upto 250 ESXi Server

Large - up to 750 ESXi server

Deploy the Appliance:

1 . Connect to vCenter Server, click on File and select deploy from OVF Template and locate the log Insight ova file and select next.

2. Accept the License and give the name for Log Insight Appliance.  Select the appropriate Cluster, Host and Datastore.

3. Select the appropriate deployment Configuration and complete the wizard to start the deployment.

After successful deployment start with the configuration.

1. Open browser and enter http://fdqn/ip_of_log_insight.
2. Click on next to the start the configuration.
   
3. As this is the first time setup, we would select the "Start New Deployment" option.
   

4. Admin Credentials: Enter the admin email and set the password and click on "save and continue".
   

5. Enter the License Key and click on continue.
   

6. General Configuration:  Enter email address to send System Notification.
   
7. Time Configuration: Enter your NTP server details and click on test to confirm. Or you may sync time with ESXi Host.
   

8. SMTP Configuration: Update the correct SMTP details. You may confirm the configuration with "Sent Test Email".

9. This concludes the configuration, go ahead and click on Done. 

Integrations:

vCenter Server:

Next it will automatically takes you to the Dashboard, as this is first time setup, we need to do the vSphere Integration. Click on the vSphere Integration hyperlink.

You may integrate multiple vCenter Servers and vCenter Operations Managers.

1. Enter the vCenter Server details and Click on Test.

2. After successful Test, click on Save.

vCenter Operations Manager:

1. Click on the vCenter Operations Manager tab under Integration.

2. Enter the vCOPS details, remember the username is "admin" not root. After successful Test, click on save.
   

vCenter Log Insight Interface:

There are 2 default Tabs: Dashboard and Interactive Analytics. The Dashboard shows your complete environment's events, faults etc graphically and has inventory of all objects at left which can show relevant information.

The Interactive Analytics shows  the real time log analysis, where you can search with query words and set the time duration starting from last 5 minutes to All Time. Once you get handy with this analytics, the root cause analysis becomes very easy.

I hope this post was informative.