SRM vApp External Automated Upgrade Script

SRM vApp External Automated Upgrade Script

Overview

This script automates the SRM update process for vApp environments. It supports both All-in-One (AIO) and Distributed deployments.

Important Notes

Windows SRM environments are not updated by this script  they are skipped and must be handled manually.

Prerequisites

Before running the script, ensure the following on the SRM Environments:
  1.  Root SSH access is available for all the target SRM environments.
  2.  All VMs are powered on and reachable over the network.
  3.  Python is installed on all SRM environments.
  4.  The script is executed from the Frontend environment.
  5.  Use `/bin/bash` to run the script (do not use `sh`).
  6.  Download the attached `vApp_SRM_Update.zip` and copy it to the Frontend Environment.

Package Contents

The `vApp_SRM_Update.zip` archive includes:
  1.  `vApp_SRM_Update.sh` – Main update script
  2. `README_vApp_SRM_Update.txt` – Reference documentation

Script Options

The script supports the following command-line options:
  1.  `-h` :  Display help and exit.
  2. `-p`  :  Pre-check VM SSH connectivity before running update.
  3. `-i`   :  Collects the SRM Linux servers and establishes SSH connection and vApp update preparation.
  4. `-r`   :  Retry SSH connections for failed servers if failed_servers_file exists and is not empty.
  5. `-c`  :  Removes CSV and log files generated by the vApp SRM Update script.

Execution Steps

1. SSH to the Frontend Environment.
2. Navigate to the directory where `vApp_SRM_Update.zip` is copied.
3. Extract the archive:
      unzip vApp_SRM_Update.zip
4. Change to the extracted directory:
     cd vApp_SRM_Update
5. Copy the SRM update file `SRM_6.1.0.0_vApp_Update_UI.zip` into vApp_SRM_Update directory.
6. Run the following command to gather all the Linux environments in the SRM setup :
      /bin/bash vApp_SRM_Update.sh -i                                                                                                                                                                                       
7. Review the generated file `SRMVMList.csv` and verify that all VM entries are correct.
8. Re-run the update command to initiate the upgrade:
      /bin/bash vApp_SRM_Update.sh -i
9. Enter the root passwords for all servers when prompted. For any SSH connection failure refer the SSH Failure Handling section.
10. After the kernel update:
  1.  All VMs will automatically reboot.
  2.  Wait for all systems to come back online.
11. Perform a connectivity pre-check after reboot, by running below command :
      /bin/bash vApp_SRM_Update.sh -p
12. Once SSH connection to all the VM's is successful, Run the update script again to complete the process:
      /bin/bash vApp_SRM_Update.sh -i
13. The update process will begin. Once update completed below message will be shown:
     "SRM Update script successfully completed on all servers"

Logs

Upgrade logs are available in the SRM Environment at below location:

        /opt/APG/Tools/Update-Manager/9.1

SSH Failure Handling

  1.  If any SSH connections fail during execution, retry them using:

                /bin/bash vApp_SRM_Update.sh -r

  1.  This will attempt reconnection for servers listed in `failed_servers.txt`.

 Once SSH connections are successful for all the SRM environments, Resume the update process from Step 8 in the Execution Steps.

Cleanup

 After completing the upgrade, remove temporary and generated files using:

       /bin/bash vApp_SRM_Update.sh -c

 This helps keep the environment clean and organized.

Notes
Note :

Additional Frontend Environment:

For SRM Environment having Additional Frontend VM, Post upgrade to SRM 6.1.0.0, Tomcat will be in stopped state and Additional frontend is inaccessible.
 Below additional steps are required to bring up Tomcat on the Additional Frontend:
          
  1. Navigate to APG startup directory 
    cd /opt/APG/bin/ 
    vi apg-tomcat-default 
  1. Search for the keyword "endorsed" and remove deprecated endorsed directory parameter. 
    Remove the line: -Djava.endorsed.dirs="/opt/APG/Web-Servers/Tomcat/Default/endorsed"    

2. Update the active uncommented  Connector configuration in server.xml from the below path. 
         /opt/APG/Web-Servers/Tomcat/Default/conf/server.xml (replace the full Connector block). 


                     <Connector port="58443" protocol="org.apache.coyote.http11.Http11NioProtocol" 
                        maxThreads="150" SSLEnabled="true" scheme="https" secure="true" 
                        compression="2048" maxSwallowSize="0" URIEncoding="UTF-8" 
                        maxParameterCount="1000" 
                        defaultSSLHostConfigName="default"> 
                      <SSLHostConfig hostName="default" 
                        certificateVerification="none" 
                        protocols="TLSv1.3" 
                        ciphers="TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, 
                                        TLS_RSA_WITH_AES_256_GCM_SHA384, 
                                        TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, 
                                        TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, 
                                        TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, 
                                        TLS_AES_128_GCM_SHA256, 
                                        TLS_AES_256_GCM_SHA384, 
                                        TLS_CHACHA20_POLY1305_SHA256, 
                                        TLS_AES_128_CCM_SHA256, 
                                        TLS_AES_128_CCM_8_SHA256, 
                                        TLS_AES_128_OCB_SHA256, 
                                        TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256"> 
                        <Certificate certificateKeystoreFile="conf/keystore" type="RSA" /> 
                </SSLHostConfig> 
        </Connector> 

 3. Restart tomcat service in the AFE 

Troubleshooting 

Script Engine Service Issue:

During upgrade of SRM from 3.x to SRM 6.1.0.0 using the External Automated Script, the Script Engine service remains in a stopped state.

Notes
Note:
This issue applies only to customers upgrading from SRM 3.x.

Root Cause:

The JDBC-Driver's module contains an outdated webservice-commons-api.jar (incomplete Jakarta migration). During startup, this older JAR is passed to the Script Engine, which results in a NoClassDefFoundError.

Although the Script Engine already includes the correct Jakarta-compatible JAR, it gets overridden by the outdated dependency from the JDBC-Driver's module.

Resolution / Fix:

An automated script has been attached to resolve this issue.

  • Script Name: Script_Engine_Fix.zip
  • Execution: Trigger the script from the Frontend environment.
  • Instructions: Refer to the README file included in the script package for detailed execution steps.

Deprecated JVM Option MaxPermSize Causing Service Startup Failure 

In earlier Java versions, the MaxPermSize parameter was used to configure the Permanent Generation (PermGen) memory. However, this parameter is no longer supported in newer Java versions and can prevent the service from starting if still present.

Sample references for affected unix-service.properties file locations are listed below:
  1. /opt/APG/Web-Servers/Tomcat/Default/conf/unix-service.properties
  2. /opt/APG/Collecting/Collector-Manager/emc-watch4net-health/conf/unix-service.properties
Example of JVM parameters:

Jvm.param.2=-XX:MaxPermSize=256m or
Jvm.param.2=-XX:MaxPermSize=128m

 

Resolution:

  • Remove or comment out the line containing the MaxPermSize parameter.
  • Update the service configuration.

        For example: /opt/APG/bin/manage-modules.sh service update tomcat

  • Restart the service to apply the changes.

Module Update Failure Due to Module Instance Ambiguity

The update script is unable to resolve the target module instance because more than one instance of the same module is deployed, and the script relies on a non-default instance selection mechanism. As a result, modules configured with default instance services will remain in a “Stopped” state.

Resolution:

    To identify modules that have not been upgraded, run the following command:

            /opt/APG/bin/manage-modules.sh list i | grep "updated"

    To explicitly update a module, specify both the module name and the instance name:

          /opt/APG/bin/manage-modules.sh update <module-name> <instance-name>