Upgrading the System Software

To download the Automated Uptime Layer with Smart Exchange upgrade kit and upgrade program, you need access to the Stratus Customer Service Portal. Log in to the portal and download the software as follows.

To download the Automated Uptime Layer with Smart Exchange upgrade kit and program

1. Click the following link to access the Stratus Customer Service Portal:

   https://www.stratus.com/support

2. Click Log-in to the Stratus Service Portal, click Login, enter your credentials, and click Sign In.

3. On the Welcome to Customer Service page, click Software Downloads.

4. On the Software Downloads page, select your product, operating system, and model. For example, if you want to download the upgrade files for a ztC Endurance 3100, 5100, or 7100 system running Automated Uptime Layer with Smart Exchange for VMware ESXi, select the following values:

   1. Under Product, select ztC Endurance.

   2. Under Operating System, select ESXi.

   3. Under Model, select 3100, 5100, 7100.

   The ztC Endurance - ESXi Downloads are displayed.

5. Click Upgrade Kit & Program.

6. Locate the files for System Software Build 1.1.0.0. Download the following files, all of which are required for the upgrade process:

   • System Software Build 1.1.0.0 for VMware vSphere zenupgrade Program

   • System Software Build 1.1.0.0 for VMware vSphere Upgrade Kit SHA256sum

   • System Software Build 1.1.0.0 for VMware vSphere Upgrade Kit

To download the required VMware ESXi upgrade depot, you need access to the Broadcom^® Support Portal. Log in to the portal and download the upgrade depot using instructions similar to the following.

To download the VMware ESXi upgrade depot

1. Click the following link to access the Broadcom Support Portal:

   https://support.broadcom.com

2. Click Login and enter your credentials to log in to the portal.

3. In the top navigation bar, to the left of your account name, click the product pulldown menu and select VMware Cloud Foundation.

4. In the left navigation bar, click My Downloads.

5. In the search box, type VMware vSphere. In the search results, click VMware vSphere.

6. In the list of VMware vSphere products, click the version to which you are licensed (for example, VMware vSphere - Enterprise Plus).

7. In the list of releases, click 8.0.

8. Locate the entry for VMware vSphere Hypervisor (ESXi) and click View Group.

9. In the release pulldown menu in the upper-right corner, select release 8.0U2.

10. Download the VMware vSphere Hypervisor (ESXi) Offline Bundle. The filename is VMware-ESXi-8.0U2-22380479-depot.zip.

    Note: You must download the ESXi depot file for this exact 8.0U2 release and build number 22380479, which is the only build supported with Automated Uptime Layer with Smart Exchange​ Release 1.1.0.0. Do not unzip the depot file; the upgrade program expects a zip file.

After downloading the upgrade files from the Stratus Customer Service Portal, upload the files from the remote management computer to the Management VM of the ztC Endurance system.

To upload the upgrade files, use options including the following:

• To upload the upgrade files over the network, use a secure copy (SCP) utility. For information about downloading an SCP utility to use for file transfers, see Configuring SSH for Remote System Management.

• To upload the upgrade files with a USB storage device, use the VMware Remote Console to connect the storage device to the Management VM as described in the VMware vSphere documentation. To mount the USB storage device at the command line of the Management VM, you can use commands similar to those described in Transferring Software ISOs to the Standby OS to determine the device name, create a directory as a mount point, and mount the device; however, remember that you are transferring files to the Management VM (and not the Standby OS) in this case.

Using either option, upload the following upgrade files to the /home/zenadmin directory of the Management VM:

• zenupgrade program

• Upgrade kit (.kit)

• Upgrade kit SHA256sum file (.sha256)

• ESXi upgrade depot (-depot.zip)

If you performed certain advanced ESXi network configuration actions on your system running Automated Uptime Layer with Smart Exchange Release 1.0.x.x, the changes could cause the upgrade to Release 1.1.x.x to fail. To ensure the success of the upgrade, perform the following steps before beginning the upgrade procedure:

1. Start by enabling ESXi Shell access to the ESXi host:

   1. Log in to the ESXi Host Client, click Manage in the left navigation pane, and click Users.

   2. If Shell Access is not enabled for the root user, select the root user, click Edit User, and activate the Enable Shell Access check box.

2. Open an ssh connection to the ESXi host as the root user.

3. Execute the following command to save the vmknic configuration to a file:

   # /usr/lib/vmware/configmanager/bin/configstorecli config current get -c esx -g network -k vmknics > /var/tmp/vmknics.orig-1.0.json

4. Open the /var/tmp/vmknics.orig-1.0.json file in the vi text editor, as follows:

   # vi /var/tmp/vmknics.orig-1.0.json

   To learn more about using vi, see help resources on the web. For example:

   https://www.redhat.com/sysadmin/get-started-vi-editor

5. In the file, locate the vmk0 entry, which is within brackets {} starting with the following device line:

   {
"device: "vmk0"

6. Examine the list of items immediately before the closing bracket of the vmk0 entry, and do one of the following:

   • If one of the lines starts with nic:, continue with the following steps.

   • If none of the lines start with nic:, then no additional action is needed. Close the text editor without saving the file (type :q!), skip the rest of the steps in this procedure, and proceed with the upgrade.

   For example, a line that begins with nic: might appear similar to the following:

   "mac_mode": "PNIC_BASED",
"nic": "vmnic0",
"net_stack_instance_key": "defaultTcpipStack",
"port_connection_type": "VSS_PORT_GROUP",
"port_group": "VMkernel",
"tso_mss": 0
}

7. If a nic: line exists, remove only that line from the file and save the changes to a new file to avoid overwriting the -orig file that contains the original settings. For example, do the following:

   1. In the vi text editor, move the cursor to the nic: line

   2. Type dd to delete the current line.

   3. Type :w /var/tmp/vmknics.new-1.0.json to save the settings to a new json file

   4. Type :q! to close the text editor without overwriting the original json file.

8. Run the following command to save the updated settings to the configuration store:

   $ /usr/lib/vmware/configmanager/bin/configstorecli config current set -c esx -g network -k vmknics -j /var/tmp/vmknics.new-1.0.json

9. In the ESXi Host Client, reboot the ESXi host to implement the changes.

10. When you finish using the ESXi Shell, disable it to secure your system. Repeat step 1 of this procedure, except disable ESXi Shell access by clearing the check box for Enable ESXi Shell.

You can now proceed with the upgrade.

In the following procedure, to initiate the system software upgrade, use the zenupgrade program.

If you followed the instructions in the previous procedure, the upgrade program is located in the Management VM in /home/zenadmin.

Upgrade program usage:

Usage : zenupgrade_version {options}

Options:
-d filename   VMware ESXi upgrade depot (.depot.zip) file
-k filename   ztC Endurance upgrade kit (.kit) file
-p password   ztC Endurance console​ administrative password
-u username   ztC Endurance console​ administrative username
              (for example, the default admin account)

To run the zenupgrade program, for example:

To upgrade from Automated Uptime Layer with Smart Exchange Release 1.0.x.x to 1.1.x.x or higher

1. Ensure that you uploaded the zenupgrade program, ztC Endurance upgrade kit, and ESXi upgrade depot to the Management VM as discussed in the previous procedures.

2. In the ztC Endurance console​, verify that the system is healthy with no outstanding alerts. Resolve and clear any alerts; otherwise, the upgrade will not proceed.

3. If applicable, in the ESXi Host Client or vSphere Web Client, shut down the virtual machines (except for the Management VM) in preparation for the upgrade; otherwise, the system will automatically shut down and restart them about halfway through the upgrade process.

4. Use one of the following methods to log in to the Management VM as the zenadmin user.

   • Open a connection with an SSH client. See Configuring SSH for Remote System Management for more information.

   • From the ESXi Host Client or vSphere Web Client, open a browser console window or remote console window. See the VMware vSphere documentation.

5. Before starting the upgrade, you must disable the ActiveService Network (ASN) on the ztC Endurance system to prevent any ASN activity during the upgrade. (Note that simply disabling the ASN on the Support Configuration page of the ztC Endurance console​ is not enough in all cases.)

   On the command line of the Management Appliance, execute the following commands to disable the ASN service:

   $ sudo systemctl stop callhome
$ sudo systemctl disable callhome
   Note: To use privileged commands in the zenadmin account, you need to add the sudo directive.

   You can restart the ASN service after the upgrade, as described in V. To complete post-upgrade tasks.

6. On the command line of the Management Appliance, ensure that you are in the directory that contains the zenupgrade program, the upgrade kit, the upgrade kit SHA256sum file, and the VMware ESXi upgrade depot. If you followed the instructions in the previous procedures, these files are located in /home/zenadmin, for example:

   $ cd /home/zenadmin

7. Execute the following command to verify the integrity of the upgrade kit (.kit) file by comparing its checksum to the one provided in the SHA256sum (.sha256) file:

   $ sha256sum -c ztcEndurance_upgrade_n.n.n.n-nnn.sha256

   If the checksum matches, the command indicates that the kit file is OK. If the checksum does not match, delete the kit file, download a new copy, and repeat this step.

8. Execute the following command to change the attributes of the zenupgrade program to make it executable:

   $ chmod +x zenupgrade_n.n.n.n-nnn

9. Execute the zenupgrade program to begin the upgrade process, as follows:

   $ sudo ./zenupgrade_n.n.n.n-nnn -k ztcEndurance_upgrade_n.n.n.n-nnn.kit -d VMware-ESXi-8.0U2-nnnnnnnn-depot.zip -u admin -p password

10. When prompted, read the ztC Endurance Service Terms and Conditions. For links to the full text of this and other agreements associated with ztC Endurance, see End-User License Agreements and Warranty. To accept the terms, type yes and press Enter.

11. The zenupgrade program prepares the software upgrade components and upgrades the following components of the Management VM:

    • Ubuntu OS

    • VMware ESXi software and utilities

    • Automated Uptime Layer with Smart Exchange software and utilities

    When the upgrade program completes these initial tasks, which take about 10 minutes, the system automatically restarts the Management VM on the upgraded software, and you lose your connection to the shell session.

    Caution: During the course of the upgrade, the system might display prompts for you to restart the Management VM, ESXi host, or Standby OS to apply changes; however, you must ignore these prompts and allow the zenupgrade program to automatically restart each component at the correct time to ensure a successful upgrade. If you manually restart any component while the zenupgrade program is still working, the upgrade will likely fail and leave the system in an unusable state.
    Notes:  

    • Because the Management VM provides the ztC Endurance console​, you lose access to the console for 1-2 minutes while the Management VM restarts.

    • Refresh your web browser and log in again to display the new ztC Endurance console​ pages included with this release, including the new Upgrade Kits page that you can use to monitor upgrade progress. The Upgrade Kits page will not appear in the user interface until you refresh the web browser.

12. To continue monitoring the upgrade progress, open the Upgrade Kits page. While an upgrade is in progress, the Upgrade Kits page lists the state of the Host (ESXi), Compute Module A, and Compute Module B as pending, updating component, or up to date. You can also monitor the progress of the upgrade through alerts displayed on the Dashboard page.

    Caution: The Upgrade Kits page allows you to monitor the upgrade; however, there will still be long periods when it appears like there is no progress even while the upgrade continues in the background.
    
    Do not interfere with the upgrade or restart the system; otherwise, you might leave the system in an unusable state. If the upgrade takes longer than 3 hours with no signs of progress on the Upgrade Kits page, contact your authorized Stratus service representative for assistance before proceeding any further.

13. The system automatically continues to complete the following tasks, which take about an hour:

    • Upgrades the ESXi host software that is currently running on the active compute module

    • Puts the standby compute module into maintenance mode

    • Shuts down the Standby OS on the standby compute module

    • Upgrades the BMC and BIOS firmware on the standby compute module

    • Reinstalls the Standby OS software (Ubuntu) on the standby compute module

    • Restarts the standby compute module

14. When the standby compute module is up to date, the system automatically initiates a reboot exchange A process for switching from the active compute module to the standby compute module when a Smart Exchange is not possible due to firmware mismatches between the two modules. During a reboot exchange, the system: 1) Instructs the standby compute module to become active. 2) Shuts down the original active compute module. 3) Reboots the new active compute module, which automatically takes ownership of all I/O and storage connected to the original active compute module. 4) Starts the system software on the new active compute module. A reboot exchange is typically initiated during software upgrades, firmware updates, and troubleshooting procedures.:

    • Automatically shuts down any running virtual machines (if applicable)

    • Automatically shuts down the ESXi host

    • Takes the upgraded compute module out of maintenance mode

    • Performs a reboot exchange to reprovision the storage and devices​ to the upgraded compute module

    • Starts the ESXi host and virtual machines (if applicable) on the upgraded compute module

    Notes:  

    • The system automatically initiates the reboot exchange about 1.5 hours into the upgrade process.

    • You can monitor when the reboot exchange is about to start, as follows:

      • On the Dashboard page, the system clears the Standby system update is in progress on compute module x alert.

      • On the Upgrade Kits page, the state of Compute Module x switches to Up to date and the state of the Host briefly switches to Reboot exchange in progress.

      At this point, the system begins to shut down the virtual machines (if applicable) and the ESXi host in preparation for the reboot exchange.

    • You lose access to the ztC Endurance console​ for 15-20 minutes. After the Management VM restarts, refresh your web browser and log in again.
    • If any virtual machines were running on the system, you lose access to the virtual machines for about 20-25 minutes during the reboot exchange process.
    • After the reboot exchange, no further downtime is needed.

15. After the reboot exchange, when the system is running on the newly upgraded compute module, software, and firmware, the system:

    • Puts the standby compute module into maintenance mode

    • Shuts down the Standby OS on the standby compute module

    • Upgrades the BMC and BIOS firmware on the standby compute module

    • Reinstalls the Standby OS software (Ubuntu) on the standby compute module

    • Restarts the standby compute module

    • Takes the standby compute module out of maintenance mode

16. Continue monitoring the upgrade progress on the Upgrade Kits page of the ztC Endurance console​. At this point, the Upgrade Kits page displays the state of the Host (ESXi) and the newly upgraded Compute Module x as up to date, and it displays the state of the standby Compute Module x as updating component. You can also view the state of the compute modules on the Dashboard page.

17. After the system upgrades the standby compute module, the upgrade process is complete. On the Upgrade Kits page, the upgrade kit State changes to Applied.

1. After the upgrade, the system automatically deletes the ESXi upgrade depot and moves the Automated Uptime Layer with Smart Exchange upgrade kit from /home/zenadmin to /opt/stratus/mgmt/upgrade_kits. Optionally, only if you need to free up disk space in the Management VM, you can delete the upgrade kit and associated files as follows:

   1. Select and delete the upgrade kit from the ztC Endurance console​ as described in the Upgrade Kits Page.

   2. Delete the files and folders located in /opt/stratus/mgmt/upgrade_kits as shown in the following example:

      $ sudo ls /opt/stratus/mgmt/upgrade_kits
      temp_kitDir_nnnnnnnnnn  ztcEndurance_upgrade_1.1.0.0-nnn  ztcEndurance_upgrade_1.1.0.0-nnn.kit
      $
      $ sudo bash -c "rm -r /opt/stratus/mgmt/upgrade_kits/*"
      $ sudo ls /opt/stratus/mgmt/upgrade_kits
      $
      

2. In the ztC Endurance console​, verify that the system is healthy with no outstanding alerts. Resolve and clear any alerts before continuing.

3. If the ztC Endurance console​ alerts you to register, register the system, download a license file, and apply the license file as described in Registering the System.

4. Verify that your system and your virtual machines are functioning properly on the new software.

5. If you disabled autostart for any virtual machines, remember to re-enable it.

6. On the command line of the Management Appliance, execute the following commands to re-enable the ASN service:

   $ sudo systemctl start callhome
$ sudo systemctl enable callhome

   If needed, confirm that the ASN connection is working by generating a test alert as described in Configuring Remote Support Settings.

7. If needed, configure ASNConnect as described in Enabling ASNConnect. (Your ztC Endurance system must have a valid product license to run ASNConnnect.)