HA Configuration

The HA Configuration panel is used to manage High Availability (HA) configurations. For more information on how HA works with the AMM system, see High Availability (HA) Failover .

This panel can only be accessed by administrators and is located under Admin > System > HA Configuration.

Workflow for setting up High Availability

For a new install

1. Install two identically sized standalone AMMs.
2. Configure the two AMMs as an HA pair using Admin > System > HA Configuration. See Setup HA Pair.

Adding HA to a standalone AMM.

1. Install an additional identically sized standalone AMM with the same version as the current AMM.
2. Configure the two AMMs as an HA pair using Admin > System > HA Configuration. See Setup HA Pair.

For an existing HA config upgrading to 2.17.5 only

1. Teardown existing HA server configuration using documented procedure from Technical Support.
2. Upgrade the secondary server using the upgrade script on the command line.
3. Upgrade the primary server using the upgrade script on the command line.
4. Configure the HA pair using Admin > System > HA Configuration. See Setup HA Pair.

For an existing HA config upgrading from 2.17.5 to a future version

1. Teardown the HA server using Admin > System > HA Configuration on both primary and secondary servers. See Teardown HA Service.

2. Upgrade each server using Admin > System > Upgrade. See Upgrade.

   When upgrading the secondary server, you may encounter a scenario where the target package appears in the Repository list but generates an error when displayed in the System Upgrade list. This situation typically arises in a High Availability (HA) system, wherein the package has been uploaded to the primary server and a corresponding record has been added to the database. However, the file replication process has not yet completed the transfer of build files to the secondary server.

   In such cases, if a user decides to dismantle the HA configuration at this point, they can proceed to the secondary server and initiate an upgrade to the new release. However, because the release files have not finished copying to the secondary server, the upgrade will fail, resulting in a ‘No such file’ error. This scenario may occur due to issues with file replication between the HA hosts or if the files for the release have not completed synchronization.

3. Configure the HA pair using Admin > System > HA Configuration. See Setup HA Pair.

Setup HA Pair

When configuring HA, make sure to monitor the process to the end.

To configure an HA pair, follow these steps:

On the primary server:

1. In the Local Server Role section, set the HAServerRole to primary.

   

2. In the Local Server Address section, enter the HAServerAddress.
3. In the Peer Server Address section, enter the HAPeerAddress.
4. Enter the Peer Authentication Fingerprint. Click Generate button to generate a public key.
5. In the HA Alert Email Account section, enter the HAAlertEmail.
6. Click SAVE button to save the changes as a Draft.
7. On the secondary server:
7. In the Local Server Role section, set HAServerRole to secondary.

   

8. In the Local Server Address section, enter the HAServerAddress. Value must be the same as HAPeerAddress on the primary server.
9. In the Peer Server Address section, enter the HAPeerAddress. Value must be the same as HAServerAddress on the primary server.
10. Enter the Peer Authentication Fingerprint. Value must be the same as Peer Authentication Fingerprint on the primary server.
11. In the HA Alert Email Account section, enter the HAAlertEmail. Value must be the same as HAAlertEmail on the primary server.
12. Click the SAVE button to save the changes as a Draft.
13. On the primary server, select APPLY button. Wait until the message HA setup change has initialized successfully. is received.

    Note: If there is a communication error between the HA pair, the primary server will show a message that there is a communication error between the HA peers.

    

    If the user selects APPLY again, it will say that there are “No Configuration Changes To Apply”

    

    The HA primary will not re-attempt communication with the HA secondary. Instead, change the primary HAServerRole to standalone and save. Then restart the setup of the HA pair.

14. On the primary server, a message will prompt the user to now apply the changes on the secondary server.

    Do not do any further actions on the primary server until prompted to.

15. On the secondary server, select APPLY button. Wait until the message HA setup change has initialized successfully. is received.

    Note: On a very rare occasion, the HA pair setup will complete successfully, but show a message that the copy is a draft.

    

    The message that the copy is a draft should be ignored.

16. On the primary server, navigate to Admin > System > Status page. Once the HA setup is completed successfully, the HA Status shows as stable.

Note: If the HA status (see Admin > System > Status ) is not stable, teardown the HA configuration on both the primary and secondary servers. Then re-start the process to setup an HA pair.

If the primary server fails to configure successfully, but the secondary server succeeds, then the secondary server must be re-configured back to standalone and then again re-configured to secondary after the primary server configures successfully.

After an HA setup is done, the user activity on the secondary is the same as on the primary, so the system loses track as to who and when the administrator committed the change, as well as the configuration details on secondary.

Teardown HA service

This process will teardown the HA service on both primary and secondary servers.

1. Using Admin > System > Status, verify the HA status is stable.
2. Using Admin > System > HA Configuration, set the Local Server Role to standalone and apply the change on primary server only.

Note: If the HA status (see Admin > System > Status ) is not stable, set the Local Server Role to standalone and apply this change on both servers.

Swap HA roles

The following process will swap the HA roles.

1. Using Admin > System > Status, verify the HA status is stable.
2. Under Admin > System > HA Configuration, set the Local Server Role to secondary and apply the change on the primary server only. This will swap the role of primary and secondary servers with each other.

Note: AMM user activity logs are saved in MySQL, and the database on the secondary server is constantly synchronized with the one on the primary server. After an HA swap is done, the user activity on the secondary server is the same as on the primary server, so the system loses track as to who and when the administrator committed the change, as well as the configuration details.

Advanced HA Configuration

The Advanced HA configuration can be used to customize some special functions as per need and also fine-tune performance.

Name	Configured on	Variable	Definition
Files Sync Frequency	Primary	HAFileSyncInterval	Number of seconds before the files are synced to the secondary server (files are also synced every 1000 changes regardless of this timer). Must be an integer between 5 and 30. Default is 5.
HA Custom Script	Primary|Secondary	HAAdminAlertScript	Copy and paste a custom bash script which will be automatically executed when an HA event occurs. Events which can trigger execution of the script are: failover, secondary offline, role swap, database replication error, and database replication recovery. See Custom Script.
Number of Continuous Failed Pings to Trigger Failover	Secondary	HAPingIterations	Number of failed pings to the primary server before the secondary server will trigger a failover. Please note that the secondary server sends a ping to the primary server every 15 seconds. Must be an integer from 2 to 20. Default is 12.
Additional Ping Targets to Confirm Failover	Secondary	HAPingMonitor1, HAPingMonitor2, HAPingMonitor3	Failover will not take place if the secondary’s default gateway and these additional targets stop responding. This is to prevent network issues on the secondary server from initiating a false failover. Must be a unique valid IP address or FQDN. Must be different fro Hostname, IPAddress, Gateway, HAServerAddress, HAPeerAddress and not in ManagementTunnelNetwork. For HA peers deployed in the same IP subnet, do not specify any HA ping targets. For HA peers deployed in different IP subnets, due to a known issue, specify only one HA ping target.

Guideline for HA Custom Script

The bash script will automatically be executed with root permission when an HA event occurs.

The script parameters for different events are listed below.

HA Event	Executed on	Parameter1	Parameter2	Parameter3
Failover	secondary	“Failover”	<active host>	<standby host>
Secondary offline	primary	“StandbyOffline”	<standby node>	<active node>
Role Swap	secondary	“ManualSwitch”	<active host>	<standby host>
Database Replication Error	secondary	“RaiseAlarm”	<alarm condition as text>	<error message>
Database Replication Recovery	secondary	“ClearAlarm”	<text containing alarm condition and error message>

Note: The StandbyOffline case does not have a clear when the secondary server comes back online.

Example script:

    #!/bin/bash

    logger -t AMM_HA "$(basename $0) args 1=$1 2=$2 3=$3"
    if [[ $1 != Failover ]] && [[ $1 != ManualSwitch ]]; then
      exit 0
    fi

    DOMAIN="domainname.acme.ca"
    DNS_SERVER="dns1.acme.ca"
    KEY_FILE="/mnt/amm_data/opt/inmotiontechnology/scripts/Kdyn.acme.ca.+157+55618.key"

    HOST=`hostname`

    echo "server $DNS_SERVER" > /tmp/nsupdate
    echo "debug yes" >> /tmp/nsupdate
    echo "zone $DOMAIN." >> /tmp/nsupdate
    echo "update delete eng.$DOMAIN" >> /tmp/nsupdate
    echo "update add eng.$DOMAIN 180 CNAME $HOST" >> /tmp/nsupdate
    echo "send" >> /tmp/nsupdate

    nsupdate -k $KEY_FILE -v /tmp/nsupdate 2>&1