Symantec PAM Documentation: Cluster Synchronization, Promotion, and Recovery

Online: The member is a fully functional group member, can connect and start executing transactions.

Recovering: The member is going through the recovery process, receiving state information from a donor.

Offline: MySQL exists on the member, but it does not belong to the group.

Unreachable: The group suspects that the server is not reachable because it has crashed or disconnected involuntarily.

Error: An error occurred on the recovery phase or while applying changes.

Missing: cannot find the member in the MySQL replication group.

In Sync: Member is in sync with the primary site.

Timeout: A yellow triangle with an exclamation mark means that the connection has timed out trying to this member. A warning appears on the Site Members table.

Lag Detected: The member lags 5 minutes behind the Primary Site. A warning appears on the Site Members table.

Out of Sync: The member lags 10 minutes behind the Primary Site. A warning appears on the Site Members table.

Inactive: The member lags 15 minutes behind the Primary Site. When Inactive, a member no longer allows password views, auto-login, and so on. An email is sent to the Monitor. Important: When you change this value, you are changing how far out of sync a member can get before stopping it from functioning. However, if the Primary Site is unavailable and it is set to Operationally Safe mode, the Secondary Site can serve its own stale data. See step 5 in Configure a Cluster .

Remove Cluster Members

Cluster Synchronization

Select the checkbox to the site you want to re-sync. Re-sync sites one at a time.

Select Re-sync Site .

On a primary site member, the Site Members section shows the members for all sites.

On a secondary site member, you can only see the secondary member that you are logged in to.

Select the checkbox to the member you want to re-sync.

Select Re-sync Site Member .

Databases During Synchronization

Note the database status for each member.

Reorder the cluster members so that a primary member with an active database is reassigned to the top of the list.

Restart the cluster: On the Status tab, select Turn Cluster Off . On the Global Settings tab, select Turn Cluster On .

On the Status tab, select Turn Cluster Off .

On the Global Settings tab, select Turn Cluster On .

Primary Site Recovery

Use the Configuration , Power , Stop Instance control.

On the Configuration , Clustering page in the UI, select the member in the Site Members table on the Status tab. Select Eject Member .

On VMware, use the Utility Console. Select Power off .

In Quorum Rules (A) , one member is not reachable from the other two. Because two members can communicate, and they constitute a majority, they continue operating as cluster members. The odd member, if operational, cannot the others, therefore it ceases to operate as a cluster member. The instance then goes into cluster recovery mode.

In Quorum Rules (B) , a network partition has divided the cluster members. Although two members can communicate with each other, they cannot form a quorum. The other three members make a quorum, and so continue as the cluster.

In Quorum Rules (C) , a network partition has divided the cluster members. The primary replication leader can only one member, while three other members can communicate among themselves. Those three members make a quorum, and so continue as the cluster.

Turn off clustering on the instance

Stop group replication

Restart the application

Send email notifications

Standard login user interface is limited to configuration, including Clustering, Diagnostics, Upgrade, and Power

If that member is managing the load-balancing VIP, the available member in the list takes up that responsibility.

Determine the replication leader by finding the member with the latest transaction

Reboot itself if it is not the leader, up to five times

Resume Cluster: Reboot all cluster members for self-healing

Eject Member: Remove an instance from the cluster. You can then diagnose and repair it apart from the cluster. Ejecting members from the Primary Site effectively reconfigures the number of members in the site. For example, in Quorum Loss (B) , if the cluster administrator ejects the unresponsive members, the remaining members make up the new cluster and can continue. This action risks losing data that could reside on the unresponsive members.

Turn off the cluster.

Determine which primary site member is the most trusted and up-to-date. See Replication Analysis . That member might be the one that failed last relative to the other members.

Reorder the most trusted member to the first member in the primary site list.

Restart the cluster from this first member.

Site Promotion Using Replication Analysis

Go to Configuration , Clustering page, Status tab and select Turn Cluster Off . A warning appears. Once you accept it, a "Turning Cluster Off" message appears.

Go to the Global Settings tab and select Replication Analysis . The Replication Analysis window opens. The page lists the site name, cluster member, and last replication ID for that member.

Determine the site and member with the highest replication ID, then close the page.

Select the site with the highest ID, then select Set Primary Site .

Move up the member in the new primary site to the first position in the list.

Select Save Config Locally .

Select Save to Cluster . When you promote a secondary site, saving the cluster might fail. To solve this problem, remove the site that is no longer reachable and save.

Select Turn Cluster On .

If you use the REST API or CLI Commands, ensure that they point to the VIP of the new primary site.

Unlock a Member When the Cluster Is Off

Unlocking permits scheduled jobs and processes to continue, which can trigger credential rotation.

If the unlocked member is not the primary member of the cluster, any changes are lost after the cluster restarts.

Application Distinct from Replication Leader

Cluster Backup and Disaster Recovery Process