PAC Switchover in an M580 Hot Standby System
Original instructions
About us
Introduction
The purpose of a Hot Standby system is to be ready to perform a switchover, if needed. A switchover is the immediate transfer of control of the network from the primary PAC to the standby PAC. The transfer needs to be swift and seamless.
The M580 Hot Standby system continuously monitors ongoing system operations, and determines if a condition requiring a switchover exists. On each scan, both the primary PAC and the standby PAC check the health of the system.
The primary PAC checks the health of the following:
The standby PAC checks the following:
Before each MAST task, the primary PAC transfers to the standby PAC system, status and I/O data, including date and time data. On switchover, the standby PAC applies this time data and continues the same time stamping sequence. The maximum amount of transferable Hot Standby data depends on the CPU.
NOTE: Both the primary PAC and the standby PAC maintain independent event logs. If a switchover occurs, the events recorded in the log of the former primary PAC will not be included in the event log of the new primary (formerly the standby) PAC.
Switchover Causes
Any one of the following events will cause a switchover:
Similar to a switchover, a swap is a controlled event that transfers control of the network from the primary PAC to the standby PAC. A swap can be caused by:
Events that Do Not Cause Switchover
These events DO NOT cause a switchover:
Switchover Execution Time
If both the primary PAC and standby PAC are operating normally, the Hot Standby system detects a switchover causal event within 15_ms.
For both a safety and non-safety PAC system, the effect of the switchover on the application reaction time is:
The application response time for a swap or a switchover can be calculated.
After the switchover, the former standby PAC becomes the primary. In the worst case, the new primary PAC operates with data of scan cycle N, while the outputs have received (from the former primary PAC) data of scan cycle N+1. The new primary PAC re-evaluates outputs beginning with scan N+1.
Because the Hot Standby switchover evaluation occurs during the MAST task, some FAST task program execution may be skipped.
Switchover Effect on Main IP Address Assignments
Distributed equipment uses the Main IP address setting, configured in the IPConfig tab, to communicate over an Ethernet network with the primary CPU. On switchover, the Main IP address setting is automatically transferred from the former primary CPU to the former standby – now the new primary – CPU. Similarly, on switchover the Main IP address + 1 setting is automatically transferred from the former standby CPU to the new standby.
In this way, the configured links between the distributed equipment and the primary CPU do not need to be edited in the event of a switchover.
NOTE:
  • A switchover does not affect the assignment of IP address A or IP address B. These assignments are made exclusively by means of the A/B/Clear rotary switch on the back of the CPU, and are not affected by a change in primary or standby Hot Standby status.
  • When connecting Control Expert to the Hot Standby system, use IP address A or IP address B to maintain the connection on a switchover. Avoid using the Main IP address, because on switchover this becomes Main IP address + 1 and will disconnect Control Expert.
Switchover Effect on Remote Outputs
For RIO drops, the switchover is bumpless: the state of outputs is not affected by the switchover. During Hot Standby operations, each PAC maintains an independent, redundant owner connection with each RIO drop. Each PAC makes this connection via IP address A or IP address B, depending on the A/B/Clear rotary switch designation for its CPU. When a switchover occurs, the new primary PAC continues to communicate with I/O via its pre-existing redundant owner connection.
NOTE: The switchover may not be bumpless with respect to distributed equipment outputs.
Switchover Effect on Distributed Equipment Outputs
The behavior of distributed equipment outputs during a switchover depends on whether the equipment supports hold up time. If the device does not support hold up time, its outputs will most likely go to fallback when the connection with the primary PAC is interrupted, and will recover their state after reconnecting with the new primary PAC.
To achieve bumpless behavior, the outputs need to support a sufficiently long hold up time.
Switchover Effect on CCOTF Changes
After the standby PAC becomes the new primary, it operates using both the firmware and the application previously configured in it. If CCOTF changes were previously made to the former primary PAC that were not transferred to the former standby PAC, these changes are not included in the configuration running in the new primary PAC.
For example, assume that an I/O module was added to a remote I/O drop in the configuration running in the former primary PAC. If the changed configuration was not transferred to the former standby PAC, the added module will not be included in the configuration running in the former standby PAC when it becomes the primary PAC after switchover.
Switchover Effect on Program Logic Changes
A logic mismatch condition exists when changes have been made to the application in the primary CPU, but not to the standby CPU. If the LOGIC_MISMATCH_ALLOWED flag is set, the standby CPU can continue to operate as standby while a logic mismatch exists. In this case, if a switchover occurs, the new primary CPU executes its own, different application using data received from the former primary CPU.
Depending on the nature of the application modification, different results occur:
Modification to initial primary CPU logic:
Effect on new primary CPU program execution:
Only code is changed (no changes to variables).
All variable values exchanged between the controllers remain the same (EQUAL).
New variables were added.
The new variables are not used by the new primary CPU.
Existing variables were deleted.
The new primary CPU includes the deleted variables in program execution, and applies the most recent values to these variables.
Switchover Effects on Time Management
In an M580 Hot Standby system, the primary CPU and the standby CPU operate their own system timers, which are not automatically synchronized. Because both the primary CPU and the standby CPU share a common configuration, both can be configured to perform as NTP client or NTP server.
When the NTP client function is enabled in a Hot Standby system, the primary CPU and the standby CPU independently receive time settings from a designated NTP server.
When the NTP server is enabled in a Hot Standby system, only the primary CPUs performs the role of server.
Before each scan, the primary CPU transfers system data to the standby CPU, including the following primary CPU system time values:
On switchover, the former standby CPU – now the new primary CPU – applies the system time values sent by the former primary CPU. Thereafter, the new primary CPU continues to execute the application in the same time context as the former primary CPU. If the NTP server function is enabled for the Hot Standby system, the new primary CPU begins to perform the function of NTP server.
Switchover Effects on IPsec Connections
On switchover, the former primary BMENOC0301/11 module closes all connections that use its main IP address. These connections are re-opened on the new primary BMENOC0301/11 module using the main IP address after the two modules swap their main and main+1 IP addresses.
Because IPsec connections take a relatively long time to establish, it can take up to 5 minutes to re-establish an IPSEC connection that uses the main IP address.
Switchover Effect on Safety Operating Mode
When an M580 safety Hot Standby PAC switches from standby PAC to primary PAC, the operating mode is automatically set to safety mode.
NOTE: The operating mode setting of a safety Hot Standby PAC – either safety mode or maintenance mode – is not included in the transfer of an application from the primary PAC to the standby PAC.
Recovery of Former Primary PAC
The former primary PAC may or may not become the standby PAC, depending on cause of switchover.
If the switchover was caused by:
Make the former primary PAC the standby by:
Primary halt (non-safety PAC)
performing an INIT command and RUN the PAC
Primary halt (safety PAC - Process and/or SAFE task)
performing an INIT command (Process task) and/or an INIT_SAFETY command (SAFE task), and then RUN the PAC
PAC stop in a non-safety PAC, or in both the Process and SAFE tasks of a safety PAC
running the PAC
Primary error detected
performing a CPU RESET command
Application transfer on Primary
completing the transfer and RUN the application
Primary power off
powering up the PAC
Loss of all RIO drops (if any) while HSBY link is still healthy and Standby CPU has access to the drops
causing the PAC to recover RIO drops
DDDT command
The former primary automatically becomes the standby, provided the necessary preconditions exist, for example:
  • Firmware mismatch is allowed, if a firmware mismatch exists.
  • Logic mismatch is allowed, if a logic mismatch exists.
  • Online modifications are allowed, if modifications have been made.
Control Expert HSBY Swap button