Testing Veritas clusters: a failover test plan

Updated November 2026

A cluster that has never been tested is not a cluster you can rely on. Run this plan before a new cluster goes live, after every configuration or software change, and at least once a year. Agree a maintenance window with the application owners first: some tests take the application down briefly.

0. Checks before testing

vxlicrep | grep -i expir                         # no licence close to expiry
grep -E 'SystemList|AutoStartList' /etc/VRTSvcs/conf/config/main.cf
hacf -verify /etc/VRTSvcs/conf/config            # configuration is valid
lltstat -nvv | head                              # every heartbeat link is UP
gabconfig -a                                     # ports a, b and h with all nodes
vxfenadm -d                                      # fencing is enabled
hastatus -sum                                    # all nodes RUNNING, nothing faulted or frozen

Each SystemList and AutoStartList should contain every node that may run the group. If the commands are not found, add /opt/VRTSvcs/bin:/opt/VRTS/bin:/sbin to root's PATH. Do not start testing until everything above is clean.

Expected starting state for a two-node failover cluster: the group ONLINE on one node and OFFLINE on the other, both nodes RUNNING, no faults.

Recording results

For every test write down the time, what you did, how long the application was unavailable, what hastatus -sum showed afterwards, and whether the result was as expected. Keep the relevant part of engine_A.log with the record.

Test 1: manual switch-over

hagrp -switch oragrp -to node2
hastatus -sum                      # repeat until the group is ONLINE on node2
hagrp -switch oragrp -to node1     # and back

Expected: resources go offline on node1 in order, then online on node2. Clients can connect to the virtual IP again within the agreed time. Ask the application owner to confirm it works on each node.

Test 2: application failure

Kill the application process on the active node (for a database, stop it with the database's own tools as an abort, not through the cluster):

pkill -9 -f ora_pmon_PROD          # example for an Oracle instance named PROD

Expected: the agent detects the failure at the next monitor interval. Depending on the resource's RestartLimit, VCS restarts it on the same node or fails the group over to the other node. The resource shows as FAULTED on the first node; clear it afterwards with hares -clear.

Test 3: public network failure

Unplug the public network cable of the active node, or disable its switch port.

Expected: the NIC resource faults and the group fails over to the other node. Reconnect the cable and clear the fault.

Disconnect one private heartbeat link.

Expected: nothing fails over. lltstat -nvv shows the link DOWN; the engine log may warn about it. Reconnect it and confirm it is UP again. If you only have one link left in total, the log reports jeopardy.

Disconnect every private link at the same time. If LLT also has a low-priority link over the public network, that link keeps the heartbeat alive and you will only see jeopardy; to test a real split, take that link down too.

Expected with fencing: the race for the coordination points decides the winner; the losing node panics and reboots, the group runs (or moves to) the surviving node, and the data stays consistent. Without fencing this test is dangerous and you should not run it on a cluster with production data, which is itself a reason to configure fencing.

Test 6: node crash

Power off or reset the active node through its management controller (iLO, iDRAC, HMC, ILOM), without a clean shutdown.

Expected: the surviving node notices the missing heartbeats, fencing removes the dead node's access to the disks, and the group comes online on the survivor. When the crashed node boots again, it rejoins the cluster with the group OFFLINE.

Test 7: clean shutdown of the active node

shutdown -r now                    # on the active node

Expected: VCS takes the group offline cleanly and brings it online on the other node before the first node goes down.

Test 8: storage path failure

If the servers have several paths to the SAN, disable one path (unplug a Fibre Channel cable or disable a switch port).

Expected: multipathing continues on the remaining paths; the application does not notice and no failover happens. vxdmpadm getsubpaths shows the failed path. Restore it and check that it is used again.

Test 9: VCS itself

hastop -local -force               # stop VCS but leave the application running
hastatus -sum                      # from the other node: this node's state

Expected: the application keeps running, and no failover happens. Start VCS again with hastart: it should find the group already online and simply take over monitoring.

After testing

hastatus -sum                      # everything back to the starting state
hares -display | grep -i faulted   # no faults left
hagrp -display -attribute Frozen   # nothing left frozen

Move the group back to its preferred node if your site requires that, and send the results to the application owners. If any test failed, see debugging tips, fix the cause and repeat the whole plan.