Veritas Cluster install: planning and installing InfoScale
Preparation is the most important part of a cluster installation. If you skip a step, you pay for it later, usually at three in the morning during the first real failover.
Plan before you install
- Both (or all) servers have the same operating system level, patches and kernel parameters, and the OS is on the InfoScale compatibility list on SORT.
- Shared storage is visible from every node with the same paths, and multipathing works (DMP or the OS's own).
- At least two private heartbeat networks, on separate switches or direct cables, and not routed. A third, low-priority link can use the public network.
- A virtual IP address and host name for each service group, in DNS.
- Three coordination points for I/O fencing (small SAN disks supporting SCSI-3 persistent reservations, or coordination point servers).
- Licence files ready, and the installation media downloaded.
- Root SSH between the nodes for the installer (it can set this up temporarily).
- Time synchronised on every node with NTP or chrony.
- The application is installed and tested on one node, with its data on the shared storage, and you know exactly how to start, stop and check it.
- A written checklist with every name, address and disk, reviewed by a second person.
Run the SORT installation and upgrade checks against your servers before you start; they catch missing packages and wrong settings.
Install the software
The Common Product Installer installs and configures all nodes from one of them:
cd /path/to/infoscale/media/rhel9_x86_64 # the directory for your OS
./installer -precheck node1 node2 # check the systems first
./installer # choose the edition, give the node names
# or install now and configure later:
./installer -configure
The installer asks for the cluster name and ID (unique on your network), the private links for LLT, and whether to configure fencing. Answer carefully: the cluster ID must not clash with other clusters on the same networks.
Check LLT and GAB
cat /etc/llttab
lltstat -nvv | head -20 # every node should see every link as UP
gabconfig -a # port a = GAB membership, port b = fencing, port h = VCS
GAB Port Memberships
===============================================================
Port a gen 4a1c01 membership 01
Port b gen 4a1c04 membership 01
Port h gen 4a1c07 membership 01
"membership 01" means nodes 0 and 1 are both members.
Configure fencing
./installer -fencing # or: /opt/VRTS/install/installer -fencing
vxfenadm -d # fencing mode and members
cat /etc/vxfenmode
Prepare the storage
On one node, create the disk group, volume and file system for the application (see Volume Manager). Do not add the file system to /etc/fstab: the cluster mounts it.
vxdisksetup -i disk_1
vxdisksetup -i disk_2
vxdg init oradg oradg01=disk_1 oradg02=disk_2
vxassist -g oradg make oravol 200g layout=mirror
mkfs -t vxfs /dev/vx/rdsk/oradg/oravol
mkdir -p /u01 # on every node
Build the service group
haconf -makerw
hagrp -add oragrp
hagrp -modify oragrp SystemList node1 0 node2 1
hagrp -modify oragrp AutoStartList node1 node2
hares -add ora_nic NIC oragrp
hares -modify ora_nic Device eth0
hares -add ora_ip IP oragrp
hares -modify ora_ip Device eth0
hares -modify ora_ip Address 10.0.0.50
hares -modify ora_ip NetMask 255.255.255.0
hares -add ora_dg DiskGroup oragrp
hares -modify ora_dg DiskGroup oradg
hares -add ora_vol Volume oragrp
hares -modify ora_vol DiskGroup oradg
hares -modify ora_vol Volume oravol
hares -add ora_mnt Mount oragrp
hares -modify ora_mnt MountPoint /u01
hares -modify ora_mnt BlockDevice /dev/vx/dsk/oradg/oravol
hares -modify ora_mnt FSType vxfs
hares -modify ora_mnt FsckOpt %-y
hares -link ora_ip ora_nic
hares -link ora_vol ora_dg
hares -link ora_mnt ora_vol
for r in ora_nic ora_ip ora_dg ora_vol ora_mnt; do hares -modify $r Enabled 1; done
haconf -dump -makero
hagrp -online oragrp -sys node1
hastatus -sum
Add the database and listener resources with the Oracle agent (or an Application resource with start, stop and monitor scripts for other software) and link them above the mount and IP. Then stop the application's own start-up at boot: from now on only the cluster starts it.
Before going live
Run the complete failover test and keep the results. Document the cluster in your runbook: how to check status, switch, freeze for maintenance, and whom to call.