Mirroring
Boot Drives with Veritas Volume Manager(VxVM)
Author: Author:
John Richardson
Date: 01/11/02
Source: http://www.sunhelpdesk.com
Credits:
This document is based upon the Sun Blueprints articles:
Authors: Gene Trantham, John S. Howard
Author: Gene Trantham
I highly recommend reading the above articles for a complete
understanding of the process.
The purpose of this document is to provide a quick reference as well as
additional methods for recovery and adittional best practices.
Introduction
Many administrators choose to use an alternative volume
manager to mirror the boot drives because of the difficulties recovering from
primary disk failure under VxVM.
Typically, Solsitce Disksuite(free product) is used to mirror
the boot drive while VxVM is used for all other storage. However, if VxVM is installed and
configured correctly, the recovery process is greatly simplified.
The problems associated with recovery are associated in part
by the way it encapsulates the boot disk(see above articles for details). We must properly install and configure
VxVM so that these issues are avoided.
In simple terms, we must first mirror the root disk. Second, we will remove the rootdisk
from rootdg and reinitialize the drive.
Third, we will add the root disk back into rootdg and re-attach the
mirror.
Note: You may
be interested in determining if a system's boot drives have already been
mirrored properly. One method to
identify if the rootdisk was properly mirrored is to look for the presence of
the "rootdisk-B0" subdisk.
This subdisk is used to map out the boot block of the rootdisk. Use the "vxprint -ht rootvol"
command to verify if it exists. If
it does exist, the boot drive has NOT been mirrored according to these
procedures.
Mirror
the Drives using VxVM
Identify the root disk and mirror disk. It is a good idea to have the drives on
different controllers if possible for improved redundancy.
# format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c0t0d0
<SUN9.0G cyl 4924 alt 2 hd 27 sec 133> <=== Root
/pci@1f,4000/scsi@3/sd@0,0
1. c0t1d0
<SUN9.0G cyl 4924 alt 2 hd 27 sec 133> <=== Root Mirror
/pci@1f,4000/scsi@3/sd@1,0
Specify disk (enter its number):
^D
Install Veritas Volume Manger according to
documentation. Only encapsulate
the root disk. Do NOT encapsulate
any other drives.
NOTE: You may
want to setup your PATH environment variable if you have not already done
so. If you installed in VxVM in
the default location you can use the following:
# PATH=$PATH:/usr/lib/vxvm/bin;
export PATH
Initialize the root mirror disk. Add it to "rootdg" group and name the disk
"rootmirror".
# vxdisksetup -i c0t1d0
# vxdg -g rootdg adddisk
rootmirror=c0t1d0
Mirror the root disk according to the partition layout.
If you have /, swap, /var, /opt, and /home defined as partitions
0,1,3, 4, and 7 respectively, you must attach the mirrors in this same exact
order. If you are not sure,
Veritas may have created notes in your /etc/vfstab which describe how the
partitions were layed before VxVM encapsulated the boot disk. It is very important to attach each
mirror in the exact numerical order as it was originaly carved on the boot
disk.
# vxrootmir rootmirror
<=== partition 0
# vxassist -g rootdg mirror
swapvol rootmirror
<===
partition 1
# vxassist -g rootdg mirror var
rootmirror
<=== partition 3
# vxassist -g rootdg mirror opt
rootmirror
<=== partition 4
# vxassist -g rootdg mirror home
rootmirror
<===
partition 7
NOTE: Be
sure to mirror any other volumes configured on the boot disk. This may be the case if you have
additional volumes configured on the boot drive. Use the "vxprint" command to list all
volumes. You can also use
"vxprint -p" to make sure there are 2 plexes associated with each
volume. Remember to modify the
remaining commands to include these additional volumes.
Disassociate and remove the existing plexes from the
rootdisk:
# vxplex -g rootdg dis rootvol-01
swapvol-01 var-01 opt-01 home-01
# vxedit -g rootdg -fr rm
rootvol-01 swapvol-01 var-01 opt-01 home-01
# vxedit rm rootdiskPriv
### May not be in rootdg - Ignore if Error
Remove the rootdisk from rootdg:
# vxdg -g rootdg rmdisk rootdisk
Re-initialize the rootdisk and add back into rootdg:
# vxdisksetup -i c0t0d0
# vxdg -g rootdg adddisk
rootdisk=c0t0d0
Re-attach the mirror in the correct order according to the
partition layout:
# vxrootmir rootdisk
# vxassist -g rootdg mirror
swapvol rootdisk
# vxassist -g rootdg mirror var
rootdisk
# vxassist -g rootdg mirror opt
rootdisk
# vxassist -g rootdg mirror home
rootdisk
Create the underlying partitions on the rootdisk and
mirrordisk - Optional Step - The benefits are worth the effort!
Update: This
step can be automated using the Veritas supplied program: vxbootsetup. Simply execute the following command:
# vxbootsetup
Or you may do it the manual way as described below:
Notice how you did NOT create the disk partitions manually
when you initialized the mirror disk.
This is because VxVM automatically does its partitioning for you. This is also true for the root disk
after it has been re-initialized and brought back into rootdg. That is, the original rootdisk
partition scheme has been replaced by Veritas's partition scheme. This means that the underlying
partitions for each volume(swap, /opt, /var, /home) are not physically layed
out on the drives corresponding to the veritas volumes. In some situations though, you might
want to be able to access the partitions corresponding to each volume by
manually mounting the slice. In a
recovery situation for example, you may want to boot from cdrom and access the
/opt partition by manually mounting the slice without the veritas drivers
loaded. This is not possible
unless you create the underlying partition mapping directly to the opt volume
using the "vxmksdpart" command.
All underlying partitions except / should be created. The / partition is automatically
created and is mapped correctly to the corresponding veritas root volume.
One Caveat, you cannot use partition 3 or 4 because this is
typically where Veritas stores is private and public region. The following is snipped from the
"format" utility listing the Veritas partitions. This shows that the root partition
exists but the swap, /var, /opt, home partitions are not mapped on the
drive(Veritas knows where they exist on the drive because of its private and
public regions):
Part Tag Flag Cylinders Size
Blocks
0 root wm 1 - 2282
3.91GB
(2282/0/0) 8194662 <===
"/" exists
1 unassigned wm 0
0 (0/0/0) 0
<=== Unassigned
2 backup wm 0 -
4923
8.43GB
(4924/0/0) 17682084
3 - wu 0 - 0 1.75MB (1/0/0)
3591
<=== Private Region
4 - wu 1 -
4923
8.43GB (4923/0/0) 17678493 <===
Public Region
5
unassigned wm 0
0 (0/0/0) 0
<=== Unassigned
6 unassigned wm 0
0 (0/0/0) 0
<=== Unassigned
7
unassigned wm 0
0 (0/0/0) 0
<=== Unassigned
You may use the unassigned partitions which are 1, 5,6, and
7 to map to the veritas volumes.
Be sure to document which partitions map to the Veritas volumes for
future reference.
The "vxmksdpart" arguments are the subdisk(which
maps directly to the volume), slice, tag and flag. Be sure to use the correct tag and flag for you particular
Solaris version. Reference the man
page for "fmthard" for correct options. Use the "vxprint" command to get the correct
subdisk names:
Rootdisk:
# vxmksdpart -g rootdg rootdisk-02
1 0x03 0x01
<=== swap
#
vxmksdpart -g rootdg rootdisk-03 5 0x07 0x00
<=== /var
# vxmksdpart -g rootdg rootdisk-04
6 0x06 0x00
<=== /opt
# vxmksdpart -g rootdg rootdisk-05
7 0x08 0x00
<=== /export/home
Specify the
primary swap partition as the dump device:
# dumpadm -d /dev/dsk/c0t0d0s1
RootMirror:
# vxmksdpart -g rootdg
rootmirror-02 1 0x03 0x01 <===
swap
# vxmksdpart -g rootdg
rootmirror-03 5 0x07 0x00 <===
/var
# vxmksdpart -g rootdg
rootmirror-04 6 0x06 0x00 <===
/opt
# vxmksdpart -g rootdg
rootmirror-05 7 0x08 0x00 <=== /export/home
For maximum redundancy, configure all private regions to
have a copy of rootdg database
# vxedit -g rootdg set nconfig=all
rootdg
# vxedit -g rootdg set nlog=all
rootdg
# vxdg flush rootdg
Configure
VxVM to use host spare rather than hot relocation
# vi /etc/rc2.d/S95vxvm-recover
Comment out the "vxrelocd
root &" and uncomment "vxsparecheck root &"
Create
OBP device alias for rootdisk and rootmirror. Set PROM to boot to the aliases.
# vxeeprom devalias rootdisk
c0t0d0s0
# vxeeprom devalias rootmirror
c0t1d0s0
# eeprom
"boot-device=rootdisk rootmirror"
Verify the correct eeprom
information:
# eeprom | grep disk
Update: The previous "vxeeprom" commands may not work for
all boot proms. If you are unable
to boot to the "rootdisk" device on reboot, you should set the device
alias using the "nvalias" command at the boot prom. This can be done by first doing a
"devalias" to display the PATH of each disk, and use the
"nvalias" command to set the rootdisk and rootmirror alias
manually. The following is an
example:
ok devalias
…
disk
/pci@1f,4000/scsi@3/disk@0,0 <===
disk is "rootdisk"
disk0
/pci@1f,4000/scsi@3/disk@0,0
disk1
/pci@1f,4000/scsi@3/disk@1,0 <===
disk1 is "rootmirror"
…
ok nvalias
rootdisk /pci@1f,4000/scsi@3/disk@0,0
ok nvalias
rootmirror /pci@1f,4000/scsi@3/disk@1,0
ok boot
How to Recover from Primary Disk Failure
Grab your backup drive from the spare parts shelf with the same
exact geometry as the failed drive. You do have one; don't you?
Configure
system to see drive
# devfsadm ### Prior to Solaris
8 try "# drvconfig; disks; devlinks"
Initialize the new drive, add to rootdg as
"newdisk", and replace old "rootdisk" with
"newdisk":
# vxdisksetup -i c0t0d0
# vxdg -g rootdg adddisk
newdisk=c0t0d0
# vxdg repldisk rootdisk=newdisk
Parition the new drive exactly the same as rootmirror:
I have not figured out a way to do this automatically
instead of using format. The
fmthard command will not work because it does not allow overlapping
partitions. The dd command should
not be used as it also copies over the Veritas diskid. In which case, there will be duplicate
diskid's on each disk causing the system not to boot. Please use the following procedure instead:
# format
; Enter 0 for the new drive
; Enter p for partition
; Enter p to view current
partition layout
; Enter s to select the
partition table of rootmirror
; Enter 1 to copy partition
table of rootmirror onto new drive
; Enter l to label disk
; Enter y to continue
; Enter q to quit partition
menu
; Enter q to quit format
utility
Re-attach the mirror for each volume:
# vxplex att rootvol rootvol-01
# vxplex att swapvol swapvol-01
# vxplex att var var-01
# vxplex att opt opt-01
# vxplex att home home-01
How
to Recover from Mirror Disk Failure
Grab your backup drive from the spare parts shelf with the
same exact geometry as the failed drive. You do have one; don't you?
Configure
system to see drive
# devfsadm ### Prior to Solaris
8 try "# drvconfig; disks; devlinks"
Initialize the new drive, add to rootdg as
"newdisk", and replace old "rootmirror" with
"newdisk"
# vxdisksetup -i c0t1d0
# vxdg -g rootdg adddisk
newdisk=c0t1d0
# vxdg repldisk rootmirror=newdisk
Parition the new drive exactly the same as rootdisk:
I have not figured out a way to do this automatically
instead of using format. The
fmthard command will not work because it does not allow overlapping
partitions. The dd command should
not be used as it also copies over the Veritas diskid. In which case, there will be duplicate
diskid's on each disk causing the system not to boot. Please use the following procedure instead:
# format
; Enter 1 for the new drive
; Enter p for partition
; Enter p to view current
partition layout
; Enter s to select the
partition table of rootdisk
; Enter 0 to copy partition
table of rootdisk onto new drive
; Enter l to label disk
; Enter y to continue
; Enter q to quit partition
menu
; Enter q to quit format
utility
Re-attach the mirror for each volume:
# vxplex att rootvol rootvol-02
# vxplex att swapvol swapvol-02
# vxplex att var var-02
# vxplex att opt opt-02
# vxplex att home home-02
