Veritas VxVM & VxFS 3.2 – 5.0 upgrade procedure

Veritas VxVM & VxFS 3.2 – 5.0 upgrade procedure(this procedure should work for upgrading pre-4.0 versions to 4.1 and above)

** prior to starting, create the file: /etc/vx/reconfig.d/state.d/install-db

SUMMARY:

o if using EMC arrays/Powerpath for external storage, configure appropriately (see DETAILS below)
o break root mirror & unencapsulate
** (if any additional volumes have been created in the rootdg diskgroup, they should be backed up
as they will go away when the root drive is unencapsulated; it is never a good idea to
define data volumes on the boot drive in rootdg diskgroup) **
o uninstall 3.2 (remove pkgs & kill processes)
o install Storage Foundation 5.0 (and Maintenance Patch; MP1)
o if data disks are connected, verif the output of vxdisk list (the TYPE column)
o encapsulate & mirror root drive
o upgrading data disks

DETAILS:

o if using EMC arrays/Powerpath for external storage, configure appropriately
- decide if PowerPath should be upgraded or leave uninstalled to use Volume Manager DMP
(DMP will be automatically configured)
o if PowerPath will be upgraded, check with EMC for current compatible version of PowerPath
- see installation guide page 71 (Unsuppressing DMP for EMC PowerPath disks).
(http://support.veritas.com/docs/283887)

o break root mirror & unencapsulate

- … deport data disk groups …
(command: vxdg deport datadg (diskgroup)
(if the root drive is not mirrored, skip to “Unencapsulate boot drive” below)
- vxplex -g rootdg -o rm dis
(do this for each of the mirror drive plexes)
- vxdg -g rootdg rmdisk
- /usr/lib/vxvm/bin/vxdiskunsetup
- cd /etc
- ls *.prevm
(verify that vfstab & system exist, if so)
mv system system-
mv vfstab vfstab-
cp system.prevm system
cp vfstab.prevm vfstab
** (the mounts in the vfstab.prevm should match the slices in the output of:
prtvtoc /dev/rdsk/s2) **
o unencapsulate boot drive
- vxunroot (yes to reboot)
(returns boot drive to booting from native slices)

o uninstall 3.2 (remove pkgs & kill processes)
- … kill vx processes if they exist (ps -aef|grep vx) …
- cp -r /etc/vx/licenses/elm /var/tmp
- pkgrm (ls /opt/VRTS*|grep -v VRTS)
- rm -r /opt/VRTS*
- rm -r /etc/vx /var/vx
- reboot

o install SF 5.0 (and MP2 or latest release)
- … verify that no vx processes have started …
- uncompress & untar installation archive
(5.0 base release; must be downloaded from Customer Care; serial # require via license upgrade)
- run installsf
(or install ; ie … installvm for Volume Manager)
- uncompress & untar MP1 archive
(http://support.veritas.com/docs/288505)
- run installmp
(see included README and release notes)

o if data disks are connected, verify the output of vxdisk list (the TYPE column)
- auto:none (new uninitialized disks)
- auto:sliced (encapsulated disks containg slices)

o encapsulate & mirror root drive
- use vxdctl bootdg rootdg to create default rootdg disk group
- use vxdiskadm to encapsulate and mirror boot drive
(a spare drive must be initialized as sliced and belong to rootdg; vxdiskadm option 1)
(options 2 & 6)
- reboot

o upgrading data disks
- … import data disk groups …
- vxdg import datadg
- use “vxdg upgrade” to upgrade diskgroup versions
command: vxdg upgrade
- mount all data volumes
- use vxupgrade to upgrade vxfs filesystem versions one version at a time if appropriate
o use ‘fstyp | grep version’ to determine starting version
- use fsck (-o full) then vxupgrade to next version
(command: vxupgrade [-n new_version] mount_point )
(repeat until version 7 is reached)
(repititious fsck -o full can be timely but is best practice)

Sed Tips

sed (Stream EDitor) refers to a Unix utility for parsing text files and the programming language it uses to apply textual transformations to a sequential stream of data. It reads input files line by line, applying the operation which has been specified via the command line (or a sed script), and then outputs the line. Getting started with sed can be a real pain if you are unfamiliar with perl for regular expressions.

FILE SPACING:

Double space a file
sed '/^$/d;G'

Undo double-spacing (assumes even-numbered lines are always blank)
sed 'n;d'

NUMBERING:

Number each line of a file (number on left, right-aligned)
sed = filename | sed 'N; s/^/ /; s/ *(.{6,})n/1 /'

Number each line of file, but only print numbers if line is not blank
sed '/./=' filename | sed '/./N; s/n/ /'

Count lines (emulates “wc -l”)
sed -n '$='

TEXT CONVERSION AND SUBSTITUTION:

Convert DOS newlines (CR/LF) to Unix format.
sed 's/.$//'

Convert Unix newlines (LF) to DOS format.
sed 's/$'"/`echo r`/"

Delete leading whitespace (spaces, tabs) from front of each line aligns all text flush left
sed 's/^[ t]*//’

Delete trailing whitespace (spaces, tabs) from end of each line
sed 's/[ t]*$//’

Delete BOTH leading and trailing whitespace from each line
sed 's/^[ t]*//;s/[ t]*$//’

Substitute (find and replace) “foo” with “bar” on each line
sed 's/foo/bar/' # replaces only 1st instance in a line

sed 's/foo/bar/4' # replaces only 4th instance in a line

sed 's/foo/bar/g' # replaces ALL instances in a line

Change “scarlet” or “ruby” or “puce” to “red”
sed 's/scarlet/red/g;s/ruby/red/g;s/puce/red/g'

SELECTIVE PRINTING OF CERTAIN LINES:

Print first 10 lines of file (emulates behavior of “head”)
sed 10q

Print first line of file (emulates “head -1″)
sed q

Print the last 10 lines of a file (emulates “tail”)
sed -e :a -e '$q;N;11,$D;ba'

Print the last 2 lines of a file (emulates “tail -2″)
sed '$!N;$!D'

Print the last line of a file (emulates “tail -1″)
sed -n '$p'

Print section of file based on line numbers (lines 8-12, inclusive)
sed -n '8,12p'

Print only lines which match regular expression (emulates “grep”)
sed '/regexp/!d'

Grep for AAA and BBB and CCC (in any order)
sed '/AAA/!d; /BBB/!d; /CCC/!d'

Grep for AAA and BBB and CCC (in that order)
sed '/AAA.*BBB.*CCC/!d'

SELECTIVE DELETION OF CERTAIN LINES:

Delete duplicate, consecutive lines from a file (emulates “uniq”).
sed '$!N; /^(.*)n1$/!P; D'

Delete duplicate, nonconsecutive lines from a file.
sed -n 'G; s/n/&&/; /^([ -~]*n).*n1/d; s/n//; h; P’

Delete all lines except duplicate lines (emulates “uniq -d”).
sed '$!N; s/^(.*)n1$/1/; t; D'

Delete the first line of a file
sed '1d'

Delete the first 10 lines of a file
sed '1,10d'

Delete the last line of a file
sed '$d'

Delete the last 2 lines of a file
sed 'N;$!P;$!D;$d'

Delete the last 10 lines of a file
sed -n -e :a -e '1,10!{P;N;D;};N;ba'

Delete lines matching pattern
sed '/pattern/d'

Delete ALL blank lines from a file (same as “grep ‘.’ “)
sed '/^$/d'

Remove most HTML tags (accommodates multiple-line tags)
sed -e :a -e 's/<[^>]*>//g;/

SVM Mirror

An SVM Mirror is good for a couple of reasons ...

1. Faster reads (Data gets pulled off either drive to fulfill a READ request)
2. Redundancy, (If one drive fails, you don't lose data; make sure you replace it quick though, right?)


There are CONS too ...

1. Slower WRITE transactions
2. Two drives yields one drive of capacity


... but like the old saying goes, "Fast, reliable, inexpensive. Pick any two!"

When setting up SVM you need to do a number of things:

1. Setup the partitions on each drive (with the "# format" command)
2. Setup the State Database Replicas (SDR)
3. Create the Mirrors and Submirrors
4. Link them
5. Sync them
6. Test


Our example here will assume this ...

- DRIVE 0 = c1d0 = 2 SDR's
- DRIVE 1 = c2d0 = 2 SDR's


Now there are two issues with this SDR setup ...

CASE 1. During operation, what happens when a drive "fails".
CASE 2. During a reboot, What happens with "one" failed drive.


For CASE 1 ...

During operation, if either drive fails, the system will auto fail-over to the operating drive and continue normal operation. A message will likely be posted in syslog, and "SysAdmin intervention" will be required to fix the problem. Fixing can happen at your leisure, but during this time there is "NO REDUNDANCY". Search this document for "SysAdmin intervention" and you can find out what you need to do.

For CASE 2 ...

During a reboot, with either drive in a "Failed" status, "Sysadmin intervention" is required to fix things.

NOTE: SysAdmin intervention required means to delete references to SDR's on the bad disk to make the system bootable.

(This is the best we can do with a 2 drive setup. To understand why, read "Understanding the Majority Consensus Algorithm" and "Administering State Database Replicas" in the SVM manual. The best setup is an "ODD drive SVM array". (Example: 3 drives, 3 SDR's with one per drive, or 6 SDR's with 2 per disk for further redundancy.)

OVERVIEW

STEP #1 -> Repartition Drives to accommodate State Database Replica partitions

For a 320GB drive each cylinder is about 8MB. I chose 2 cylinders for each SDR. (Slice 6; size = 2 cyl = 16MB; UFS requires minimum 10MB per partition)

Here is the partition table of DRIVE 0 (320GB) ...

Volume: DRIVE0 Current partition table (original): Total disk cylinders available: 38910 + 2 (reserved cylinders)

Partition Tag Flag Cylinders Size Blocks

0 root wm 526 - 5750 40.03GB (5225/0/0) 83939625 -> / (ROOT)
1 swap wu 3 - 525 4.01GB (523/0/0) 8401995 -> /swap
2 backup wm 0 - 38909 298.07GB (38910/0/0) 625089150 -> ENTIRE DRIVE (Leave this alone)
3 unassigned wm 0 0 (0/0/0) 0
4 unassigned wm 0 0 (0/0/0) 0
5 unassigned wm 38906 - 38907 15.69MB (2/0/0) 32130 -> SDR
6 unassigned wm 38908 - 38909 15.69MB (2/0/0) 32130 -> SDR
7 home wm 5751 - 38905 253.98GB (33155/0/0) 532635075 -> /export/home
8 boot wu 0 - 0 7.84MB (1/0/0) 16065 -> GRUB Stage 1?
9 alternates wu 1 - 2 15.69MB (2/0/0) 32130 -> GRUB Stage 2?
Here is the partition table of DRIVE 1 (320GB) ...

Volume: DRIVE1 Current partition table (original): Total disk cylinders available: 38910 + 2 (reserved cylinders)

Partition Tag Flag Cylinders Size Blocks

0 root wm 526 - 5750 40.03GB (5225/0/0) 83939625 -> / (ROOT)
1 swap wu 3 - 525 4.01GB (523/0/0) 8401995 -> /swap
2 backup wm 0 - 38909 298.07GB (38910/0/0) 625089150 -> ENTIRE DRIVE (Leave this alone)
3 unassigned wm 0 0 (0/0/0) 0
4 unassigned wm 0 0 (0/0/0) 0
5 unassigned wm 38906 - 38907 15.69MB (2/0/0) 32130 -> SDR
6 unassigned wm 38908 - 38909 15.69MB (2/0/0) 32130 -> SDR
7 home wm 5751 - 38905 253.98GB (33155/0/0) 532635075 -> /export/home
8 boot wu 0 - 0 7.84MB (1/0/0) 16065 -> GRUB Stage 1?
9 alternates wu 1 - 2 15.69MB (2/0/0) 32130 -> GRUB Stage 2?
NOTE: Both drives are setup identical

STEP #2 -> Confirm Boot Order in BIOS = Boot from Disk 0 (On fail, Boot from Disk 1)

(NOTE: Don't worry if your BIOS doesn't support skipping a FAILED drive and auto-booting the next drive in ORDER. Solaris may do this automatically for you. It does for me. Remove one of the drives and boot to test it out on your system.)

STEP #3 -> Specify the master boot program for DRIVE 1

# fdisk -b /usr/lib/fs/ufs/mboot /dev/rdsk/c2d0p0 -> This means make sure the drive is "Active"
STEP #4 -> Make the Secondary Disk Bootable!

# /sbin/installgrub -m /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c2d0s0
STEP #5 -> Create the SDR's

# metadb -a -f c1d0s5 -> Create State Database Replica (slice 5 must be unmounted File system)
# metadb -a -f c1d0s6 -> Create State Database Replica (slice 6 must be unmounted File system)
# metadb -a -f c2d0s5 -> Create State Database Replica (slice 5 must be unmounted File system)
# metadb -a -f c2d0s6 -> Create State Database Replica (slice 6 must be unmounted File system)
# metadb -i -> Check your handy work
NOTE: Yes, I know, you can create SDR's as part of an existing partition because the first portion of the partition is reserved for SDR's BEFORE you actually create the partition and put data on it. If you know how to do this, go for it. In my opinion separate partitions has it's own benefits and I personally recommend it.

STEP #6 -> MIRROR BOOT(/) Partition

At this point we need to understand the nomenclature we are about to use.

d10 = this is the name we are arbitrarily assigning to DRIVE 0 / slice 0 (the root slice or partition) d20 = this is the name we are arbitrarily assigning to DRIVE 1 / slice 0 (the root slice or partition)

NOTE: You can use your own numbering scheme. I set up one that makes sense to me, and allows me to track "what's, what"! NOTE: Nevada supports "friendly names for metadevices". (Ex. Instead of "d0" you can use "Drive-1" or "whatever".)

d0 = this represents the NEW virtual drive we are creating that represents the SVM array for the "ROOT" partition.

Graphically this is what we are creating ...

D0
/ \
D10 D20
Before SVM, you would access partitions on the D10 (or D20) drive directly like this ... /dev/dsk/c1d0s0
After SVM, you will have a 'virtual' drive in place that you will access instead. (Meaning, you won't access the drives directly anymore. Get it?)

NOTE: All references in (/etc/vfstab) will be updated to point to this new drive (d0). When SVM is active, we don't want to communicate with "/dev/dsk/c1d0s0" anymore. We want to communicate with the new VIRTUAL drive "/dev/md/dsk/d0". The metaroot command updates (/etc/vfstab) for us automatically for the "ROOT" partition. For the other partitions we need to edit (/etc/vfstab) manually.

Let's get started ...

# metainit -f d10 1 1 c1d0s0 -> d10: Concat/Stripe is setup (Note: -f= force, 1 = one stripe, 1 = one slice)
# metainit d20 1 1 c2d0s0 -> d20: Concat/Stripe is setup
# metainit d0 -m d10 -> d0: Mirror is setup
# metaroot d0 -> DO THIS ONLY for "root" partition
# metastat d0 -> View current status (View your handy work!)
# reboot -> Need to reboot to effect changes
# metattach d0 d20 -> d0: submirror d20 is attached (and Sync'ing begins magically!)
# metastat d0 -> Check the Sync'ing (See?)
NOTE: Wait for Sync'ing to finish before rebooting, otherwise I think it restarts. You can test it and tell me!

STEP #7 -> MIRROR (/SWAP) Partition

# metainit -f d11 1 1 c1d0s1 -> d11: Concat/Stripe is setup
# metainit d21 1 1 c2d0s1 -> d21: Concat/Stripe is setup
# metainit d1 -m d11 -> d1: Mirror is setup
# vi /etc/vfstab -> (Edit the /etc/vfstab file so that /swap references the mirror)
"/dev/md/dsk/d1 - - swap - no -" -> Add this line to /etc/vfstab and comment out the old line. Remember, no quotes, right?

# reboot
# metattach d1 d21 -> d1: submirror d21 is attached (and Sync'ing begins magically!)
STEP #8 -> MIRROR (/export/home) partition

# umount /dev/dsk/c1d0s7 -> First umount the partition you want to mirror (-f to force)
# metainit d17 1 1 c1d0s7 -> d17: Concat/Stripe is setup
# metainit d27 1 1 c2d0s7 -> d27: Concat/Stripe is setup
# metainit d7 -m d17 -> d7: Mirror is setup
# vi /etc/vfstab -> (Edit the /etc/vfstab file so that /export/home references the mirror)
"/dev/md/dsk/d7 /dev/md/rdsk/d7 /export/home ufs 2 yes -" -> Add this line to /etc/vfstab and comment out the old line. Again, no quotes.

# mount /dev/md/dsk/d7 /export/home -> Remount this partition
# metattach d7 d27 -> d7: submirror d27 is attached (and Sync'ing begins magically!)
STEP #9 -> TIPS

# metastat d0 -> Check Status of "d0" Mirror
# metadb -d -f c1d0s6 -> If there is trouble, you can delete an SDR
EXAMPLE: Failed DRIVE 1 and "Sysadmin intervention" required ...

To Fix the problem temporarily ...

1. Power down
2. Remove Bad Drive 1
3. Boot into single user mode
4. Remove the "bad" SDR's on the 'Failed drive", Drive 1
5. Reboot (And the System should run fine, a little slow)


When you get a replacement drive ...

1. Power down
2. Insert the replacement drive (Same size, or bigger, right?)
3. Boot into multi-user mode
4. Repartition "NEW DRIVE 1" as per specs above
5. Make sure you create the SDR's as well
6. Build and link Mirrors together as per docs above
7. Resync drives as per these 3 commands ...


# metareplace -e d0 c1d0s0 -> d0: device c1d0s0 is enabled (SYNC ONE AT A TIME!)
# metareplace -e d1 c1d0s1 -> d1: device c1d0s1 is enabled (SYNC ONE AT A TIME!)
# metareplace -e d7 c1d0s7 -> d7: device c1d0s7 is enabled (SYNC ONE AT A TIME!)

NOTE: Additional commands that are handy!

# metadetach mirror submirror -> Detach a Mirror
# metaoffline mirror submirror -> Puts Submirror "OFFLINE"
# metaonline mirror submirror -> Puts Submirror "ONLINE"; Resync'ing begins immediately
# newfs /dev/rdsk/c1d0s1 -> newfs a Filesystem

NOTE SPECIAL FILES:

# pico /etc/lvm/mddb.cf -> (DO NO EDIT) records the locations of state database replicas
# pico /etc/lvm/md.cf -> (DO NO EDIT) contains auto generated config info for the default (unspecified or local) disk set
# /kernel/drv/md.conf -> (DO NO EDIT) contains the state database replica config info and is read by SVM at startup
# /etc/lvm/md.tab -> contains SVM config info that can be used to reconstruct your SVM config (Manually)
# metastat -p > /etc/lvm/md.tab -> This file created manually (just a dump to view info; save it!)
# metainit -> This commands can use the md.tab file as input to do their thing!! Like, RECOVER DATA!
# metadb -> This commands can use the md.tab file as input to do their thing!! Like, RECOVER DATA!
# metahs -> This commands can use the md.tab file as input to do their thing!! Like, RECOVER DATA!

Different ways to count the number of lines in a file

-bash-3.00$ awk 'END{print NR}' file
16
-bash-3.00$ wc -l file
16 file
-bash-3.00$ sed -n '$=' file
16
-bash-3.00$

awk one-liner Tips

Print column1, column5 and column7 of a data file or output of any columns list

$awk ‘{print $1, $5, $7}’ data_file

$cat file_name |awk ‘{print $1 $5 $7}’

$ls –al |awk ‘{print $1, $5, $7}’ -- Prints file_permissions,size and date

Syntax of running an awk program

Awk ‘program’ input file(s)

List all files names whose file size greater than zero.

$ls –al |awk ‘$5 > 0 {print $9}’

List all files whose file size equal to 512bytes.

$ls –al |awk ‘$5 == 0 {print $9}’

print all lines

$awk ‘{print }’ file_name

$awk ‘{print 0}’ file_name

Number of lines in a file

$awk ‘ END {print NR}’ file_name

Number of columns in each row of a file

$awk ‘ {print NF’} file_name

Sort the output of file and eliminate duplicate rows

$awk ‘{print $1, $5, $7}’ |sort –u

List all file names whose file size is greater than 512bytes and owner is "oracle"

$ls –al |awk ‘$3 == "oracle" && $5 > 512 {print $9}’

List all file names whose owner could be either "oracle" or "root"

$ls –al |awk ‘$3 == "oracle" || $3 == "root" {print $9}’

list all the files whose owner is not "oracle

$ls –al |awk ‘$3 != "oracle" {print $9}’

List all lines which has atlease one or more characters

$awk ‘NF > 0 {print }’ file_name

List all lines longer that 50 characters

$awk ‘length($0) > 50 ‘{print }’ file_name

List first two columns

$awk ‘{print $1, $2}’ file_name

Swap first two columns of a file and print

$awk ‘{temp = $1; $1 = $2; $2 = temp; print }’ file_name

Replace first column as "ORACLE" in a data file

$awk ‘{$1 = "ORACLE"; print }’ data_file

Remove first column values in a data file

$awk ‘{$1 =""; print }’ data_file

Calculate total size of a directory in Mb

$ls –al |awk ‘{total +=$5};END {print "Total size: " total/1024/1024 " Mb"}’

Calculate total size of a directory including sub directories in Mb

$ls –lR |awk ‘{total +=$5};END {print "Total size: " total/1024/1024 " Mb"}’

Find largest file in a directory including sub directories

$ls –lR |awk ‘{print $5 "\t" $9}’ |sort –n |tail -1

Year 2038 bug

What is the year 2038 bug?

In the first month of the year 2038 C.E. many computers will encounter a date-related bug in their operating systems and/or in the applications they run. This can result in incorrect and grossly inaccurate dates being reported by the operating system and/or applications. The effect of this bug is hard to predict, because many applications are not prepared for the resulting "skip" in reported time - anywhere from 1901 to a "broken record" repeat of the reported time at the second the bug occurs. Also, leap seconds may make some small adjustment to the actual time the bug expresses itself. I expect this bug to cause serious problems on many platforms, especially Unix and Unix-like platforms, because these systems will "run out of time". Starting at GMT 03:14:07, Tuesday, January 19, 2038, expect to see lots of systems around the world breaking magnificently: satellites falling out of orbit, massive power outages (like the 2003 North American blackout), hospital life support system failures, phone system interruptions (including 911 emergency services), banking system crashes, etc. One second after this critical second, many of these systems will have wildly inaccurate date settings, producing all kinds of unpredictable consequences. In short, many of the dire predictions for the year 2000 are much more likely to actually occur in the year 2038! Consider the year 2000 just a dry run. In case you think we can sit on this issue for another 30 years before addressing it, consider that reports of temporal echoes of the 2038 problem are already starting to appear in future date calculations for mortgages and vital statistics!.

What causes it?

What makes January 19, 2038 a special day? Unix and Unix-like operating systems do not calculate time in the Gregorian calendar, they simply count time in seconds since their arbitrary "birthday", GMT 00:00:00, Thursday, January 1, 1970 C.E. The industry-wide practice is to use a 32-bit variable for this number (32-bit signed time_t). Imagine an odometer with 32 wheels, each marked to count from 0 and 1 (for base-2 counting), with the end wheel used to indicate a positive or negative integer. The largest possible value for this integer is 2**31-1 = 2,147,483,647 (over two billion). 2,147,483,647 seconds after Unix's birthday corresponds to GMT 03:14:07, Tuesday, January 19, 2038. One second later, many Unix systems will revert to their birth date (like an odometer rollover from 999999 to 000000). Because the end bit indicating positive/negative integer may flip over, some systems may revert the date to 20:45:52, Friday, December 13, 1901 (which corresponds to GMT 00:00:00 Thursday, January 1, 1970 minus 2**31 seconds). Hence the media may nickname this the "Friday the Thirteenth Bug".



Sources:

http://en.wikipedia.org/wiki/Year_2038_problem

http://www.2038bug.com/

http://computer.howstuffworks.com/question75.htm

Error : "Error in Semaphore Operation - No space left on device"

NBU Error:
"Error in Semaphore Operation - No space left on device"

Solution: Refer Symantec TechNote.
http://support.veritas.com/docs/238063 (Requires reboot).