Jan 072009

Steps to updating flashimage on Sun M5000 servers:

Download latest flash image

Download to usb drive.

I had to rename the file “[]” were put into the name on download.

I made a directory named images

Moved renamed file (FFXCP1071.tar.gz) into images directory.

(The usb port is on the XSCF card in rear of system)

Load image into system memory:

XSCF>getflashimage file:///media/usb_msd/images/FFXCP1071.tar.gz

Answer: Y (Loads image into memory)

Install image:

XSCF>flashupdate -c update -m xcp -s 1071

Answer: Y

Oct 162008

Since the past several months, we have been encountering a baffling problem with Solaris 10 systems hanging on their way up after a reboot (or simply a boot) in the multi-user mode (run level 3). Since we do our system patching at regular intervals (and since solaris runs so well that we don’t need to reboot our servers otherwise), we didn’t notice this until we came upon our latest patch cycle last month.

We noticed that our systems were not booting up the normal way. The only way to boot was to “trick” the system into booting into single-user mode and then exiting out to the multi-user run-level.

This was a problem happening only on systems running Solaris 10, with Veritas Volume Manager 5.0 MP1. So naturally we leaned towards tackling this first as a possible VxVM problem (introduced by some patch during the patch cycle). And the patch rev we had decided to apply had the Veritas Storage Foundation 5.0 MP1 RP4 in it.

Again, to reiterate the problem description —


Systems running Solaris 10 and Veritas Volume Manager tend to hang when booted with a normal init 6 (ie while booting into Run Level 3).


Boot the system in run level S (single user mode) and then exit out of it to boot into multi-user mode.


Opened a case with Veritas and sent them VRTSexplorer outputs and copies of our messages file from problem host nodeA.

After several iterations of generating explorers and veritas configuration information, Veritas still didn’t have anything substantial they could pin this issue on. They asked me to generate a crash dump of the hanging system (usually should be able to coredump a running host by breaking it and running “sync” from the OBP).

After repeated attempts at generating the core, I was unable to do so. It seems like the system hangs before the dump device is initialized/configured by the OS.

Using the work around, I enabled the solaris deadman timer (which incidentally we should have on all our servers). This involves setting the following line in the /etc/system file —

* set snooping = 1

What the deadman timer does is sents a high-priority hardware interrupt to each CPU (or Core or strand depending on the platform) and updates a counter upon successful response by the CPU. In case of a system being hung (due to hardware issues especially), this count might not increase with an increase in clock tick (interrupts aresent to CPUs every tick). When this counter is not incremented, the kernel panics and kills itself.

This didn’t work because we didn’t encounter that kind of a problem (but is a good idea to have enabled nonetheless).

Another alternative for panicking a system is as follows

Boot the server in kernel debug mode (in SPARC systems it is done by running boot -k from the OBP). This loads the kmdb module into the kernel as it boots up. Breaking the system while in kernel debug mode will not drop it into the OBP but instead launch an mdb interface.

One can generate a coredump by running $ < i="" tried="" that="" but="" no="">

[17]> $ <>
panic[cpu17]/thread=2a101a3fca0: BAD TRAP: type=9 rp=2a101a3f760 addr=0 mmu_fsr=0

sched: trap type = 0x9
pid=0, pc=0x0, sp=0x2a101a3f001, tstate=0x1606, context=0x0
g1-g7: 1011be4, 1, 1870000, 20040, 1860c00, 0, 2a101a3fca0

000002a101a3f480 unix:die+9c (9, 2a101a3f760, 0, 0, 2a101a3f540, 1901ec0)
%l0-3: 0000000000000000 0000000000000009 0000000000000009 0000000000000000
%l4-7: 0000000000000000 00000000018f1800 0000000000001606 0000000001097000
000002a101a3f560 unix:trap+6cc (2a101a3f760, 10000, 0, 0, 30003dac000, 2a101a3fca0)
%l0-3: 0000000000000000 0000000001859480 0000000000000009 0000000000000000
%l4-7: 0000000000000000 00000000018f1800 0000000000001606 0000000000010200
000002a101a3f6b0 unix:ktl0+64 (0, 20040, 1860c00, 1, 1260ce4, 1901ec0)
%l0-3: 0000030003dac000 0000000000000088 0000000000001606 0000000001020354
%l4-7: 0000000000000000 00000000018f1800 000000000000000c 000002a101a3f760
000002a101a3f800 unix:debug_enter+108 (0, a, a, 1826400, 0, 1870000)
%l0-3: 0000000001834060 00000048df61937c 0000000000000000 0000000000000000
%l4-7: 0000000000000000 00000000018f1800 0000000000000000 0000000001011b30
000002a101a3f8d0 unix:abort_seq_softintr+94 (1826400, 18f1800, 30003dac000, 2a101a3fd78, 1, 189f800)
%l0-3: 0000000001834060 00000048df61937c 0000000000000000 0000000000000000
%l4-7: 0000000000000000 00000000018f1800 0000000000000000 0000000001011b30

syncing file systems… done
skipping system dump – no dump device configured

SC Alert: Host System has Reset

SC Alert: Failed to send email alert for recent event.

SC Alert: Indicator SYS/ACT is now STANDBY BLINK

So back to the kernel debug mode again. This time I decided to try and investigate the kernel (you can do this against a coredump as well).

[17]> ::msgbuf
/pci@780/pci@0/pci@9/scsi@0 (mpt0):
mpt0 supports power management.
/pci@780/pci@0/pci@9/scsi@0 (mpt0):
DMA restricted to lower 4GB due to errata
/pci@780/pci@0/pci@9/scsi@0 (mpt0):
mpt0 Firmware version v1.9.0.0 (IR)
/pci@780/pci@0/pci@9/scsi@0 (mpt0):
mpt0: IOC Operational.
/pci@780/pci@0/pci@9/scsi@0 (mpt0):
mpt0: Initiator WWNs: 0x5080020000262858-0x508002000026285b
PCI-device: scsi@0, mpt0
mpt0 is /pci@780/pci@0/pci@9/scsi@0
sd1 at mpt0: target 0 lun 0
sd1 is /pci@780/pci@0/pci@9/scsi@0/sd@0,0
/pci@780/pci@0/pci@9/scsi@0/sd@0,0 (sd1) online
root on /pci@780/pci@0/pci@9/scsi@0/disk@0,0:a fstype ufs
px1 at root: 0x7c0 0x0
px1 is /pci@7c0
PCI Express-device: pci@0, pxb_plx5
pxb_plx5 is /pci@7c0/pci@0
PCI-device: pci@1, pxb_plx6
pxb_plx6 is /pci@7c0/pci@0/pci@1
PCI-device: pci@0, px_pci0
px_pci0 is /pci@7c0/pci@0/pci@1/pci@0
PCI-device: ide@8, uata0
uata0 is /pci@7c0/pci@0/pci@1/pci@0/ide@8
WARNING: px1: spurious interrupt from ino 0x4

sd2 at uata0: target 0 lun 0
sd2 is /pci@7c0/pci@0/pci@1/pci@0/ide@8/sd@0,0
sd3 at mpt0: target 1 lun 0
sd3 is /pci@780/pci@0/pci@9/scsi@0/sd@1,0
/pci@780/pci@0/pci@9/scsi@0/sd@1,0 (sd3) online
sd4 at mpt0: target 2 lun 0
sd4 is /pci@780/pci@0/pci@9/scsi@0/sd@2,0
/pci@780/pci@0/pci@9/scsi@0/sd@2,0 (sd4) online
sd5 at mpt0: target 3 lun 0
sd5 is /pci@780/pci@0/pci@9/scsi@0/sd@3,0
/pci@780/pci@0/pci@9/scsi@0/sd@3,0 (sd5) online
iscsi0 at root
iscsi0 is /iscsi
root remounted on /pseudo/vxio@0:0 fstype ufs
virtual-device: cnex0
cnex0 is /virtual-devices@100/channel-devices@200
PCI-device: isa@2, ebus0
ebus0 is /pci@7c0/pci@0/pci@1/pci@0/isa@2
pseudo-device: dld0
dld0 is /pseudo/dld@0
PCI-device: usb@5, ohci0
ohci0 is /pci@7c0/pci@0/pci@1/pci@0/usb@5
PCI-device: usb@6, ohci1
ohci1 is /pci@7c0/pci@0/pci@1/pci@0/usb@6
cpu0: UltraSPARC-T1 (cpuid 0 clock 1000 MHz)
cpu1: UltraSPARC-T1 (cpuid 1 clock 1000 MHz)
cpu1 initialization complete – online
cpu2: UltraSPARC-T1 (cpuid 2 clock 1000 MHz)
cpu2 initialization complete – online
cpu3: UltraSPARC-T1 (cpuid 3 clock 1000 MHz)
cpu3 initialization complete – online
cpu4: UltraSPARC-T1 (cpuid 4 clock 1000 MHz)
cpu4 initialization complete – online
cpu5: UltraSPARC-T1 (cpuid 5 clock 1000 MHz)
cpu5 initialization complete – online
cpu6: UltraSPARC-T1 (cpuid 6 clock 1000 MHz)
cpu6 initialization complete – online
cpu7: UltraSPARC-T1 (cpuid 7 clock 1000 MHz)
cpu7 initialization complete – online
cpu8: UltraSPARC-T1 (cpuid 8 clock 1000 MHz)
cpu8 initialization complete – online
cpu9: UltraSPARC-T1 (cpuid 9 clock 1000 MHz)
cpu9 initialization complete – online
cpu10: UltraSPARC-T1 (cpuid 10 clock 1000 MHz)
cpu10 initialization complete – online
cpu11: UltraSPARC-T1 (cpuid 11 clock 1000 MHz)
cpu11 initialization complete – online
cpu12: UltraSPARC-T1 (cpuid 12 clock 1000 MHz)
cpu12 initialization complete – online
cpu13: UltraSPARC-T1 (cpuid 13 clock 1000 MHz)
cpu13 initialization complete – online
cpu14: UltraSPARC-T1 (cpuid 14 clock 1000 MHz)
cpu14 initialization complete – online
cpu15: UltraSPARC-T1 (cpuid 15 clock 1000 MHz)
cpu15 initialization complete – online
cpu16: UltraSPARC-T1 (cpuid 16 clock 1000 MHz)
cpu16 initialization complete – online
cpu17: UltraSPARC-T1 (cpuid 17 clock 1000 MHz)
cpu17 initialization complete – online
cpu18: UltraSPARC-T1 (cpuid 18 clock 1000 MHz)
cpu18 initialization complete – online
cpu19: UltraSPARC-T1 (cpuid 19 clock 1000 MHz)
cpu19 initialization complete – online
cpu20: UltraSPARC-T1 (cpuid 20 clock 1000 MHz)
cpu20 initialization complete – online
cpu21: UltraSPARC-T1 (cpuid 21 clock 1000 MHz)
cpu21 initialization complete – online
cpu22: UltraSPARC-T1 (cpuid 22 clock 1000 MHz)
cpu22 initialization complete – online
cpu23: UltraSPARC-T1 (cpuid 23 clock 1000 MHz)
cpu23 initialization complete – online
USB 1.10 device (usb3eb,3301) operating at full speed (USB 1.x) on USB 1.10 root
hub: hub@1, hubd1 at bus address 2
hubd1 is /pci@7c0/pci@0/pci@1/pci@0/usb@6/hub@1
/pci@7c0/pci@0/pci@1/pci@0/usb@6/hub@1 (hubd1) online
PCI-device: pci@0,2, px_pci1
px_pci1 is /pci@7c0/pci@0/pci@1/pci@0,2
PCI-device: pci@1, pxb_plx1
pxb_plx1 is /pci@780/pci@0/pci@1
PCI-device: pci@2, pxb_plx2
pxb_plx2 is /pci@780/pci@0/pci@2
PCI-device: pci@8, pxb_plx3
pxb_plx3 is /pci@780/pci@0/pci@8
PCI-device: pci@2, pxb_plx7
pxb_plx7 is /pci@7c0/pci@0/pci@2
PCI-device: pci@8, pxb_plx8
pxb_plx8 is /pci@7c0/pci@0/pci@8
PCI-device: pci@9, pxb_plx9
pxb_plx9 is /pci@7c0/pci@0/pci@9
NOTICE: e1000g0 registered
Intel(R) PRO/1000 Network Connection, Driver Ver. 5.1.11
NOTICE: pciex8086,105e – e1000g[0] : Adapter copper link is down.
NOTICE: e1000g1 registered
Intel(R) PRO/1000 Network Connection, Driver Ver. 5.1.11
NOTICE: pciex8086,105e – e1000g[1] : Adapter copper link is down.
NOTICE: e1000g3 registered
Intel(R) PRO/1000 Network Connection, Driver Ver. 5.1.11
NOTICE: pciex8086,105e – e1000g[3] : Adapter copper link is down.
NOTICE: pciex8086,105e – e1000g[0] : Adapter 1000Mbps full duplex copper link is
NOTICE: pciex8086,105e – e1000g[1] : Adapter 1000Mbps full duplex copper link is
pseudo-device: devinfo0
devinfo0 is /pseudo/devinfo@0
NOTICE: pciex8086,105e – e1000g[3] : Adapter 1000Mbps full duplex copper link is
NOTICE: VxVM vxdmp V-5-0-34 added disk array DISKS, datype = Disk

NOTICE: VxVM vxdmp V-5-3-1700 dmpnode 300/0x0 has migrated from enclosure FAKE_E
NCLR_SNO to enclosure DISKS

NOTICE: e1000g2 registered
Intel(R) PRO/1000 Network Connection, Driver Ver. 5.1.11

Look at the process table…

[17]> ::ps
R 0 0 0 0 0 0x00000001 0000000001859480 sched
R 3 0 0 0 0 0x00020001 000006001172d838 fsflush
R 2 0 0 0 0 0x00020001 000006001172e450 pageout
R 1 0 0 0 0 0x4a004000 000006001172f068 init
R 79 1 78 78 0 0x42010000 0000060017f30028 ssmagent.bin
R 9 1 9 9 0 0x42000000 000006001292f070 svc.configd
R 7 1 7 7 0 0x42000000 000006001172c008 svc.startd
R 54 7 7 7 0 0x4a004000 0000060017e43080 vxvm-sysboot
R 56 54 7 7 0 0x4a004000 0000060017e42468 vxconfigd
R 57 56 57 57 0 0x42020000 0000060017dc8018 vxconfigd

Pick a suspect thread and dig deeper —

[17]> 0000060017e43080::findstack -v
kmdb: thread 60017e43080 isn’t in memory
[17]> 0000060017e43080::walk thread
[17]> 0000060017e43080::walk thread| ::findstack -v
stack pointer for thread 30003eba480: 2a101514ff1
[ 000002a101514ff1 cv_wait_sig_swap_core+0x130() ]
000002a1015150a1 waitid+0x484(0, 60017e42468, 0, 60017e430e8, 0, 1)
000002a101515171 waitsys32+0x10(0, 38, ffbffa80, 83, 39590, 3a57c)
000002a1015152e1 syscall_trap32+0xcc(0, 38, ffbffa80, 83, 39590, 3a57c)

[17]> 0000060017e42468::walk thread| ::findstack -v
stack pointer for thread 30003fae200: 2a101dbeff1
[ 000002a101dbeff1 cv_wait_sig_swap_core+0x130() ]
000002a101dbf0a1 waitid+0x484(0, 0, 0, 60017e424d0, 7, 1)
000002a101dbf171 waitsys32+0x10(7, 0, ffbffaa8, 3, 2909e8, 0)
000002a101dbf2e1 syscall_trap32+0xcc(7, 0, ffbffaa8, 3, 2909e8, 0)

[17]> 0000060017dc8018::walk thread | ::findstack -v
stack pointer for thread 30003eb0420: 2a10072ec21
[ 000002a10072ec21 cv_wait+0x38() ]
000002a10072ecd1 delay+0x90(1, 1870000, df39, df3a, 1, 18f0818)
000002a10072ed81 vxio`volkio_flush_cached_io+0xdc(7006ea30, 30003eb0420,
7006789c, 7006e9e8, 0, 0)
000002a10072ee31 vxio`vol_commit_flush_cached_io+0x60(60011503240, 70046d28, 0
, 0, 0, 0)
000002a10072eee1 vxio`vol_ktrans_commit+0x6fc(7006b2f0, 1, 7006789c, 7006d92c
, 60011503240, 0)
000002a10072ef91 vxio`volsioctl_real+0x4c8(70034000, 564f4c86, 0, 100003,
60011403768, 2a10072fadc)
000002a10072f0e1 fop_ioctl+0x20(60012a7f900, 564f4c86, 0, 100003, 60011403768
, 12f1770)
000002a10072f191 ioctl+0x184(1, 600115e8a50, 0, 564f4c86, 0, 564f4c86)
000002a10072f2e1 syscall_trap32+0xcc(1, 564f4c86, 0, 564f4c86, 0, 820770)

Found something in the last one, but no immediate red flags. But then I saw this (ssmagent running). SSM Agent is a SNMP agent that is used to monitor our systems and report back to Netcool, our Fault Management server.

[17]> ::ps
R 0 0 0 0 0 0x00000001 0000000001859480 sched
R 3 0 0 0 0 0x00020001 000006001172d838 fsflush
R 2 0 0 0 0 0x00020001 000006001172e450 pageout
R 1 0 0 0 0 0x4a004000 000006001172f068 init
R 79 1 78 78 0 0x42010000 0000060017f5e028 ssmagent.bin
R 43 1 42 42 0 0x42020000 0000060017db7078 dhcpagent
R 9 1 9 9 0 0x42000000 0000060012922458 svc.configd
R 7 1 7 7 0 0x42000000 000006001172c008 svc.startd
R 54 7 7 7 0 0x4a004000 000006001172cc20 vxvm-sysboot
R 56 54 7 7 0 0x4a004000 0000060017e10468 vxconfigd
R 57 56 57 57 0 0x42020000 0000060017db4018 vxconfigd
[17]> 0000060017f5e028::walk thread| ::findstack -v
stack pointer for thread 30003fb55a0: 2a101e02961
[ 000002a101e02961 sema_p+0x130() ]
000002a101e02a11 biowait+0x6c(60017dda100, 0, 1870000, 30003dbe000, 2000,
000002a101e02ac1 ufs`ufs_getpage_miss+0x2ec(60019a0f300, 40000, 4de,
600129cea20, fdba0000, 2a101e03760)
000002a101e02bc1 ufs`ufs_getpage+0x694(300014b7e00, 40000, 1, 0, 1, 3)
000002a101e02d21 fop_getpage+0x44(60019a0f300, 600114659c0, 60011403978, 3,
fdba0000, 3)
000002a101e02df1 segvn_fault+0xb04(8000, 600129cea20, 3, 2000, 40000, 0)
000002a101e02fc1 as_fault+0x4c8(600129cea20, 600129d9200, fdba0000,
60011736320, 189eb00, 0)
000002a101e030d1 pagefault+0x68(fdba14a8, 0, 3, 0, 60017f5e028, 600117362a8)
000002a101e03191 trap+0xd50(2a101e03b90, 10000, 0, 3, fdba14a8, 0)
000002a101e032e1 utl0+0x4c(ff3f40fc, ff3f5a70, 1, 0, ff3f4910, 821)
stack pointer for thread 30003f08820: 2a102005091
[ 000002a102005091 cv_wait_sig_swap_core+0x130() ]
000002a102005141 lwp_park+0x130(0, 1, 30003f089c6, 30003f08820, 0, 100000)
000002a102005231 syslwp_park+0x54(0, 0, 0, 0, ff092010, 1)
000002a1020052e1 syscall_trap32+0xcc(0, 0, 0, 0, ff092010, 1)
stack pointer for thread 30003fb4560: 2a101d9f091
[ 000002a101d9f091 cv_wait_sig_swap_core+0x130() ]
000002a101d9f141 lwp_park+0x130(0, 1, 30003fb4706, 30003fb4560, 0, 100000)
000002a101d9f231 syslwp_park+0x54(0, 0, 0, 0, ff092020, 1)
000002a101d9f2e1 syscall_trap32+0xcc(0, 0, 0, 0, ff092020, 1)
stack pointer for thread 30003ef77c0: 2a1008e9091
[ 000002a1008e9091 cv_wait_sig_swap_core+0x130() ]
000002a1008e9141 lwp_park+0x130(0, 1, 30003ef7966, 30003ef77c0, 0, 100000)
000002a1008e9231 syslwp_park+0x54(0, 0, 0, 0, ff092030, 1)
000002a1008e92e1 syscall_trap32+0xcc(0, 0, 0, 0, ff092030, 1)
stack pointer for thread 30003fd35c0: 2a101db6f91
[ 000002a101db6f91 cv_timedwait_sig+0x16c() ]
000002a101db7041 cv_waituntil_sig+0x8c(60017dc8592, 60017dc8558, 2a101db7ad0,
2, 18f0800, 2)
000002a101db7111 poll_common+0x4e8(60012996580, 60017f5e028, 2a101db7ad0, 0,
fe57bcd0, 2)
000002a101db7201 pollsys+0xf8(fe57bcd0, 1, fe57bd70, 0, 2a101db7ad0, 0)
000002a101db72e1 syscall_trap32+0xcc(fe57bcd0, 1, fe57bd70, 0, fe57bd70, 0)
stack pointer for thread 30003efa460: 2a101df3091
[ 000002a101df3091 cv_wait_sig_swap_core+0x130() ]
000002a101df3141 lwp_park+0x130(0, 1, 30003efa606, 30003efa460, 0, 100000)
000002a101df3231 syslwp_park+0x54(0, 0, 0, 0, ff092050, 1)
000002a101df32e1 syscall_trap32+0xcc(0, 0, 0, 0, ff092050, 1)
stack pointer for thread 30003fe55e0: 2a10089efc1
[ 000002a10089efc1 cv_timedwait_sig+0x16c() ]
000002a10089f071 cv_waituntil_sig+0x8c(30003fe5786, 30003fe5788, 2a10089fa10,
2, 18f0800, 2)
000002a10089f141 lwp_park+0x130(fdffbd50, 0, 30003fe5786, 30003fe55e0, 0,
000002a10089f231 syslwp_park+0x54(0, fdffbd50, 0, 0, ff092060, 1)
000002a10089f2e1 syscall_trap32+0xcc(0, fdffbd50, 0, 0, ff092060, 1)
stack pointer for thread 30003ff4a20: 2a10083f091
[ 000002a10083f091 cv_wait_sig_swap_core+0x130() ]
000002a10083f141 lwp_park+0x130(0, 0, 30003ff4bc6, 30003ff4a20, 0, 100000)
000002a10083f231 syslwp_park+0x54(0, 0, 0, 0, ff092070, 1)
000002a10083f2e1 syscall_trap32+0xcc(0, 0, 0, 0, ff092070, 1)

It wasn’t vxconfigd, but was ssmagent (a Netcool/Micromuse/IBM SNMP monitoring agent) instead that was sitting in a blocked i/o wait state (biowait) and effectively preventing VxVM from starting up the rootdg and mounting the encapsulated volumes.
An svcadm disable ssmagent from the running OS, fixed the problem and system now boots just fine.

 Posted by at 3:20 pm
Aug 222008

It’s been a while since I’ve made any posts. I am looking to migrate the site shortly (host it privately). But here’s one I thought I’d share with my thousands of anonymous readers.

Really folks…if you like what you read here…post a few lines. If you hate it…post that too!

Global Zone configuration

Add the oracle user and dba group

vistatst-01:$() # grep dba /etc/group<br /><span class="anchor" id="line-11"></span>dba::600:<br /><span class="anchor" id="line-12"></span>vistatst-01:$() # grep oracle /etc/passwd<br /><span class="anchor" id="line-13"></span>oracle:x:101:600:Oracle DBA:/u01/app/oracle:/usr/bin/ksh<br /><span class="anchor" id="line-14"></span>

Configure LUNs for Oracle ASM formatting

c4t6006016030C01C00EC2DC1FE8052DD11d0s2 is ASM disk...Let's inspect it's partition table...<br /><span class="anchor" id="line-20"></span>       2      5    01          0 104852480 104852479<br /><span class="anchor" id="line-21"></span>       6      0    00       2560 104849920 104852479<br /><span class="anchor" id="line-22"></span><br /><span class="anchor" id="line-23"></span>c4t6006016030C01C009698C4458152DD11d0s2 is ASM disk...Let's inspect it's partition table...<br /><span class="anchor" id="line-24"></span>       2      5    01          0 104852480 104852479<br /><span class="anchor" id="line-25"></span>       6      0    00       2560 104849920 104852479<br /><span class="anchor" id="line-26"></span><br /><span class="anchor" id="line-27"></span>c4t6006016030C01C0046A8487A8152DD11d0s2 is ASM disk...Let's inspect it's partition table...<br /><span class="anchor" id="line-28"></span>       2      5    01          0 104852480 104852479<br /><span class="anchor" id="line-29"></span>       6      0    00       2560 104849920 104852479<br /><span class="anchor" id="line-30"></span><br /><span class="anchor" id="line-31"></span>c4t6006016030C01C00CEF5BFAA8152DD11d0s2 is ASM disk...Let's inspect it's partition table...<br /><span class="anchor" id="line-32"></span>       2      5    01          0 104852480 104852479<br /><span class="anchor" id="line-33"></span>       6      0    00       2560 104849920 104852479<br /><span class="anchor" id="line-34"></span><br /><span class="anchor" id="line-35"></span>vistatst-01:$() #<br /><span class="anchor" id="line-36"></span>

(!) Here we have chosen to set up slice 6 as the “disk” with cylinder 0 being excluded by manually partitioning it. The starting sector count (as you can observe is 2560 instead of sector 0).

Make oracle:dba the owner of these LUNS.

vistatst-01:$() # for i in `vxdisk list|grep invalid|awk '{print $1}><br /><span class="anchor" id="line-44"></span>lrwxrwxrwx   1 oracle   dba           67 Jul 15 16:50 /dev/rdsk/c4t6006016030C01C00EC2DC1FE8052DD11d0s2 -> ../../devices/scsi_vhci/ssd@g6006016030c01c00ec2dc1fe8052dd11:c,raw<br /><span class="anchor" id="line-45"></span><br /><span class="anchor" id="line-46"></span>lrwxrwxrwx   1 oracle   dba           67 Jul 15 16:50 /dev/rdsk/c4t6006016030C01C009698C4458152DD11d0s2 -> ../../devices/scsi_vhci/ssd@g6006016030c01c009698c4458152dd11:c,raw<br /><span class="anchor" id="line-47"></span><br /><span class="anchor" id="line-48"></span>lrwxrwxrwx   1 oracle   dba           67 Jul 15 16:50 /dev/rdsk/c4t6006016030C01C0046A8487A8152DD11d0s2 -> ../../devices/scsi_vhci/ssd@g6006016030c01c0046a8487a8152dd11:c,raw<br /><span class="anchor" id="line-49"></span><br /><span class="anchor" id="line-50"></span>lrwxrwxrwx   1 oracle   dba           67 Jul 15 16:50 /dev/rdsk/c4t6006016030C01C00CEF5BFAA8152DD11d0s2 -> ../../devices/scsi_vhci/ssd@g6006016030c01c00cef5bfaa8152dd11:c,raw<br /><span class="anchor" id="line-51"></span>

And change the ownerships of the underlying files…

crw-r-----   1 oracle   dba      118, 426 Aug 14 13:41 devices/scsi_vhci/ssd@g6006016030c01c00ec2dc1fe8052dd11:c,raw<br /><span class="anchor" id="line-57"></span>crw-r-----   1 oracle   dba      118, 418 Aug 14 13:41 devices/scsi_vhci/ssd@g6006016030c01c009698c4458152dd11:c,raw<br /><span class="anchor" id="line-58"></span>crw-r-----   1 oracle   dba      118, 402 Aug 14 13:41 devices/scsi_vhci/ssd@g6006016030c01c0046a8487a8152dd11:c,raw<br /><span class="anchor" id="line-59"></span>crw-r-----   1 oracle   dba      118, 434 Aug 14 13:41 devices/scsi_vhci/ssd@g6006016030c01c00cef5bfaa8152dd11:c,raw<br /><span class="anchor" id="line-60"></span>

Configure Solaris Non-Global Zone (NGZ)

vistatst-01:$() # zonecfg -z vistatst-01-z01 set limitpriv=def<span class="anchor" id="line-66"></span>ault,proc_priocntl<br />vistatst-01:$() # zonecfg -z vistatst-01-z01 export<br /><span class="anchor" id="line-67"></span>create -b<br /><span class="anchor" id="line-68"></span>set zonepath=/zonefiles/vistatst-01-z01<br /><span class="anchor" id="line-69"></span>set autoboot=true<br /><span class="anchor" id="line-70"></span>set limitpriv=default,proc_priocntl<br /><span class="anchor" id="line-71"></span>set ip-type=shared<br /><span class="anchor" id="line-72"></span>add net<br /><span class="anchor" id="line-73"></span>set address=<br /><span class="anchor" id="line-74"></span>set physical=ce0<br /><span class="anchor" id="line-75"></span>end<br /><span class="anchor" id="line-76"></span>add device<br /><span class="anchor" id="line-77"></span>set match=/dev/rdsk/c4t6006016030C01C00EC2DC1FE8052DD11d0s6<br /><span class="anchor" id="line-78"></span>end<br /><span class="anchor" id="line-79"></span>add device<br /><span class="anchor" id="line-80"></span>set match=/dev/dsk/c4t6006016030C01C00EC2DC1FE8052DD11d0s6<br /><span class="anchor" id="line-81"></span>end<br /><span class="anchor" id="line-82"></span>add device<br /><span class="anchor" id="line-83"></span>set match=/dev/rdsk/c4t6006016030C01C009698C4458152DD11d0s6<br /><span class="anchor" id="line-84"></span>end<br /><span class="anchor" id="line-85"></span>add device<br /><span class="anchor" id="line-86"></span>set match=/dev/dsk/c4t6006016030C01C009698C4458152DD11d0s6<br /><span class="anchor" id="line-87"></span>end<br /><span class="anchor" id="line-88"></span>add device<br /><span class="anchor" id="line-89"></span>set match=/dev/rdsk/c4t6006016030C01C0046A8487A8152DD11d0s6<br /><span class="anchor" id="line-90"></span>end<br /><span class="anchor" id="line-91"></span>add device<br /><span class="anchor" id="line-92"></span>set match=/dev/dsk/c4t6006016030C01C0046A8487A8152DD11d0s6<br /><span class="anchor" id="line-93"></span>end<br /><span class="anchor" id="line-94"></span>add device<br /><span class="anchor" id="line-95"></span>set match=/dev/rdsk/c4t6006016030C01C00CEF5BFAA8152DD11d0s6<br /><span class="anchor" id="line-96"></span>end<br /><span class="anchor" id="line-97"></span>add device<br /><span class="anchor" id="line-98"></span>set match=/dev/dsk/c4t6006016030C01C00CEF5BFAA8152DD11d0s6<br /><span class="anchor" id="line-99"></span>end<br /><span class="anchor" id="line-100"></span>add dataset<br /><span class="anchor" id="line-101"></span>set name=vinmarttstpool<br /><span class="anchor" id="line-102"></span>end<br /><span class="anchor" id="line-103"></span>vistatst-01:$() #<br /><span class="anchor" id="line-104"></span>vistatst-01:$() # zoneadm -z vistatst-01-z01 reboot<br /><span class="anchor" id="line-105"></span>

(!) Set up CSSD to start up

/!\ If CSSD doesn’t start up at this point — run the following command from $ORACLE_HOME/bin

#./localconfig reset<br /><span class="anchor" id="line-113"></span>vinmartst-01:$(bin) # ./localconfig reset<br /><span class="anchor" id="line-114"></span>Successfully accumulated necessary OCR keys.<br /><span class="anchor" id="line-115"></span>Creating OCR keys for user 'root', privgrp 'root'..<br /><span class="anchor" id="line-116"></span>Operation successful.<br /><span class="anchor" id="line-117"></span>Configuration for local CSS has been initialized<br /><span class="anchor" id="line-118"></span><br /><span class="anchor" id="line-119"></span>Adding to inittab<br /><span class="anchor" id="line-120"></span>Aug 14 11:58:36 vinmartst-01 root: Oracle Cluster Synchronization Service starting by user request.<br /><span class="anchor" id="line-121"></span>Startup will be queued to init within 30 seconds.<br /><span class="anchor" id="line-122"></span>Checking the status of new Oracle init process...<br /><span class="anchor" id="line-123"></span>Expecting the CRS daemons to be up within 600 seconds.<br /><span class="anchor" id="line-124"></span>Aug 14 11:58:36 vinmartst-01 root: Cluster Ready Services completed waiting on dependencies.<br /><span class="anchor" id="line-125"></span>CSS is active on these nodes.<br /><span class="anchor" id="line-126"></span>        vinmartst-01<br /><span class="anchor" id="line-127"></span>CSS is active on all nodes.<br /><span class="anchor" id="line-128"></span>Oracle CSS service is installed and running under init(1M)<br /><span class="anchor" id="line-129"></span>

Verify that cssd is running in the process table —

vinmartst-01:$(bin) # ps -ef|grep css<br /><span class="anchor" id="line-135"></span>  oracle  9120  8207   0 11:58:36 ?           0:00 /u01/app/oracle/product/10.2.0/db_1/bin/ocssd.bin<br /><span class="anchor" id="line-136"></span>    root  9220  8373   0 11:59:59 console     0:00 grep css<br /><span class="anchor" id="line-137"></span>

Verify that the LUNs are visible to the NGZ —

vinmartst-01:$(.oracle) # ls -alrt /dev/rdsk<br /><span class="anchor" id="line-143"></span>total 4<br /><span class="anchor" id="line-144"></span>drwxr-xr-x   2 root     root         512 Aug 12 16:34 .<br /><span class="anchor" id="line-145"></span>drwxr-xr-x  12 root     root        1024 Aug 14 11:57 ..<br /><span class="anchor" id="line-146"></span>crw-r-----   1 oracle   dba      118, 422 Aug 14 13:46 c4t6006016030C01C009698C4458152DD11d0s6<br /><span class="anchor" id="line-147"></span>crw-r-----   1 oracle   dba      118, 430 Aug 14 13:55 c4t6006016030C01C00EC2DC1FE8052DD11d0s6<br /><span class="anchor" id="line-148"></span>crw-r-----   1 oracle   dba      118, 438 Aug 14 14:10 c4t6006016030C01C00CEF5BFAA8152DD11d0s6<br /><span class="anchor" id="line-149"></span>crw-r-----   1 oracle   dba      118, 406 Aug 14 14:10 c4t6006016030C01C0046A8487A8152DD11d0s6<br /><span class="anchor" id="line-150"></span>

vinmartst-01:$(.oracle) # ls -lart /dev/dsk<br /><span class="anchor" id="line-154"></span>total 4<br /><span class="anchor" id="line-155"></span>brw-r-----   1 oracle   dba      118, 430 Aug 12 16:34 c4t6006016030C01C00EC2DC1FE8052DD11d0s6<br /><span class="anchor" id="line-156"></span>brw-r-----   1 oracle   dba      118, 438 Aug 12 16:34 c4t6006016030C01C00CEF5BFAA8152DD11d0s6<br /><span class="anchor" id="line-157"></span>brw-r-----   1 oracle   dba      118, 422 Aug 12 16:34 c4t6006016030C01C009698C4458152DD11d0s6<br /><span class="anchor" id="line-158"></span>brw-r-----   1 oracle   dba      118, 406 Aug 12 16:34 c4t6006016030C01C0046A8487A8152DD11d0s6<br /><span class="anchor" id="line-159"></span>drwxr-xr-x   2 root     root         512 Aug 12 16:34 .<br /><span class="anchor" id="line-160"></span>drwxr-xr-x  12 root     root        1024 Aug 14 11:57 ..<br /><span class="anchor" id="line-161"></span>

/!\ Run Oracle Installer and do the rest of the stuff that needs to be done!


 Posted by at 5:16 pm
May 232008

This is a study we’d (a former colleague of mine and yours truly) done last year of a log analysis tool called Splunk
What you will read in this article are the results/excerpts of that study.

Some of the questions we asked are as follows —

  • Why use a log analysis tool?
  • What do most shops use
  • What does a tool such as splunk buy us (as an IT shop)
  • What are it’s benefits and pit-falls?
  • What is the cost of ownership?

Why use a log-analysis tool?

The biggest reason to use such a tool would be to move from a Reactive to Proactive Systems Management paradigm

With the number of systems (about 900+ *nix servers in that shop) and the criticality (many systems cost millions of dollars in down-time) of availability of these, it is imperative to find a tool that can actually be used quickly and effortlessly to analyze valuable log information

If such a tool can look at various layers of a “delivered stack” (aka hardware, os, application, network, san, etc), it would be a gold-mine by virtue of being able to link the stack “end-to-end” and by speeding up the analysis process.

What do most shops use?

Most shops I’ve been in do log analysis like this —

a) Don’t do any log analysis unless absolutely required. And if it is required, admins log into the individual servers and parse through the logs using vi (or using a combination of grep/awk/sed if they are script-savvy)

b) Have a centralized ssh (or god forbid! rsh) trusted admin host from where they launch a log parser script that filters specific key words and that gets emailed to a mailbox or to the individual admins’ email boxes

c) have a centralized log host where they run a script akin to the one mentioned above

I’ve worked in shops of varying sizes — from a ISP/Telecom giant who ran 4000+ sun servers to a 50-server tiny sweatshop. Most of the shops I’ve been in fall some where in between (with hosts ranging from 200 – 1000 in number). That’s a lot of hosts to manage and a lot of logging that needs to be parsed.

What does a log-analysis tool buy an IT shop?

You’ve all probably thought about this — a centralized, easy-to-use log analysis tool buys an IT shop valuable time!

So what does Splunk claim to do?

In there own words —

“The Splunk Server indexes IT data from ANY source. No need to configure it for specific formats, write regular expressions or change your logging output. Search mountains of data by time, keywords, type of event, source, host or relationships to other events. “

Some key features of Splunk:

  • Universal Indexing
  • Can index terabytes of data all from one place
  • Capable of indexing approx. 22,000 events/second at density of 150 bytes/event.

How does splunk acquire data?

Access data from any live source:

  • Mounted files: NFS/SMB, CIFS/AFP, NAS/SAN, FIFO,
  • Remote files: rsync, scp/ftp/rcp,
  • Network ports: UDP & TCP, syslog/syslog-ng, log4j/log4php, JMX/JMS, SNMP
  • Databases: SQL/ODBC
  • Splunk Servers: Access data locally on production hosts and forward it to another Splunk Server over SSL/TCP

The actual evaluation results will be the next article.

 Posted by at 6:45 pm
Mar 072008

/!\ Remember to delete the service maps for NFS using svccfg command

With Solaris 10 and VCS 5.x, nfsd HAS TO run under VCS control. In order to achieve that, the following needs ton happen (on every node that will host the NFS share) —

Disable/Delete the NFS services from SMF

<span class="anchor" id="line-11"></span># svccfg delete -f svc:/network/nfs/server:default<br /><span class="anchor" id="line-12"></span># svccfg delete -f svc:/network/nfs/status:default<br /><span class="anchor" id="line-13"></span># svccfg delete -f svc:/network/nfs/nlockmgr:default<br /><span class="anchor" id="line-14"></span>

Manually restart lockd, statd and automountd

# /usr/lib/nfs/lockd<br /><span class="anchor" id="line-20"></span># /usr/lib/nfs/statd<br /><span class="anchor" id="line-21"></span># /usr/lib/fs/autofs/automount<br /><span class="anchor" id="line-22"></span># /usr/lib/autofs/automountd<br /><span class="anchor" id="line-23"></span>

NOTE: In this example (see below), the NFSgrp is configured only for one node. To add another node, add the node name and number to SystemList and AutoStartList

       group NFSgrp (<br />               SystemList = { hostA = 0 }<br />               AutoStartList = { hostA }<br /><span class="anchor" id="line-30"></span>        )<br /><span class="anchor" id="line-31"></span><br /><span class="anchor" id="line-32"></span>        DiskGroup nfsDG (<br /><span class="anchor" id="line-33"></span>               Critical = 0<br /><span class="anchor" id="line-34"></span>               DiskGroup = testdg<br /><span class="anchor" id="line-35"></span>        )<br /><span class="anchor" id="line-36"></span><br /><span class="anchor" id="line-37"></span>        Volume nfsVOL (<br /><span class="anchor" id="line-38"></span>              Critical = 0<br /><span class="anchor" id="line-39"></span>              Volume = testnfshome<br /><span class="anchor" id="line-40"></span>              DiskGroup = testdg<br /><span class="anchor" id="line-41"></span>        )<br /><span class="anchor" id="line-42"></span><br /><span class="anchor" id="line-43"></span><br /><span class="anchor" id="line-44"></span>        IP IPres (<br /><span class="anchor" id="line-45"></span>                Device = bge0<br />               Address = ""<br /><span class="anchor" id="line-47"></span>                NetMask = ""<br /><span class="anchor" id="line-48"></span>        )<br /><span class="anchor" id="line-49"></span>        Mount Mountres (<br /><span class="anchor" id="line-50"></span>                MountPoint = "/nfs/testnfs"<br /><span class="anchor" id="line-51"></span>                BlockDevice = "/dev/vx/dsk/testdg/testnfshome"<br /><span class="anchor" id="line-52"></span>                FSType = vxfs<br /><span class="anchor" id="line-53"></span>                MountOpt = rw<br /><span class="anchor" id="line-54"></span>                FsckOpt = "-y"<br /><span class="anchor" id="line-55"></span>        )<br /><span class="anchor" id="line-56"></span>        NFS NFSres (<br /><span class="anchor" id="line-57"></span>                Nservers = 16<br /><span class="anchor" id="line-58"></span>        )<br /><span class="anchor" id="line-59"></span><br /><span class="anchor" id="line-60"></span>        NFSLock NFSLockres (<br /><span class="anchor" id="line-61"></span>                PathName = "/nfs/testnfs"<br /><span class="anchor" id="line-62"></span>        )<br /><span class="anchor" id="line-63"></span>        NIC NICres (<br /><span class="anchor" id="line-64"></span>                Device = bge0<br /><span class="anchor" id="line-65"></span>        )<br /><span class="anchor" id="line-66"></span>        Share Shareres (<br /><span class="anchor" id="line-67"></span>                PathName = "/nfs/testnfs"<br /><span class="anchor" id="line-68"></span>                Options = "-o rw -d \"test home dirs\""<br /><span class="anchor" id="line-69"></span>        )<br /><span class="anchor" id="line-70"></span><br /><span class="anchor" id="line-71"></span><br /><span class="anchor" id="line-72"></span><br /><span class="anchor" id="line-73"></span>        // IPres requires Shareres<br /><span class="anchor" id="line-74"></span>        IPres requires NICres<br /><span class="anchor" id="line-75"></span>        nfsVOL requires nfsDG<br /><span class="anchor" id="line-76"></span>        Mountres requires nfsVOL<br /><span class="anchor" id="line-77"></span>        NFSLockres requires Mountres<br /><span class="anchor" id="line-78"></span>        Shareres requires NFSLockres<br /><span class="anchor" id="line-79"></span>        Shareres requires NFSres<br /><span class="anchor" id="line-80"></span><br /><span class="anchor" id="line-81"></span>        // resource dependency tree<br /><span class="anchor" id="line-82"></span>        //<br /><span class="anchor" id="line-83"></span>        // group NFSgrp<br /><span class="anchor" id="line-84"></span>        // {<br /><span class="anchor" id="line-85"></span>        // IP IPres<br /><span class="anchor" id="line-86"></span>        //      {<br /><span class="anchor" id="line-87"></span>        //      NIC NICres<br /><span class="anchor" id="line-88"></span>        //      Share Shareres<br /><span class="anchor" id="line-89"></span>        //          {<br /><span class="anchor" id="line-90"></span>        //          NFSLock NFSLockres<br /><span class="anchor" id="line-91"></span>        //              {<br /><span class="anchor" id="line-92"></span>        //              Mount Mountres<br /><span class="anchor" id="line-93"></span>        //                  {<br /><span class="anchor" id="line-94"></span>        //                  Volume nfsVOL<br /><span class="anchor" id="line-95"></span>        //                      {<br /><span class="anchor" id="line-96"></span>        //                      DG nfsDG<br /><span class="anchor" id="line-97"></span>        //                      }<br /><span class="anchor" id="line-98"></span>        //                  }<br /><span class="anchor" id="line-99"></span>        //              }<br /><span class="anchor" id="line-100"></span>        //          NFS NFSres<br /><span class="anchor" id="line-101"></span>        //          }<br /><span class="anchor" id="line-102"></span>        //       }<br /><span class="anchor" id="line-103"></span>        // }<br />

 Posted by at 7:17 pm
Jan 102008

Solaris8 BrandZ pre-requisites

Requires kernel patch 127111-05 (or latest version) for sparc. Find all dependencies and fulfill then (ie patch requirements).

# ls<br /><span class="anchor" id="line-8"></span>SUNWs8brandr  SUNWs8brandu  SUNWs8p2v<br /><span class="anchor" id="line-9"></span># pwd<br /><span class="anchor" id="line-10"></span>/mypool/software/sol8p2v/s8ma-1_0-rr/Product<br /><span class="anchor" id="line-11"></span><br /><span class="anchor" id="line-12"></span># pkgadd -d .<br /><span class="anchor" id="line-13"></span><br /><span class="anchor" id="line-14"></span>The following packages are available:<br /><span class="anchor" id="line-15"></span>  1  SUNWs8brandr     Solaris 8 Migration Assistant: solaris8 brand support (Root)<br /><span class="anchor" id="line-16"></span>                      (sparc) 11.10.0,REV=2007.<br /><span class="anchor" id="line-17"></span>  2  SUNWs8brandu     Solaris 8 Migration Assistant: solaris8 brand support (Usr)<br /><span class="anchor" id="line-18"></span>                      (sparc) 11.10.0,REV=2007.<br /><span class="anchor" id="line-19"></span>  3  SUNWs8p2v        Solaris 8 p2v Tool<br /><span class="anchor" id="line-20"></span>                      (sparc) 11.10.0,REV=2007.<br /><span class="anchor" id="line-21"></span><br /><span class="anchor" id="line-22"></span>Select package(s) you wish to process (or 'all' to process<br /><span class="anchor" id="line-23"></span>all packages). (default: all) [?,??,q]:<br /><span class="anchor" id="line-24"></span>

The SUNWs8brandr and SUNWs8brandu packages need to be added to the Solaris 10 Host OS (Global Zone).

Zone configuration

Then configure the Zone —

# zonecfg -z s8-zone<br /><span class="anchor" id="line-34"></span>s8-zone: No such zone configured<br /><span class="anchor" id="line-35"></span>Use 'create' to begin configuring a new zone.<br /><span class="anchor" id="line-36"></span>zonecfg:s8-zone> create -t SUNWsolaris8<br /><span class="anchor" id="line-37"></span>zonecfg:s8-zone> set zonepath=/mypool/zones/s8-zone<br /><span class="anchor" id="line-38"></span>zonecfg:s8-zone><br /><span class="anchor" id="line-39"></span>zonecfg:s8-zone> set autoboot=true<br /><span class="anchor" id="line-40"></span>zonecfg:s8-zone> add net<br /><span class="anchor" id="line-41"></span>zonecfg:s8-zone:net> set address=<br /><span class="anchor" id="line-42"></span>zonecfg:s8-zone:net> set physical=bge1<br /><span class="anchor" id="line-43"></span>zonecfg:s8-zone:net> end<br /><span class="anchor" id="line-44"></span>zonecfg:s8-zone> add fs<br /><span class="anchor" id="line-45"></span>zonecfg:s8-zone:fs> set type=zfs<br /><span class="anchor" id="line-46"></span>zonecfg:s8-zone:fs> set dir=/mypool/vol1<br /><span class="anchor" id="line-47"></span>zonecfg:s8-zone:fs> end<br /><span class="anchor" id="line-48"></span>special not specified<br /><span class="anchor" id="line-49"></span>zonecfg:s8-zone:fs> set special=share/zone/s8-zone<br /><span class="anchor" id="line-50"></span>zonecfg:s8-zone:fs> end<br /><span class="anchor" id="line-51"></span>zonecfg:s8-zone><br /><span class="anchor" id="line-52"></span><br /><span class="anchor" id="line-53"></span>zonecfg:sol8zone> add attr<br /><span class="anchor" id="line-54"></span>zonecfg:sol8zone:attr> set name=hostid<br /><span class="anchor" id="line-55"></span>zonecfg:sol8zone:attr> set type=string<br /><span class="anchor" id="line-56"></span>zonecfg:sol8zone:attr> set value=8325f14d<br /><span class="anchor" id="line-57"></span>zonecfg:sol8zone:attr> end<br /><span class="anchor" id="line-58"></span>zonecfg:sol8zone> verify<br /><span class="anchor" id="line-59"></span>zonecfg:sol8zone> commit<br /><span class="anchor" id="line-60"></span>zonecfg:sol8zone> exit<br /><span class="anchor" id="line-61"></span>dwailsun:$() # zonecfg -z sol8zone info<br /><span class="anchor" id="line-62"></span>zonename: sol8zone<br /><span class="anchor" id="line-63"></span>zonepath: /mypool/zones/sol8zone<br /><span class="anchor" id="line-64"></span>brand: solaris8<br /><span class="anchor" id="line-65"></span>autoboot: false<br /><span class="anchor" id="line-66"></span>bootargs:<br /><span class="anchor" id="line-67"></span>pool:<br /><span class="anchor" id="line-68"></span>limitpriv:<br /><span class="anchor" id="line-69"></span>scheduling-class:<br /><span class="anchor" id="line-70"></span>ip-type: shared<br /><span class="anchor" id="line-71"></span>fs:<br /><span class="anchor" id="line-72"></span>        dir: /mypool/vol1<br /><span class="anchor" id="line-73"></span>        special: share/zone/sol8zone<br /><span class="anchor" id="line-74"></span>        raw not specified<br /><span class="anchor" id="line-75"></span>        type: zfs<br /><span class="anchor" id="line-76"></span>        options: []<br /><span class="anchor" id="line-77"></span>net:<br /><span class="anchor" id="line-78"></span>        address:<br /><span class="anchor" id="line-79"></span>        physical: bge1<br /><span class="anchor" id="line-80"></span>attr:<br /><span class="anchor" id="line-81"></span>        name: hostid<br /><span class="anchor" id="line-82"></span>        type: string<br /><span class="anchor" id="line-83"></span>        value: 8325f14d<br /><span class="anchor" id="line-84"></span>dwailsun:$() # zonecfg -z sol8zone info attr<br /><span class="anchor" id="line-85"></span>attr:<br /><span class="anchor" id="line-86"></span>        name: hostid<br /><span class="anchor" id="line-87"></span>        type: string<br /><span class="anchor" id="line-88"></span>        value: 8325f14d<br /><span class="anchor" id="line-89"></span>dwailsun:$() #<br /><span class="anchor" id="line-90"></span>

Install the zone

dwailsun:$() # zonecfg -z sol8zone export > /var/tmp/safe/sol8zone.config<br /><span class="anchor" id="line-96"></span>dwailsun:$(safe) # zoneadm -z s8-zone install -u -a /mypool/software/sol8p2v/solaris8-image.flar<br /><span class="anchor" id="line-97"></span>could not verify fs /mypool/vol1: could not access zfs dataset 'share/zone/s8-zone'<br /><span class="anchor" id="line-98"></span>zoneadm: zone s8-zone failed to verify<br /><span class="anchor" id="line-99"></span><br /><span class="anchor" id="line-100"></span>dwailsun:$(safe) # zfs list<br /><span class="anchor" id="line-101"></span>NAME                  USED  AVAIL  REFER  MOUNTPOINT<br /><span class="anchor" id="line-102"></span>mypool               3.75G  15.4G  39.3K  /mypool<br /><span class="anchor" id="line-103"></span>mypool/software      3.22G  6.78G  3.22G  /mypool/software<br /><span class="anchor" id="line-104"></span>mypool/vol1          66.6K  5.00G  34.0K  /mypool/vol1<br /><span class="anchor" id="line-105"></span>mypool/vol1/s8-zone  32.6K  5.00G  32.6K  /mypool/vol1/s8-zone<br /><span class="anchor" id="line-106"></span>mypool/www            544M  3.47G   544M  /mypool/www<br /><span class="anchor" id="line-107"></span>mypool/zones         34.0K  5.00G  34.0K  /mypool/zones<br /><span class="anchor" id="line-108"></span>dwailsun:$(safe) # zfs set mountpoint=legacy mypool/vol1/s8-zone<br /><span class="anchor" id="line-109"></span>dwailsun:$(safe) # zfs list<br /><span class="anchor" id="line-110"></span>NAME                  USED  AVAIL  REFER  MOUNTPOINT<br /><span class="anchor" id="line-111"></span>mypool               3.75G  15.4G  39.3K  /mypool<br /><span class="anchor" id="line-112"></span>mypool/software      3.22G  6.78G  3.22G  /mypool/software<br /><span class="anchor" id="line-113"></span>mypool/vol1          65.3K  5.00G  32.6K  /mypool/vol1<br /><span class="anchor" id="line-114"></span>mypool/vol1/s8-zone  32.6K  5.00G  32.6K  legacy<br /><span class="anchor" id="line-115"></span>mypool/www            544M  3.47G   544M  /mypool/www<br /><span class="anchor" id="line-116"></span>mypool/zones         34.0K  5.00G  34.0K  /mypool/zones<br /><span class="anchor" id="line-117"></span>dwailsun:$(safe) # zoneadm -z s8-zone install -u -a /mypool/software/sol8p2v/solaris8-image.flar<br /><span class="anchor" id="line-118"></span>      Log File: /var/tmp/s8-zone.install.987.log<br /><span class="anchor" id="line-119"></span>        Source: /mypool/software/sol8p2v/solaris8-image.flar<br /><span class="anchor" id="line-120"></span>    Installing: This may take several minutes...<br /><span class="anchor" id="line-121"></span>Postprocessing: This may take several minutes...<br /><span class="anchor" id="line-122"></span><br /><span class="anchor" id="line-123"></span>        Result: Installation completed successfully.<br /><span class="anchor" id="line-124"></span>      Log File: /mypool/zones/sol8zone/root/var/log/s8-zone.install.987.log<br /><span class="anchor" id="line-125"></span><br /><span class="anchor" id="line-126"></span>

Solaris8 P2V

Run sol8-p2v —

dwailsun:$(safe) # /usr/lib/brand/solaris8/s8_p2v s8-zone<br /><span class="anchor" id="line-135"></span>[Fri Dec 28 12:36:01 PST 2007]         S20_apply_patches:  Unpacking patch:  109 147-44<br /><span class="anchor" id="line-136"></span>[Fri Dec 28 12:36:01 PST 2007]         S20_apply_patches: Installing patch:  109 147-44<br /><span class="anchor" id="line-137"></span><br /><span class="anchor" id="line-138"></span>Checking installed patches...<br /><span class="anchor" id="line-139"></span>Patch 109147-44 has already been applied.<br /><span class="anchor" id="line-140"></span>See patchadd(1M) for instructions.<br /><span class="anchor" id="line-141"></span><br /><span class="anchor" id="line-142"></span>Patchadd is terminating.<br /><span class="anchor" id="line-143"></span>[Fri Dec 28 12:36:09 PST 2007]         S20_apply_patches:  Unpacking patch:  111 023-03<br /><span class="anchor" id="line-144"></span>[Fri Dec 28 12:36:09 PST 2007]         S20_apply_patches: Installing patch:  111 023-03<br /><span class="anchor" id="line-145"></span><br /><span class="anchor" id="line-146"></span>Checking installed patches...<br /><span class="anchor" id="line-147"></span>Patch 111023-03 has already been applied.<br /><span class="anchor" id="line-148"></span>See patchadd(1M) for instructions.<br /><span class="anchor" id="line-149"></span><br /><span class="anchor" id="line-150"></span>Patchadd is terminating.<br /><span class="anchor" id="line-151"></span>[Fri Dec 28 12:36:11 PST 2007]         S20_apply_patches:  Unpacking patch:  111 431-01<br /><span class="anchor" id="line-152"></span>[Fri Dec 28 12:36:11 PST 2007]         S20_apply_patches: Installing patch:  111 431-01<br /><span class="anchor" id="line-153"></span><br /><span class="anchor" id="line-154"></span>Checking installed patches...<br /><span class="anchor" id="line-155"></span>This patch is obsoleted by patch 108993-67 which has already<br /><span class="anchor" id="line-156"></span>been applied to this system.<br /><span class="anchor" id="line-157"></span><br /><span class="anchor" id="line-158"></span>Patchadd is terminating.<br /><span class="anchor" id="line-159"></span>[Fri Dec 28 12:36:13 PST 2007]         S20_apply_patches:  Unpacking patch:  112 605-04<br /><span class="anchor" id="line-160"></span>[Fri Dec 28 12:36:13 PST 2007]         S20_apply_patches: Installing patch:  112 605-04<br /><span class="anchor" id="line-161"></span><br /><span class="anchor" id="line-162"></span>Checking installed patches...<br /><span class="anchor" id="line-163"></span>This patch is obsoleted by patch 108993-67 which has already<br /><span class="anchor" id="line-164"></span>been applied to this system.<br /><span class="anchor" id="line-165"></span><br /><span class="anchor" id="line-166"></span>Patchadd is terminating.<br /><span class="anchor" id="line-167"></span>[Fri Dec 28 12:36:15 PST 2007]         S20_apply_patches:  Unpacking patch:  112 050-04<br /><span class="anchor" id="line-168"></span>[Fri Dec 28 12:36:15 PST 2007]         S20_apply_patches: Installing patch:  112 050-04<br /><span class="anchor" id="line-169"></span><br /><span class="anchor" id="line-170"></span>Checking installed patches...<br /><span class="anchor" id="line-171"></span>Patch 112050-04 has already been applied.<br /><span class="anchor" id="line-172"></span>See patchadd(1M) for instructions.<br /><span class="anchor" id="line-173"></span><br /><span class="anchor" id="line-174"></span>Patchadd is terminating.<br /><span class="anchor" id="line-175"></span>[Fri Dec 28 12:36:17 PST 2007]         S20_apply_patches:  Unpacking patch:  109 221-01<br /><span class="anchor" id="line-176"></span>[Fri Dec 28 12:36:17 PST 2007]         S20_apply_patches: Installing patch:  109 221-01<br /><span class="anchor" id="line-177"></span><br /><span class="anchor" id="line-178"></span>Checking installed patches...<br /><span class="anchor" id="line-179"></span>This patch is obsoleted by patch 109318-39 which has already<br /><span class="anchor" id="line-180"></span>been applied to this system.<br /><span class="anchor" id="line-181"></span><br /><span class="anchor" id="line-182"></span>Patchadd is terminating.<br /><span class="anchor" id="line-183"></span>dwailsun:$(safe) #<br /><span class="anchor" id="line-184"></span><br /><span class="anchor" id="line-185"></span>

dwailsun:$(safe) # zoneadm -z s8-zone boot<br /><span class="anchor" id="line-189"></span>dwailsun:$(safe) # zoneadm list -v<br /><span class="anchor" id="line-190"></span>  ID NAME             STATUS     PATH                           BRAND    IP<br /><span class="anchor" id="line-191"></span>   0 global           running    /                              native   shared<br /><span class="anchor" id="line-192"></span>   3 s8-zone          running    /mypool/zones/sol8zone         solaris8 shared<br /><span class="anchor" id="line-193"></span>dwailsun:$(safe) # zlogin -C s8-zone<br /><span class="anchor" id="line-194"></span>[Connected to zone 's8-zone' console]<br /><span class="anchor" id="line-195"></span><br /><span class="anchor" id="line-196"></span><br /><span class="anchor" id="line-197"></span>You did not enter a selection.<br /><span class="anchor" id="line-198"></span>What type of terminal are you using?<br /><span class="anchor" id="line-199"></span> 1) ANSI Standard CRT<br /><span class="anchor" id="line-200"></span> 2) DEC VT52<br /><span class="anchor" id="line-201"></span> 3) DEC VT100<br /><span class="anchor" id="line-202"></span> 4) Heathkit 19<br /><span class="anchor" id="line-203"></span> 5) Lear Siegler ADM31<br /><span class="anchor" id="line-204"></span> 6) PC Console<br /><span class="anchor" id="line-205"></span> 7) Sun Command Tool<br /><span class="anchor" id="line-206"></span> 8) Sun Workstation<br /><span class="anchor" id="line-207"></span> 9) Televideo 910<br /><span class="anchor" id="line-208"></span> 10) Televideo 925<br /><span class="anchor" id="line-209"></span> 11) Wyse Model 50<br /><span class="anchor" id="line-210"></span> 12) X Terminal Emulator (xterms)<br /><span class="anchor" id="line-211"></span> 13) Other<br /><span class="anchor" id="line-212"></span>Type the number of your choice and press Return: 12<br /><span class="anchor" id="line-213"></span>Configuring network interface addresses: bge1.<br /><span class="anchor" id="line-214"></span>RPC: Timed out<br /><span class="anchor" id="line-215"></span>

Then it goes through and does the sysidcfg bit…

System identification is completed.<br /><span class="anchor" id="line-222"></span><br /><span class="anchor" id="line-223"></span>rebooting system due to change(s) in /etc/default/init<br /><span class="anchor" id="line-224"></span><br /><span class="anchor" id="line-225"></span>Dec 28 12:41:25 rpcbind: rpcbind terminating on signal.<br /><span class="anchor" id="line-226"></span>System identification is completed.<br /><span class="anchor" id="line-227"></span><br /><span class="anchor" id="line-228"></span><br /><span class="anchor" id="line-229"></span>[NOTICE: Zone rebooting]<br /><span class="anchor" id="line-230"></span><br /><span class="anchor" id="line-231"></span>SunOS Release 5.8 Version Generic_Virtual 64-bit<br /><span class="anchor" id="line-232"></span>Copyright 1983-2000 Sun Microsystems, Inc.  All rights reserved<br /><span class="anchor" id="line-233"></span><br /><span class="anchor" id="line-234"></span>Hostname: sol8virt<br /><span class="anchor" id="line-235"></span>The system is coming up.  Please wait.<br /><span class="anchor" id="line-236"></span>starting rpc services: rpcbind done.<br /><span class="anchor" id="line-237"></span>syslog service starting.<br /><span class="anchor" id="line-238"></span>Print services started.<br /><span class="anchor" id="line-239"></span>Dec 28 14:41:37 sol8virt sendmail[4102]: My unqualified host name (sol8virt) unknown; sleeping for retry<br /><span class="anchor" id="line-240"></span>The system is ready.<br /><span class="anchor" id="line-241"></span><br /><span class="anchor" id="line-242"></span>sol8virt console login:<br /><span class="anchor" id="line-243"></span><br /><span class="anchor" id="line-244"></span>

# uname -a<br /><span class="anchor" id="line-248"></span>SunOS sol8virt 5.8 Generic_Virtual sun4u sparc SUNW,A70<br /><span class="anchor" id="line-249"></span># exit<br /><span class="anchor" id="line-250"></span><br /><span class="anchor" id="line-251"></span>[Connection to zone 's8-zone' pts/5 closed]<br /><span class="anchor" id="line-252"></span>dwailsun:$(safe) # uname -a<br /><span class="anchor" id="line-253"></span>SunOS dwailsun 5.10 Generic_127111-05 sun4u sparc SUNW,A70<br /><span class="anchor" id="line-254"></span>dwailsun:$(safe) # zlogin s8-zone<br /><span class="anchor" id="line-255"></span>[Connected to zone 's8-zone' pts/5]<br /><span class="anchor" id="line-256"></span>Last login: Fri Dec 28 14:43:35 on pts/5<br /><span class="anchor" id="line-257"></span>Sun Microsystems Inc.   SunOS 5.8       Generic Patch   February 2004<br /><span class="anchor" id="line-258"></span># uname -a<br /><span class="anchor" id="line-259"></span>SunOS sol8virt 5.8 Generic_Virtual sun4u sparc SUNW,A70<br /><span class="anchor" id="line-260"></span>#<br /><span class="anchor" id="line-261"></span># cat /etc/release<br /><span class="anchor" id="line-262"></span>                       Solaris 8 2/04 s28s_hw4wos_05a SPARC<br /><span class="anchor" id="line-263"></span>           Copyright 2004 Sun Microsystems, Inc.  All Rights Reserved.<br /><span class="anchor" id="line-264"></span>                            Assembled 08 January 2004<br /><span class="anchor" id="line-265"></span>#<br /><span class="anchor" id="line-266"></span><br /><span class="anchor" id="line-267"></span>

/!\ Think of a optimal battery of tests that can help us determine whether this virtualized solaris 8 is a viable platform for servers that cannot be migrated….

  • Adding packages — pkgadd works

# uname -a<br /><span class="anchor" id="line-275"></span>SunOS sol8virt 5.8 Generic_Virtual sun4u sparc SUNW,A70<br /><span class="anchor" id="line-276"></span># pkginfo|grep -i smc<br /><span class="anchor" id="line-277"></span>application SMCgcc         gcc<br /><span class="anchor" id="line-278"></span>application SMCliconv      libiconv<br /><span class="anchor" id="line-279"></span>application SMClintl       libintl<br /><span class="anchor" id="line-280"></span>application SMCosh471      openssh<br /><span class="anchor" id="line-281"></span>application SMCossl        openssl<br /><span class="anchor" id="line-282"></span>application SMCzlib        zlib<br /><span class="anchor" id="line-283"></span>

(!) Set up sshd after adding these packages, complete with start up scripts, sshd privsep user id in the system accounts files (passwd and shadow).

# /etc/init.d/sshd start<br /><span class="anchor" id="line-288"></span>Could not load host key: /usr/local/etc/ssh_host_key<br /><span class="anchor" id="line-289"></span>Could not load host key: /usr/local/etc/ssh_host_dsa_key<br /><span class="anchor" id="line-290"></span>Disabling protocol version 1. Could not load host key<br /><span class="anchor" id="line-291"></span># ps -ef|grep sshd<br /><span class="anchor" id="line-292"></span>    root  5086  4609  0 15:18:13 ?        0:00 /usr/local/sbin/sshd<br /><span class="anchor" id="line-293"></span>

Installing Oracle 8i

Setting up Oracle 8i was a breeze. Simply dumped the 2 cds of Oracle 8i 64-bit installation media onto a solaris8 zone visible fileystem and ran the runInstaller with all defaults and the demo database (scott/tiger) getting created as the end step.

/!\ Make sure to copy the media to local disk when installing inside the zone. The reason being, even though the cdrom can be exported to the local zone from the Global zone this way —

add fs<br /><span class="anchor" id="line-303"></span>set dir=/mnt<br /><span class="anchor" id="line-304"></span>set special=/cdrom<br /><span class="anchor" id="line-305"></span>set type=lofs<br /><span class="anchor" id="line-306"></span>add options ro<br /><span class="anchor" id="line-307"></span>add options nodevices<br /><span class="anchor" id="line-308"></span>end<br /><span class="anchor" id="line-309"></span>

We would have issues ejecting and inserting new cdroms, etc.

dwailsun:$() # ssh oracle@sol8virt<br /><span class="anchor" id="line-315"></span>oracle@sol8virt's password:<br /><span class="anchor" id="line-316"></span>Last login: Thu Jan  3 11:27:25 2008 from<br /><span class="anchor" id="line-317"></span>Sun Microsystems Inc.   SunOS 5.8       Generic Patch   February 2004<br /><span class="anchor" id="line-318"></span>Sun Microsystems Inc.   SunOS 5.8       Generic Patch   February 2004<br /><span class="anchor" id="line-319"></span>$ ps -ef|grep ora<br /><span class="anchor" id="line-320"></span>  oracle 22608 22152  0 11:25:48 ?        0:00 ora_reco_brandz<br /><span class="anchor" id="line-321"></span>  oracle 22610 22152  0 11:25:48 ?        0:00 ora_snp0_brandz<br /><span class="anchor" id="line-322"></span>  oracle 22626 22152  0 11:26:55 ?        0:00 /export/shared/oracle/OraHome1/bin/tnslsnr LISTENER -inherit<br /><span class="anchor" id="line-323"></span>  oracle 22614 22152  0 11:25:48 ?        0:00 ora_snp2_brandz<br /><span class="anchor" id="line-324"></span>  oracle 22687 22685  0 11:56:04 ?        0:00 /usr/local/sbin/sshd -R<br /><span class="anchor" id="line-325"></span>  oracle 22695 22689  0 11:56:09 pts/6    0:00 grep ora<br /><span class="anchor" id="line-326"></span>  oracle 22689 22687  0 11:56:04 pts/6    0:00 -ksh<br /><span class="anchor" id="line-327"></span>  oracle 22604 22152  4 11:25:48 ?        1:04 ora_ckpt_brandz<br /><span class="anchor" id="line-328"></span>  oracle 22600 22152  0 11:25:48 ?        0:00 ora_dbw0_brandz<br /><span class="anchor" id="line-329"></span>  oracle 22598 22152  0 11:25:48 ?        0:00 ora_pmon_brandz<br /><span class="anchor" id="line-330"></span>  oracle 22620 22152  0 11:25:48 ?        0:00 ora_d000_brandz<br /><span class="anchor" id="line-331"></span>  oracle 22602 22152  0 11:25:48 ?        0:02 ora_lgwr_brandz<br /><span class="anchor" id="line-332"></span>  oracle 22618 22152  0 11:25:48 ?        0:00 ora_s000_brandz<br /><span class="anchor" id="line-333"></span>  oracle 22616 22152  0 11:25:48 ?        0:00 ora_snp3_brandz<br /><span class="anchor" id="line-334"></span>  oracle 22612 22152  0 11:25:48 ?        0:00 ora_snp1_brandz<br /><span class="anchor" id="line-335"></span>  oracle 22606 22152  0 11:25:48 ?        0:00 ora_smon_brandz<br /><span class="anchor" id="line-336"></span>$

 Posted by at 10:07 pm
Oct 102007

A few thoughts on using new(er) technology to manage/maintain and collaborate docs and projects.

In my previous gig with a major Pharmaceutical company, I had taken on the onus of documenting the environment at the onset. That way I could learn the “whole picture” while creating good reference material. But the ordinary way of writing was both hard and cumbersome.

1) Old hacks and tricks in vi-edited text files on my workstation were hard to manage and were hard to distribute publicly.
2) Environment design, etc were pain-stakingly created with visio and took hours.
3) Word is a lousy product to use for technical documentation, imho.

So decided to try out a bunch of technologies. Set up two content management engines (both FOSS — Joomla! and Drupal) that were basically LAMP apps (Linux, Apache, MySQL, PHP) and were very powerful. Joomla! caught my eye due to the flexibility and out-of-the-box toolset it provided. Drupal was minimalistic and needed more effort to “mold”. I went on and created a couple of public-consumption web sites using Joomla! ( and

However, the most powerful tool that struck a optimistic chord in mine and the rest of the teams’ heart was a minimalist tool called Moinmoin ( that ran on a python-based engine, maintained the content without any RDBMS backing and was a breeze to setup and use.

With excellent markup tags, it has made documentation a breeze and we actually ended up creating a lot of docs using this tool and managing some multi-million dollar projects (infrastructure side) as well.

Of course there were the PM-drones who ran Project, GANTT charts, etc, and they have their own rightful place in the hierarchy of things in the PM framework. But to make a dreary task (that most admins fear) surprisingly easy, the Wiki was the perfect tool.

 Posted by at 7:26 pm