Jul 052013
 

It’s been a while since I’ve posted anything. So, here goes:

Identifying kernel memory leak on myhost-ldom (after P2V and OS upgrade from Solaris 9 to Solaris 10u10):

kmem_track.d sees this:

Tracing...If you see more allocs than frees, there is a potential issue...
Check against the cache name that is suspect

CACHE NAME                       ALLOCS   FREES
kmem_bufctl_audit_cache          0        47775
kmem_alloc_256                   0        87805
streams_dblk_1040                26072    0
kmem_alloc_40                    63752    0
vn_cache                         64883    64484
kmem_alloc_1152                  85338    82191
rctl_val_cache                   98712    99425
anonmap_cache                    105272   106039
kmem_alloc_96                    109072   108992
sfmmu8_cache                     109171   112886
kmem_alloc_32                    134058   135008
zio_cache                        146456   146456
streams_dblk_80                  162260   162274
kmem_alloc_160                   167740   167783
kmem_alloc_80                    187855   188146
sfmmu1_cache                     190247   194525
segvn_cache                      217514   218797
seg_cache                        232548   233831
kmem_alloc_8                     283391   283672
kmem_alloc_64                    286856   286354
streams_mblk                     313263   313281
anon_cache                       330058   336717
Tracing...If you see more allocs than frees, there is a potential issue...
Check against the cache name that is suspect

CACHE NAME                       ALLOCS   FREES
kmem_bufctl_audit_cache          0        47778
kmem_alloc_256                   0        87807
streams_dblk_1040                26216    0
kmem_alloc_40                    63777    0
vn_cache                         64887    64488
kmem_alloc_1152                  85383    82236
rctl_val_cache                   98787    99500
anonmap_cache                    105331   106098
kmem_alloc_96                    109075   108995
sfmmu8_cache                     109226   112967
kmem_alloc_32                    134132   135082
zio_cache                        146468   146468
streams_dblk_80                  162472   162486
kmem_alloc_160                   167875   167918
kmem_alloc_80                    187950   188241
sfmmu1_cache                     190362   194689
segvn_cache                      217628   218911
seg_cache                        232666   233949
kmem_alloc_8                     283452   283733
kmem_alloc_64                    286923   286421
streams_mblk                     313688   313706
anon_cache                       330176   336835
Tracing...If you see more allocs than frees, there is a potential issue...
Check against the cache name that is suspect

The two caches ( streams_dblk_1040 and kmem_alloc_40 are growing rapidly), associated with growth of kernel memory (validated by following output):

-----------------------------------
05-22-13-15-15
-----------------------------------
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                      99310               775   10%
ZFS File Data               31104               243    3%
Anon                        36342               283    4%
Exec and libs                4090                31    0%
Page cache                  20560               160    2%
Free (cachelist)             3905                30    0%
Free (freelist)            833012              6507   81%

Total                     1028323              8033
Physical                  1008210              7876

-----------------------------------
05-22-13-15-30
-----------------------------------
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                     150859              1178   15%
ZFS File Data               65092               508    6%
Anon                        68299               533    7%
Exec and libs                9371                73    1%
Page cache                  22618               176    2%
Free (cachelist)            11104                86    1%
Free (freelist)            700980              5476   68%

Total                     1028323              8033
Physical                  1005627              7856
-----------------------------------

The kernel memory utilization has grown 400MB in a space of 15 minutes.

So, running dtrace on streams_dblk_1040 cache shows us this:

$ sudo dtrace -n 'fbt::kmem_cache_alloc:entry /args[0]->cache_name == "streams_dblk_1040"/ \
 { @[uid,pid,ppid,curpsinfo->pr_psargs,execname,stack()] = count(); trunc(@,10);}'
dtrace: description 'fbt::kmem_cache_alloc:entry ' matched 1 probe
CPU     ID                    FUNCTION:NAME

        0     2122        1  /usr/local/sbin/snmpd -Lsd -Lf /dev/null -p /var/run/snmpd.pid -a  snmpd
              genunix`allocb+0x94
              ip`snmpcom_req+0x2bc
              ip`ip_wput_nondata+0x7b0
              unix`putnext+0x218
              unix`putnext+0x218
              ip`snmpcom_req+0x37c
              ip`ip_snmpmod_wput+0xe4
              unix`putnext+0x218
              ip`snmpcom_req+0x37c
              ip`ip_snmpmod_wput+0xe4
              unix`putnext+0x218
              genunix`strput+0x1d8
              genunix`strputmsg+0x2d4
              genunix`msgio32+0x354
              genunix`putmsg32+0x98
              unix`syscall_trap32+0xcc
               11
        0     2122        1  /usr/local/sbin/snmpd -Lsd -Lf /dev/null -p /var/run/snmpd.pid -a  snmpd
              genunix`allocb+0x94
              ip`snmp_append_data2+0x70
              ip`tcp_snmp_get+0x5d4
              ip`snmpcom_req+0x350
              ip`ip_snmpmod_wput+0xe4
              unix`putnext+0x218
              ip`snmpcom_req+0x37c
              ip`ip_snmpmod_wput+0xe4
              unix`putnext+0x218
              genunix`strput+0x1d8
              genunix`strputmsg+0x2d4
              genunix`msgio32+0x354
              genunix`putmsg32+0x98
              unix`syscall_trap32+0xcc
               22
        0     2122        1  /usr/local/sbin/snmpd -Lsd -Lf /dev/null -p /var/run/snmpd.pid -a  snmpd
              genunix`allocb+0x94
              ip`snmp_append_data2+0x70
              ip`ip_snmp_get2_v4+0x310
              ip`ire_walk_ill_tables+0x30c
              ip`ire_walk_ipvers+0x64
              ip`ip_snmp_get_mib2_ip_route_media+0x74
              ip`ip_snmp_get+0x298
              ip`snmpcom_req+0x350
              ip`ip_wput_nondata+0x7b0
              unix`putnext+0x218
              unix`putnext+0x218
              ip`snmpcom_req+0x37c
              ip`ip_snmpmod_wput+0xe4
              unix`putnext+0x218
              ip`snmpcom_req+0x37c
              ip`ip_snmpmod_wput+0xe4
              unix`putnext+0x218
              genunix`strput+0x1d8
              genunix`strputmsg+0x2d4
              genunix`msgio32+0x354
               77

This is snmpd. I had already noted that snmpd had an issues (complained during boot up about configuration file entries).

Stopping net-smpd resulted in the kernel memory being released back into the freelist.

The other cache — kmem_alloc_40 shows this:

$ sudo dtrace -n 'fbt::kmem_cache_alloc:entry /args[0]->cache_name == "kmem_alloc_40"/ \
{ @[uid,pid,ppid,curpsinfo->pr_psargs,execname,stack()] = count(); trunc(@,10);}'
dtrace: description 'fbt::kmem_cache_alloc:entry ' matched 1 probe
CPU     ID                    FUNCTION:NAME
        0      586        1  /opt/quest/sbin/vasd -p /var/opt/quest/vas/vasd/.vasd.pid  vasd
              genunix`kmem_zalloc+0x28
              zfs`dsl_dir_tempreserve_impl+0x1e0
              zfs`dsl_dir_tempreserve_space+0x128
              zfs`dmu_tx_try_assign+0x220
              zfs`dmu_tx_assign+0xc
              zfs`zfs_write+0x4b8
              genunix`fop_write+0x20
              genunix`write+0x268
              unix`syscall_trap32+0xcc
               39
        0      586        1  /opt/quest/sbin/vasd -p /var/opt/quest/vas/vasd/.vasd.pid  vasd
              genunix`kmem_zalloc+0x28
              zfs`dsl_dir_tempreserve_space+0x68
              zfs`dmu_tx_try_assign+0x220
              zfs`dmu_tx_assign+0xc
              zfs`zfs_write+0x4b8
              genunix`fop_write+0x20
              genunix`write+0x268
              unix`syscall_trap32+0xcc
               39
        0      586        1  /opt/quest/sbin/vasd -p /var/opt/quest/vas/vasd/.vasd.pid  vasd
              genunix`kmem_zalloc+0x28
              zfs`dsl_dir_tempreserve_space+0xd4
              zfs`dmu_tx_try_assign+0x220
              zfs`dmu_tx_assign+0xc
              zfs`zfs_write+0x4b8
              genunix`fop_write+0x20
              genunix`write+0x268
              unix`syscall_trap32+0xcc
               39
        0      586        1  /opt/quest/sbin/vasd -p /var/opt/quest/vas/vasd/.vasd.pid  vasd
              genunix`kmem_zalloc+0x28
              zfs`zfs_get_data+0x80
              zfs`zil_lwb_commit+0x188
              zfs`zil_commit_writer+0xbc
              zfs`zil_commit+0x90
              zfs`zfs_fsync+0xfc
              genunix`fop_fsync+0x14
              genunix`fdsync+0x20
              unix`syscall_trap32+0xcc
               40

Looks like vintela is writing to ZFS…

After shutting down snmpd, it seems like kernel stopped growing like crazy…

$ echo "::memstat"|sudo mdb -k
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                     151839              1186   15%
ZFS File Data               65746               513    6%
Anon                        58443               456    6%
Exec and libs                6927                54    1%
Page cache                  22475               175    2%
Free (cachelist)            13836               108    1%
Free (freelist)            709057              5539   69%

Total                     1028323              8033
Physical                  1005627              7856
[$:/var/adm (56)]
$ echo "::memstat"|sudo mdb -k
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                     135583              1059   13%
ZFS File Data               65876               514    6%
Anon                        55253               431    5%
Exec and libs                7202                56    1%
Page cache                  22475               175    2%
Free (cachelist)            13589               106    1%
Free (freelist)            728345              5690   71%

Total                     1028323              8033
Physical                  1005627              7856
[$:/var/adm (57)]
$ echo "::memstat"|sudo mdb -k
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                     135839              1061   13%
ZFS File Data               65941               515    6%
Anon                        55161               430    5%
Exec and libs                6909                53    1%
Page cache                  22477               175    2%
Free (cachelist)            13890               108    1%
Free (freelist)            728106              5688   71%

Total                     1028323              8033
Physical                  1005627              7856

It’s now hovering around 1000MB, down from ~ 1200MB..

05-24-2013 Revisit

anon_cache                       1162987  1161545
Tracing...If you see more allocs than frees, there is a potential issue...
Check against the cache name that is suspect

CACHE NAME                       ALLOCS   FREES
kmem_alloc_16                    0        204696
kmem_alloc_128                   111705   111703
vn_cache                         182607   0
kmem_alloc_256                   186757   186900
file_cache                       217169   217300
kmem_alloc_40                    242674   242645
zio_cache                        279855   279855
kmem_alloc_1152                  295560   295553
rctl_val_cache                   354759   355428
sfmmu8_cache                     358797   360868
kmem_alloc_96                    370087   370046
anonmap_cache                    380465   380542
kmem_alloc_160                   474039   474385
kmem_alloc_32                    476749   476818
kmem_alloc_8                     588250   587960
kmem_alloc_80                    674765   675200
sfmmu1_cache                     683800   684676
segvn_cache                      785621   785685
kmem_alloc_64                    789370   788307
seg_cache                        839840   839904
anon_cache                       1163105  1161663

Only one stands out — vn_cache. Therefore, run following dtrace command:

$ sudo dtrace -qn 'fbt::kmem_cache_alloc:entry /args[0]->cache_name == "vn_cache"/  \
{ @[uid,pid,ppid,curpsinfo->pr_psargs,execname,stack()] = count(); trunc(@,10);} \
profile:::tick-15sec { printa(@); }'

This will show stack trace aggregations of top 10 processes entering kmem_cache_alloc at a 15 second interval. Turns out it is HPOV:

        0     2104        1  /opt/perf/bin/perfd                                 perfd
              genunix`vn_alloc+0xc
              procfs`prgetnode+0x38
              procfs`pr_lookup_procdir+0x94
              procfs`prlookup+0x198
              genunix`fop_lookup+0x28
              genunix`lookuppnvp+0x30c
              genunix`lookuppnat+0x120
              genunix`lookupnameat+0x5c
              genunix`vn_openat+0x168
              genunix`copen+0x260
              unix`syscall_trap32+0xcc
             1087
        0     2104        1  /opt/perf/bin/perfd                                 perfd
              genunix`vn_alloc+0xc
              procfs`prgetnode+0x38
              procfs`prgetnode+0x110
              procfs`pr_lookup_procdir+0x94
              procfs`prlookup+0x198
              genunix`fop_lookup+0x28
              genunix`lookuppnvp+0x30c
              genunix`lookuppnat+0x120
              genunix`lookupnameat+0x5c
              genunix`vn_openat+0x168
              genunix`copen+0x260
              unix`syscall_trap32+0xcc
             1087
        0     2046        1  /opt/perf/bin/scopeux                               scopeux
              genunix`vn_alloc+0xc
              procfs`prgetnode+0x38
              procfs`pr_lookup_procdir+0x94
              procfs`prlookup+0x198
              genunix`fop_lookup+0x28
              genunix`lookuppnvp+0x30c
              genunix`lookuppnat+0x120
              genunix`lookupnameat+0x5c
              genunix`vn_openat+0x168
              genunix`copen+0x260
              unix`syscall_trap32+0xcc
             2168
        0     2046        1  /opt/perf/bin/scopeux                               scopeux
              genunix`vn_alloc+0xc
              procfs`prgetnode+0x38
              procfs`prgetnode+0x110
              procfs`pr_lookup_procdir+0x94
              procfs`prlookup+0x198
              genunix`fop_lookup+0x28
              genunix`lookuppnvp+0x30c
              genunix`lookuppnat+0x120
              genunix`lookupnameat+0x5c
              genunix`vn_openat+0x168
              genunix`copen+0x260
              unix`syscall_trap32+0xcc

Which lines up with the output of the heap/stack growth (hpstckgrow.d) script —

Tracing...Ctrl-C to exit
Tracking processes that are growing their heap size...
aggregation printed at 60s intervals
EXEC     PID      COMMAND                                  COUNT
kstat    23772    /usr/perl5/bin/perl /usr/bin/kstat -n dnlcstats 412
kstat    26262    /usr/perl5/bin/perl /usr/bin/kstat -n dnlcstats 412
kstat    28069    /usr/perl5/bin/perl /usr/bin/kstat -n dnlcstats 412
kstat    647      /usr/perl5/bin/perl /usr/bin/kstat -p unix:::boot_time 1028
kstat    6674     /usr/perl5/bin/perl /usr/bin/kstat -p unix:::boot_time 1028
kstat    24090    /usr/perl5/bin/perl /usr/bin/kstat -p unix:::boot_time 1028
kstat    1410     /usr/perl5/bin/perl /usr/bin/kstat -p unix:::boot_time 1030
facter   395      /opt/puppet/bin/ruby /opt/puppet/bin/facter --puppet --yaml 2106
facter   6561     /opt/puppet/bin/ruby /opt/puppet/bin/facter --puppet --yaml 2106
facter   23793    /opt/puppet/bin/ruby /opt/puppet/bin/facter --puppet --yaml 2107

Tracking processes that are growing their stack size...
aggregation printed at 60s intervals
EXEC     PID      COMMAND                                  COUNT
zstatd   27933    /opt/sun/xvm/lib/zstat/zstatd            1
zstatd   28306    /opt/sun/xvm/lib/zstat/zstatd            1
zstatd   28763    /opt/sun/xvm/lib/zstat/zstatd            1
zstatd   28918    /opt/sun/xvm/lib/zstat/zstatd            1
zstatd   29227    /opt/sun/xvm/lib/zstat/zstatd            1
zstatd   29491    /opt/sun/xvm/lib/zstat/zstatd            1
zstatd   29664    /opt/sun/xvm/lib/zstat/zstatd            1
java     952      /usr/jdk/instances/jdk1.6.0/bin/java -Xmx150m -Xss192k -XX:MinHeapFreeRatio=10  5
scopeux  2046     /opt/perf/bin/scopeux                    126
perfd    2104     /opt/perf/bin/perfd                      1260
Uninstalling and reinstalling the net-snmpd and HP Openview packages ultimately stopped the memory leaks.
Links:
Both the scripts listed above are modifications/adaptations of code from Brendan Gregg’s excellent DTrace book —

Oct 162008
 

Since the past several months, we have been encountering a baffling problem with Solaris 10 systems hanging on their way up after a reboot (or simply a boot) in the multi-user mode (run level 3). Since we do our system patching at regular intervals (and since solaris runs so well that we don’t need to reboot our servers otherwise), we didn’t notice this until we came upon our latest patch cycle last month.

We noticed that our systems were not booting up the normal way. The only way to boot was to “trick” the system into booting into single-user mode and then exiting out to the multi-user run-level.

This was a problem happening only on systems running Solaris 10, with Veritas Volume Manager 5.0 MP1. So naturally we leaned towards tackling this first as a possible VxVM problem (introduced by some patch during the patch cycle). And the patch rev we had decided to apply had the Veritas Storage Foundation 5.0 MP1 RP4 in it.

Again, to reiterate the problem description —

Description:

Systems running Solaris 10 and Veritas Volume Manager tend to hang when booted with a normal init 6 (ie while booting into Run Level 3).

Workaround:

Boot the system in run level S (single user mode) and then exit out of it to boot into multi-user mode.


Details:

Opened a case with Veritas and sent them VRTSexplorer outputs and copies of our messages file from problem host nodeA.

After several iterations of generating explorers and veritas configuration information, Veritas still didn’t have anything substantial they could pin this issue on. They asked me to generate a crash dump of the hanging system (usually should be able to coredump a running host by breaking it and running “sync” from the OBP).

After repeated attempts at generating the core, I was unable to do so. It seems like the system hangs before the dump device is initialized/configured by the OS.

Using the work around, I enabled the solaris deadman timer (which incidentally we should have on all our servers). This involves setting the following line in the /etc/system file —

* set snooping = 1

What the deadman timer does is sents a high-priority hardware interrupt to each CPU (or Core or strand depending on the platform) and updates a counter upon successful response by the CPU. In case of a system being hung (due to hardware issues especially), this count might not increase with an increase in clock tick (interrupts aresent to CPUs every tick). When this counter is not incremented, the kernel panics and kills itself.

This didn’t work because we didn’t encounter that kind of a problem (but is a good idea to have enabled nonetheless).

Another alternative for panicking a system is as follows

Boot the server in kernel debug mode (in SPARC systems it is done by running boot -k from the OBP). This loads the kmdb module into the kernel as it boots up. Breaking the system while in kernel debug mode will not drop it into the OBP but instead launch an mdb interface.

One can generate a coredump by running $ < i="" tried="" that="" but="" no="">


[17]> $ <>
panic[cpu17]/thread=2a101a3fca0: BAD TRAP: type=9 rp=2a101a3f760 addr=0 mmu_fsr=0

sched: trap type = 0x9
pid=0, pc=0x0, sp=0x2a101a3f001, tstate=0x1606, context=0x0
g1-g7: 1011be4, 1, 1870000, 20040, 1860c00, 0, 2a101a3fca0

000002a101a3f480 unix:die+9c (9, 2a101a3f760, 0, 0, 2a101a3f540, 1901ec0)
%l0-3: 0000000000000000 0000000000000009 0000000000000009 0000000000000000
%l4-7: 0000000000000000 00000000018f1800 0000000000001606 0000000001097000
000002a101a3f560 unix:trap+6cc (2a101a3f760, 10000, 0, 0, 30003dac000, 2a101a3fca0)
%l0-3: 0000000000000000 0000000001859480 0000000000000009 0000000000000000
%l4-7: 0000000000000000 00000000018f1800 0000000000001606 0000000000010200
000002a101a3f6b0 unix:ktl0+64 (0, 20040, 1860c00, 1, 1260ce4, 1901ec0)
%l0-3: 0000030003dac000 0000000000000088 0000000000001606 0000000001020354
%l4-7: 0000000000000000 00000000018f1800 000000000000000c 000002a101a3f760
000002a101a3f800 unix:debug_enter+108 (0, a, a, 1826400, 0, 1870000)
%l0-3: 0000000001834060 00000048df61937c 0000000000000000 0000000000000000
%l4-7: 0000000000000000 00000000018f1800 0000000000000000 0000000001011b30
000002a101a3f8d0 unix:abort_seq_softintr+94 (1826400, 18f1800, 30003dac000, 2a101a3fd78, 1, 189f800)
%l0-3: 0000000001834060 00000048df61937c 0000000000000000 0000000000000000
%l4-7: 0000000000000000 00000000018f1800 0000000000000000 0000000001011b30

syncing file systems… done
skipping system dump – no dump device configured
rebooting…

SC Alert: Host System has Reset

SC Alert: Failed to send email alert for recent event.

SC Alert: Indicator SYS/ACT is now STANDBY BLINK

So back to the kernel debug mode again. This time I decided to try and investigate the kernel (you can do this against a coredump as well).


[17]> ::msgbuf
MESSAGE
/pci@780/pci@0/pci@9/scsi@0 (mpt0):
mpt0 supports power management.
/pci@780/pci@0/pci@9/scsi@0 (mpt0):
DMA restricted to lower 4GB due to errata
/pci@780/pci@0/pci@9/scsi@0 (mpt0):
mpt0 Firmware version v1.9.0.0 (IR)
/pci@780/pci@0/pci@9/scsi@0 (mpt0):
mpt0: IOC Operational.
/pci@780/pci@0/pci@9/scsi@0 (mpt0):
mpt0: Initiator WWNs: 0x5080020000262858-0x508002000026285b
PCI-device: scsi@0, mpt0
mpt0 is /pci@780/pci@0/pci@9/scsi@0
sd1 at mpt0: target 0 lun 0
sd1 is /pci@780/pci@0/pci@9/scsi@0/sd@0,0
/pci@780/pci@0/pci@9/scsi@0/sd@0,0 (sd1) online
root on /pci@780/pci@0/pci@9/scsi@0/disk@0,0:a fstype ufs
px1 at root: 0x7c0 0x0
px1 is /pci@7c0
PCI Express-device: pci@0, pxb_plx5
pxb_plx5 is /pci@7c0/pci@0
PCI-device: pci@1, pxb_plx6
pxb_plx6 is /pci@7c0/pci@0/pci@1
PCI-device: pci@0, px_pci0
px_pci0 is /pci@7c0/pci@0/pci@1/pci@0
PCI-device: ide@8, uata0
uata0 is /pci@7c0/pci@0/pci@1/pci@0/ide@8
WARNING: px1: spurious interrupt from ino 0x4
uata-0#0

sd2 at uata0: target 0 lun 0
sd2 is /pci@7c0/pci@0/pci@1/pci@0/ide@8/sd@0,0
sd3 at mpt0: target 1 lun 0
sd3 is /pci@780/pci@0/pci@9/scsi@0/sd@1,0
/pci@780/pci@0/pci@9/scsi@0/sd@1,0 (sd3) online
sd4 at mpt0: target 2 lun 0
sd4 is /pci@780/pci@0/pci@9/scsi@0/sd@2,0
/pci@780/pci@0/pci@9/scsi@0/sd@2,0 (sd4) online
sd5 at mpt0: target 3 lun 0
sd5 is /pci@780/pci@0/pci@9/scsi@0/sd@3,0
/pci@780/pci@0/pci@9/scsi@0/sd@3,0 (sd5) online
iscsi0 at root
iscsi0 is /iscsi
root remounted on /pseudo/vxio@0:0 fstype ufs
virtual-device: cnex0
cnex0 is /virtual-devices@100/channel-devices@200
PCI-device: isa@2, ebus0
ebus0 is /pci@7c0/pci@0/pci@1/pci@0/isa@2
pseudo-device: dld0
dld0 is /pseudo/dld@0
PCI-device: usb@5, ohci0
ohci0 is /pci@7c0/pci@0/pci@1/pci@0/usb@5
PCI-device: usb@6, ohci1
ohci1 is /pci@7c0/pci@0/pci@1/pci@0/usb@6
cpu0: UltraSPARC-T1 (cpuid 0 clock 1000 MHz)
cpu1: UltraSPARC-T1 (cpuid 1 clock 1000 MHz)
cpu1 initialization complete – online
cpu2: UltraSPARC-T1 (cpuid 2 clock 1000 MHz)
cpu2 initialization complete – online
cpu3: UltraSPARC-T1 (cpuid 3 clock 1000 MHz)
cpu3 initialization complete – online
cpu4: UltraSPARC-T1 (cpuid 4 clock 1000 MHz)
cpu4 initialization complete – online
cpu5: UltraSPARC-T1 (cpuid 5 clock 1000 MHz)
cpu5 initialization complete – online
cpu6: UltraSPARC-T1 (cpuid 6 clock 1000 MHz)
cpu6 initialization complete – online
cpu7: UltraSPARC-T1 (cpuid 7 clock 1000 MHz)
cpu7 initialization complete – online
cpu8: UltraSPARC-T1 (cpuid 8 clock 1000 MHz)
cpu8 initialization complete – online
cpu9: UltraSPARC-T1 (cpuid 9 clock 1000 MHz)
cpu9 initialization complete – online
cpu10: UltraSPARC-T1 (cpuid 10 clock 1000 MHz)
cpu10 initialization complete – online
cpu11: UltraSPARC-T1 (cpuid 11 clock 1000 MHz)
cpu11 initialization complete – online
cpu12: UltraSPARC-T1 (cpuid 12 clock 1000 MHz)
cpu12 initialization complete – online
cpu13: UltraSPARC-T1 (cpuid 13 clock 1000 MHz)
cpu13 initialization complete – online
cpu14: UltraSPARC-T1 (cpuid 14 clock 1000 MHz)
cpu14 initialization complete – online
cpu15: UltraSPARC-T1 (cpuid 15 clock 1000 MHz)
cpu15 initialization complete – online
cpu16: UltraSPARC-T1 (cpuid 16 clock 1000 MHz)
cpu16 initialization complete – online
cpu17: UltraSPARC-T1 (cpuid 17 clock 1000 MHz)
cpu17 initialization complete – online
cpu18: UltraSPARC-T1 (cpuid 18 clock 1000 MHz)
cpu18 initialization complete – online
cpu19: UltraSPARC-T1 (cpuid 19 clock 1000 MHz)
cpu19 initialization complete – online
cpu20: UltraSPARC-T1 (cpuid 20 clock 1000 MHz)
cpu20 initialization complete – online
cpu21: UltraSPARC-T1 (cpuid 21 clock 1000 MHz)
cpu21 initialization complete – online
cpu22: UltraSPARC-T1 (cpuid 22 clock 1000 MHz)
cpu22 initialization complete – online
cpu23: UltraSPARC-T1 (cpuid 23 clock 1000 MHz)
cpu23 initialization complete – online
USB 1.10 device (usb3eb,3301) operating at full speed (USB 1.x) on USB 1.10 root
hub: hub@1, hubd1 at bus address 2
hubd1 is /pci@7c0/pci@0/pci@1/pci@0/usb@6/hub@1
/pci@7c0/pci@0/pci@1/pci@0/usb@6/hub@1 (hubd1) online
PCI-device: pci@0,2, px_pci1
px_pci1 is /pci@7c0/pci@0/pci@1/pci@0,2
PCI-device: pci@1, pxb_plx1
pxb_plx1 is /pci@780/pci@0/pci@1
PCI-device: pci@2, pxb_plx2
pxb_plx2 is /pci@780/pci@0/pci@2
PCI-device: pci@8, pxb_plx3
pxb_plx3 is /pci@780/pci@0/pci@8
PCI-device: pci@2, pxb_plx7
pxb_plx7 is /pci@7c0/pci@0/pci@2
PCI-device: pci@8, pxb_plx8
pxb_plx8 is /pci@7c0/pci@0/pci@8
PCI-device: pci@9, pxb_plx9
pxb_plx9 is /pci@7c0/pci@0/pci@9
NOTICE: e1000g0 registered
Intel(R) PRO/1000 Network Connection, Driver Ver. 5.1.11
NOTICE: pciex8086,105e – e1000g[0] : Adapter copper link is down.
NOTICE: e1000g1 registered
Intel(R) PRO/1000 Network Connection, Driver Ver. 5.1.11
NOTICE: pciex8086,105e – e1000g[1] : Adapter copper link is down.
NOTICE: e1000g3 registered
Intel(R) PRO/1000 Network Connection, Driver Ver. 5.1.11
NOTICE: pciex8086,105e – e1000g[3] : Adapter copper link is down.
NOTICE: pciex8086,105e – e1000g[0] : Adapter 1000Mbps full duplex copper link is
up.
NOTICE: pciex8086,105e – e1000g[1] : Adapter 1000Mbps full duplex copper link is
up.
pseudo-device: devinfo0
devinfo0 is /pseudo/devinfo@0
NOTICE: pciex8086,105e – e1000g[3] : Adapter 1000Mbps full duplex copper link is
up.
NOTICE: VxVM vxdmp V-5-0-34 added disk array DISKS, datype = Disk

NOTICE: VxVM vxdmp V-5-3-1700 dmpnode 300/0x0 has migrated from enclosure FAKE_E
NCLR_SNO to enclosure DISKS

NOTICE: e1000g2 registered
Intel(R) PRO/1000 Network Connection, Driver Ver. 5.1.11

Look at the process table…


[17]> ::ps
S PID PPID PGID SID UID FLAGS ADDR NAME
R 0 0 0 0 0 0x00000001 0000000001859480 sched
R 3 0 0 0 0 0x00020001 000006001172d838 fsflush
R 2 0 0 0 0 0x00020001 000006001172e450 pageout
R 1 0 0 0 0 0x4a004000 000006001172f068 init
R 79 1 78 78 0 0x42010000 0000060017f30028 ssmagent.bin
R 9 1 9 9 0 0x42000000 000006001292f070 svc.configd
R 7 1 7 7 0 0x42000000 000006001172c008 svc.startd
R 54 7 7 7 0 0x4a004000 0000060017e43080 vxvm-sysboot
R 56 54 7 7 0 0x4a004000 0000060017e42468 vxconfigd
R 57 56 57 57 0 0x42020000 0000060017dc8018 vxconfigd

Pick a suspect thread and dig deeper —


[17]> 0000060017e43080::findstack -v
kmdb: thread 60017e43080 isn’t in memory
[17]> 0000060017e43080::walk thread
30003eba480
[17]> 0000060017e43080::walk thread| ::findstack -v
stack pointer for thread 30003eba480: 2a101514ff1
[ 000002a101514ff1 cv_wait_sig_swap_core+0x130() ]
000002a1015150a1 waitid+0x484(0, 60017e42468, 0, 60017e430e8, 0, 1)
000002a101515171 waitsys32+0x10(0, 38, ffbffa80, 83, 39590, 3a57c)
000002a1015152e1 syscall_trap32+0xcc(0, 38, ffbffa80, 83, 39590, 3a57c)
[17]>

[17]> 0000060017e42468::walk thread| ::findstack -v
stack pointer for thread 30003fae200: 2a101dbeff1
[ 000002a101dbeff1 cv_wait_sig_swap_core+0x130() ]
000002a101dbf0a1 waitid+0x484(0, 0, 0, 60017e424d0, 7, 1)
000002a101dbf171 waitsys32+0x10(7, 0, ffbffaa8, 3, 2909e8, 0)
000002a101dbf2e1 syscall_trap32+0xcc(7, 0, ffbffaa8, 3, 2909e8, 0)

[17]> 0000060017dc8018::walk thread | ::findstack -v
stack pointer for thread 30003eb0420: 2a10072ec21
[ 000002a10072ec21 cv_wait+0x38() ]
000002a10072ecd1 delay+0x90(1, 1870000, df39, df3a, 1, 18f0818)
000002a10072ed81 vxio`volkio_flush_cached_io+0xdc(7006ea30, 30003eb0420,
7006789c, 7006e9e8, 0, 0)
000002a10072ee31 vxio`vol_commit_flush_cached_io+0x60(60011503240, 70046d28, 0
, 0, 0, 0)
000002a10072eee1 vxio`vol_ktrans_commit+0x6fc(7006b2f0, 1, 7006789c, 7006d92c
, 60011503240, 0)
000002a10072ef91 vxio`volsioctl_real+0x4c8(70034000, 564f4c86, 0, 100003,
60011403768, 2a10072fadc)
000002a10072f0e1 fop_ioctl+0x20(60012a7f900, 564f4c86, 0, 100003, 60011403768
, 12f1770)
000002a10072f191 ioctl+0x184(1, 600115e8a50, 0, 564f4c86, 0, 564f4c86)
000002a10072f2e1 syscall_trap32+0xcc(1, 564f4c86, 0, 564f4c86, 0, 820770)
[17]>

Found something in the last one, but no immediate red flags. But then I saw this (ssmagent running). SSM Agent is a SNMP agent that is used to monitor our systems and report back to Netcool, our Fault Management server.


[17]> ::ps
S PID PPID PGID SID UID FLAGS ADDR NAME
R 0 0 0 0 0 0x00000001 0000000001859480 sched
R 3 0 0 0 0 0x00020001 000006001172d838 fsflush
R 2 0 0 0 0 0x00020001 000006001172e450 pageout
R 1 0 0 0 0 0x4a004000 000006001172f068 init
R 79 1 78 78 0 0x42010000 0000060017f5e028 ssmagent.bin
R 43 1 42 42 0 0x42020000 0000060017db7078 dhcpagent
R 9 1 9 9 0 0x42000000 0000060012922458 svc.configd
R 7 1 7 7 0 0x42000000 000006001172c008 svc.startd
R 54 7 7 7 0 0x4a004000 000006001172cc20 vxvm-sysboot
R 56 54 7 7 0 0x4a004000 0000060017e10468 vxconfigd
R 57 56 57 57 0 0x42020000 0000060017db4018 vxconfigd
[17]> 0000060017f5e028::walk thread| ::findstack -v
stack pointer for thread 30003fb55a0: 2a101e02961
[ 000002a101e02961 sema_p+0x130() ]
000002a101e02a11 biowait+0x6c(60017dda100, 0, 1870000, 30003dbe000, 2000,
60017dda100)
000002a101e02ac1 ufs`ufs_getpage_miss+0x2ec(60019a0f300, 40000, 4de,
600129cea20, fdba0000, 2a101e03760)
000002a101e02bc1 ufs`ufs_getpage+0x694(300014b7e00, 40000, 1, 0, 1, 3)
000002a101e02d21 fop_getpage+0x44(60019a0f300, 600114659c0, 60011403978, 3,
fdba0000, 3)
000002a101e02df1 segvn_fault+0xb04(8000, 600129cea20, 3, 2000, 40000, 0)
000002a101e02fc1 as_fault+0x4c8(600129cea20, 600129d9200, fdba0000,
60011736320, 189eb00, 0)
000002a101e030d1 pagefault+0x68(fdba14a8, 0, 3, 0, 60017f5e028, 600117362a8)
000002a101e03191 trap+0xd50(2a101e03b90, 10000, 0, 3, fdba14a8, 0)
000002a101e032e1 utl0+0x4c(ff3f40fc, ff3f5a70, 1, 0, ff3f4910, 821)
stack pointer for thread 30003f08820: 2a102005091
[ 000002a102005091 cv_wait_sig_swap_core+0x130() ]
000002a102005141 lwp_park+0x130(0, 1, 30003f089c6, 30003f08820, 0, 100000)
000002a102005231 syslwp_park+0x54(0, 0, 0, 0, ff092010, 1)
000002a1020052e1 syscall_trap32+0xcc(0, 0, 0, 0, ff092010, 1)
stack pointer for thread 30003fb4560: 2a101d9f091
[ 000002a101d9f091 cv_wait_sig_swap_core+0x130() ]
000002a101d9f141 lwp_park+0x130(0, 1, 30003fb4706, 30003fb4560, 0, 100000)
000002a101d9f231 syslwp_park+0x54(0, 0, 0, 0, ff092020, 1)
000002a101d9f2e1 syscall_trap32+0xcc(0, 0, 0, 0, ff092020, 1)
stack pointer for thread 30003ef77c0: 2a1008e9091
[ 000002a1008e9091 cv_wait_sig_swap_core+0x130() ]
000002a1008e9141 lwp_park+0x130(0, 1, 30003ef7966, 30003ef77c0, 0, 100000)
000002a1008e9231 syslwp_park+0x54(0, 0, 0, 0, ff092030, 1)
000002a1008e92e1 syscall_trap32+0xcc(0, 0, 0, 0, ff092030, 1)
stack pointer for thread 30003fd35c0: 2a101db6f91
[ 000002a101db6f91 cv_timedwait_sig+0x16c() ]
000002a101db7041 cv_waituntil_sig+0x8c(60017dc8592, 60017dc8558, 2a101db7ad0,
2, 18f0800, 2)
000002a101db7111 poll_common+0x4e8(60012996580, 60017f5e028, 2a101db7ad0, 0,
fe57bcd0, 2)
000002a101db7201 pollsys+0xf8(fe57bcd0, 1, fe57bd70, 0, 2a101db7ad0, 0)
000002a101db72e1 syscall_trap32+0xcc(fe57bcd0, 1, fe57bd70, 0, fe57bd70, 0)
stack pointer for thread 30003efa460: 2a101df3091
[ 000002a101df3091 cv_wait_sig_swap_core+0x130() ]
000002a101df3141 lwp_park+0x130(0, 1, 30003efa606, 30003efa460, 0, 100000)
000002a101df3231 syslwp_park+0x54(0, 0, 0, 0, ff092050, 1)
000002a101df32e1 syscall_trap32+0xcc(0, 0, 0, 0, ff092050, 1)
stack pointer for thread 30003fe55e0: 2a10089efc1
[ 000002a10089efc1 cv_timedwait_sig+0x16c() ]
000002a10089f071 cv_waituntil_sig+0x8c(30003fe5786, 30003fe5788, 2a10089fa10,
2, 18f0800, 2)
000002a10089f141 lwp_park+0x130(fdffbd50, 0, 30003fe5786, 30003fe55e0, 0,
100000)
000002a10089f231 syslwp_park+0x54(0, fdffbd50, 0, 0, ff092060, 1)
000002a10089f2e1 syscall_trap32+0xcc(0, fdffbd50, 0, 0, ff092060, 1)
stack pointer for thread 30003ff4a20: 2a10083f091
[ 000002a10083f091 cv_wait_sig_swap_core+0x130() ]
000002a10083f141 lwp_park+0x130(0, 0, 30003ff4bc6, 30003ff4a20, 0, 100000)
000002a10083f231 syslwp_park+0x54(0, 0, 0, 0, ff092070, 1)
000002a10083f2e1 syscall_trap32+0xcc(0, 0, 0, 0, ff092070, 1)
[17]>

It wasn’t vxconfigd, but was ssmagent (a Netcool/Micromuse/IBM SNMP monitoring agent) instead that was sitting in a blocked i/o wait state (biowait) and effectively preventing VxVM from starting up the rootdg and mounting the encapsulated volumes.
An svcadm disable ssmagent from the running OS, fixed the problem and system now boots just fine.

 Posted by at 3:20 pm
Aug 222008
 

It’s been a while since I’ve made any posts. I am looking to migrate the site shortly (host it privately). But here’s one I thought I’d share with my thousands of anonymous readers.

Really folks…if you like what you read here…post a few lines. If you hate it…post that too!

Global Zone configuration

Add the oracle user and dba group

vistatst-01:$() # grep dba /etc/group<br /><span class="anchor" id="line-11"></span>dba::600:<br /><span class="anchor" id="line-12"></span>vistatst-01:$() # grep oracle /etc/passwd<br /><span class="anchor" id="line-13"></span>oracle:x:101:600:Oracle DBA:/u01/app/oracle:/usr/bin/ksh<br /><span class="anchor" id="line-14"></span>

Configure LUNs for Oracle ASM formatting

c4t6006016030C01C00EC2DC1FE8052DD11d0s2 is ASM disk...Let's inspect it's partition table...<br /><span class="anchor" id="line-20"></span>       2      5    01          0 104852480 104852479<br /><span class="anchor" id="line-21"></span>       6      0    00       2560 104849920 104852479<br /><span class="anchor" id="line-22"></span><br /><span class="anchor" id="line-23"></span>c4t6006016030C01C009698C4458152DD11d0s2 is ASM disk...Let's inspect it's partition table...<br /><span class="anchor" id="line-24"></span>       2      5    01          0 104852480 104852479<br /><span class="anchor" id="line-25"></span>       6      0    00       2560 104849920 104852479<br /><span class="anchor" id="line-26"></span><br /><span class="anchor" id="line-27"></span>c4t6006016030C01C0046A8487A8152DD11d0s2 is ASM disk...Let's inspect it's partition table...<br /><span class="anchor" id="line-28"></span>       2      5    01          0 104852480 104852479<br /><span class="anchor" id="line-29"></span>       6      0    00       2560 104849920 104852479<br /><span class="anchor" id="line-30"></span><br /><span class="anchor" id="line-31"></span>c4t6006016030C01C00CEF5BFAA8152DD11d0s2 is ASM disk...Let's inspect it's partition table...<br /><span class="anchor" id="line-32"></span>       2      5    01          0 104852480 104852479<br /><span class="anchor" id="line-33"></span>       6      0    00       2560 104849920 104852479<br /><span class="anchor" id="line-34"></span><br /><span class="anchor" id="line-35"></span>vistatst-01:$() #<br /><span class="anchor" id="line-36"></span>

(!) Here we have chosen to set up slice 6 as the “disk” with cylinder 0 being excluded by manually partitioning it. The starting sector count (as you can observe is 2560 instead of sector 0).

Make oracle:dba the owner of these LUNS.

vistatst-01:$() # for i in `vxdisk list|grep invalid|awk '{print $1}><br /><span class="anchor" id="line-44"></span>lrwxrwxrwx   1 oracle   dba           67 Jul 15 16:50 /dev/rdsk/c4t6006016030C01C00EC2DC1FE8052DD11d0s2 -> ../../devices/scsi_vhci/ssd@g6006016030c01c00ec2dc1fe8052dd11:c,raw<br /><span class="anchor" id="line-45"></span><br /><span class="anchor" id="line-46"></span>lrwxrwxrwx   1 oracle   dba           67 Jul 15 16:50 /dev/rdsk/c4t6006016030C01C009698C4458152DD11d0s2 -> ../../devices/scsi_vhci/ssd@g6006016030c01c009698c4458152dd11:c,raw<br /><span class="anchor" id="line-47"></span><br /><span class="anchor" id="line-48"></span>lrwxrwxrwx   1 oracle   dba           67 Jul 15 16:50 /dev/rdsk/c4t6006016030C01C0046A8487A8152DD11d0s2 -> ../../devices/scsi_vhci/ssd@g6006016030c01c0046a8487a8152dd11:c,raw<br /><span class="anchor" id="line-49"></span><br /><span class="anchor" id="line-50"></span>lrwxrwxrwx   1 oracle   dba           67 Jul 15 16:50 /dev/rdsk/c4t6006016030C01C00CEF5BFAA8152DD11d0s2 -> ../../devices/scsi_vhci/ssd@g6006016030c01c00cef5bfaa8152dd11:c,raw<br /><span class="anchor" id="line-51"></span>

And change the ownerships of the underlying files…

crw-r-----   1 oracle   dba      118, 426 Aug 14 13:41 devices/scsi_vhci/ssd@g6006016030c01c00ec2dc1fe8052dd11:c,raw<br /><span class="anchor" id="line-57"></span>crw-r-----   1 oracle   dba      118, 418 Aug 14 13:41 devices/scsi_vhci/ssd@g6006016030c01c009698c4458152dd11:c,raw<br /><span class="anchor" id="line-58"></span>crw-r-----   1 oracle   dba      118, 402 Aug 14 13:41 devices/scsi_vhci/ssd@g6006016030c01c0046a8487a8152dd11:c,raw<br /><span class="anchor" id="line-59"></span>crw-r-----   1 oracle   dba      118, 434 Aug 14 13:41 devices/scsi_vhci/ssd@g6006016030c01c00cef5bfaa8152dd11:c,raw<br /><span class="anchor" id="line-60"></span>

Configure Solaris Non-Global Zone (NGZ)

vistatst-01:$() # zonecfg -z vistatst-01-z01 set limitpriv=def<span class="anchor" id="line-66"></span>ault,proc_priocntl<br />vistatst-01:$() # zonecfg -z vistatst-01-z01 export<br /><span class="anchor" id="line-67"></span>create -b<br /><span class="anchor" id="line-68"></span>set zonepath=/zonefiles/vistatst-01-z01<br /><span class="anchor" id="line-69"></span>set autoboot=true<br /><span class="anchor" id="line-70"></span>set limitpriv=default,proc_priocntl<br /><span class="anchor" id="line-71"></span>set ip-type=shared<br /><span class="anchor" id="line-72"></span>add net<br /><span class="anchor" id="line-73"></span>set address=10.10.10.119<br /><span class="anchor" id="line-74"></span>set physical=ce0<br /><span class="anchor" id="line-75"></span>end<br /><span class="anchor" id="line-76"></span>add device<br /><span class="anchor" id="line-77"></span>set match=/dev/rdsk/c4t6006016030C01C00EC2DC1FE8052DD11d0s6<br /><span class="anchor" id="line-78"></span>end<br /><span class="anchor" id="line-79"></span>add device<br /><span class="anchor" id="line-80"></span>set match=/dev/dsk/c4t6006016030C01C00EC2DC1FE8052DD11d0s6<br /><span class="anchor" id="line-81"></span>end<br /><span class="anchor" id="line-82"></span>add device<br /><span class="anchor" id="line-83"></span>set match=/dev/rdsk/c4t6006016030C01C009698C4458152DD11d0s6<br /><span class="anchor" id="line-84"></span>end<br /><span class="anchor" id="line-85"></span>add device<br /><span class="anchor" id="line-86"></span>set match=/dev/dsk/c4t6006016030C01C009698C4458152DD11d0s6<br /><span class="anchor" id="line-87"></span>end<br /><span class="anchor" id="line-88"></span>add device<br /><span class="anchor" id="line-89"></span>set match=/dev/rdsk/c4t6006016030C01C0046A8487A8152DD11d0s6<br /><span class="anchor" id="line-90"></span>end<br /><span class="anchor" id="line-91"></span>add device<br /><span class="anchor" id="line-92"></span>set match=/dev/dsk/c4t6006016030C01C0046A8487A8152DD11d0s6<br /><span class="anchor" id="line-93"></span>end<br /><span class="anchor" id="line-94"></span>add device<br /><span class="anchor" id="line-95"></span>set match=/dev/rdsk/c4t6006016030C01C00CEF5BFAA8152DD11d0s6<br /><span class="anchor" id="line-96"></span>end<br /><span class="anchor" id="line-97"></span>add device<br /><span class="anchor" id="line-98"></span>set match=/dev/dsk/c4t6006016030C01C00CEF5BFAA8152DD11d0s6<br /><span class="anchor" id="line-99"></span>end<br /><span class="anchor" id="line-100"></span>add dataset<br /><span class="anchor" id="line-101"></span>set name=vinmarttstpool<br /><span class="anchor" id="line-102"></span>end<br /><span class="anchor" id="line-103"></span>vistatst-01:$() #<br /><span class="anchor" id="line-104"></span>vistatst-01:$() # zoneadm -z vistatst-01-z01 reboot<br /><span class="anchor" id="line-105"></span>

(!) Set up CSSD to start up

/!\ If CSSD doesn’t start up at this point — run the following command from $ORACLE_HOME/bin

#./localconfig reset<br /><span class="anchor" id="line-113"></span>vinmartst-01:$(bin) # ./localconfig reset<br /><span class="anchor" id="line-114"></span>Successfully accumulated necessary OCR keys.<br /><span class="anchor" id="line-115"></span>Creating OCR keys for user 'root', privgrp 'root'..<br /><span class="anchor" id="line-116"></span>Operation successful.<br /><span class="anchor" id="line-117"></span>Configuration for local CSS has been initialized<br /><span class="anchor" id="line-118"></span><br /><span class="anchor" id="line-119"></span>Adding to inittab<br /><span class="anchor" id="line-120"></span>Aug 14 11:58:36 vinmartst-01 root: Oracle Cluster Synchronization Service starting by user request.<br /><span class="anchor" id="line-121"></span>Startup will be queued to init within 30 seconds.<br /><span class="anchor" id="line-122"></span>Checking the status of new Oracle init process...<br /><span class="anchor" id="line-123"></span>Expecting the CRS daemons to be up within 600 seconds.<br /><span class="anchor" id="line-124"></span>Aug 14 11:58:36 vinmartst-01 root: Cluster Ready Services completed waiting on dependencies.<br /><span class="anchor" id="line-125"></span>CSS is active on these nodes.<br /><span class="anchor" id="line-126"></span>        vinmartst-01<br /><span class="anchor" id="line-127"></span>CSS is active on all nodes.<br /><span class="anchor" id="line-128"></span>Oracle CSS service is installed and running under init(1M)<br /><span class="anchor" id="line-129"></span>

Verify that cssd is running in the process table —

vinmartst-01:$(bin) # ps -ef|grep css<br /><span class="anchor" id="line-135"></span>  oracle  9120  8207   0 11:58:36 ?           0:00 /u01/app/oracle/product/10.2.0/db_1/bin/ocssd.bin<br /><span class="anchor" id="line-136"></span>    root  9220  8373   0 11:59:59 console     0:00 grep css<br /><span class="anchor" id="line-137"></span>

Verify that the LUNs are visible to the NGZ —

vinmartst-01:$(.oracle) # ls -alrt /dev/rdsk<br /><span class="anchor" id="line-143"></span>total 4<br /><span class="anchor" id="line-144"></span>drwxr-xr-x   2 root     root         512 Aug 12 16:34 .<br /><span class="anchor" id="line-145"></span>drwxr-xr-x  12 root     root        1024 Aug 14 11:57 ..<br /><span class="anchor" id="line-146"></span>crw-r-----   1 oracle   dba      118, 422 Aug 14 13:46 c4t6006016030C01C009698C4458152DD11d0s6<br /><span class="anchor" id="line-147"></span>crw-r-----   1 oracle   dba      118, 430 Aug 14 13:55 c4t6006016030C01C00EC2DC1FE8052DD11d0s6<br /><span class="anchor" id="line-148"></span>crw-r-----   1 oracle   dba      118, 438 Aug 14 14:10 c4t6006016030C01C00CEF5BFAA8152DD11d0s6<br /><span class="anchor" id="line-149"></span>crw-r-----   1 oracle   dba      118, 406 Aug 14 14:10 c4t6006016030C01C0046A8487A8152DD11d0s6<br /><span class="anchor" id="line-150"></span>

vinmartst-01:$(.oracle) # ls -lart /dev/dsk<br /><span class="anchor" id="line-154"></span>total 4<br /><span class="anchor" id="line-155"></span>brw-r-----   1 oracle   dba      118, 430 Aug 12 16:34 c4t6006016030C01C00EC2DC1FE8052DD11d0s6<br /><span class="anchor" id="line-156"></span>brw-r-----   1 oracle   dba      118, 438 Aug 12 16:34 c4t6006016030C01C00CEF5BFAA8152DD11d0s6<br /><span class="anchor" id="line-157"></span>brw-r-----   1 oracle   dba      118, 422 Aug 12 16:34 c4t6006016030C01C009698C4458152DD11d0s6<br /><span class="anchor" id="line-158"></span>brw-r-----   1 oracle   dba      118, 406 Aug 12 16:34 c4t6006016030C01C0046A8487A8152DD11d0s6<br /><span class="anchor" id="line-159"></span>drwxr-xr-x   2 root     root         512 Aug 12 16:34 .<br /><span class="anchor" id="line-160"></span>drwxr-xr-x  12 root     root        1024 Aug 14 11:57 ..<br /><span class="anchor" id="line-161"></span>

/!\ Run Oracle Installer and do the rest of the stuff that needs to be done!

References

 Posted by at 5:16 pm