Clustered LDOM Configuration generic

The uxpsh application

         <site A>                                                         <site B>
        _______________                                                     ________________
       |   bastion1  |                                                     | bastion 2      |
       |               |                                                   |                |
       |               |                                                   |                |
       |               |                                                   |                |
       |_______________|                                                   |________________|
                    \ _____                                                        _____/
                     |/var |                                                      |/var |
                     |_____|                                                      |_____|
                           \ _______                                      _______ /
                            |/uxadm |=================================>  |/uxadm |
                            |_______|          replicated                |_______|
                                               via ssh tunneled
                                               rsync every day

The password vault tool uses the “Crypt::Simple” perl module (don’t go by the name, it’s got a bunch of stuff happening under the hood) to encrypt anything that is typed into the interface and saves it on a file-system acl’ed back end store.

Only a select group (the System Admins) have ability to access this backend store (as a result the Vault application). The appliction has been written to search, modify, add and delete entries and each action is audited in the backend repository. Using a simply Curses based menu, the app is self-intuitive and requires no training.

About Crypt::Simple

• The name of the program is used as the key to create an MD5 hash of the string that is being encrypted.
• FreezeThaw is used to squish all the data into a concise textual representation
• Compress::Zlib is used to compress this string
• Crypt::Blowfish in a home-brew CBC mode is used to actually encrypt this string
• Then the string is processed via Mime::Base64 to actually produce the text that is being displayed.

For instance,

password123 becomes ==> 4vXAfgpKoyEZN67CpgmATZgV4nXirSvhzCWwQSY54GF2Wol0noNCqIujvUXCRVWj

Get the application here:

uxpsh

The Code is here:

#!/opt/perl/bin/perl

#——————————————————————————
# uxpsh — UNIX password shell
# Author — Dwai Lahiri
# Created — 4/13/2004
# Credits to : Steven L. Kunz’s perlmenu demo program(s)
# Note: Even though advisory exclusive write locks are in place,
# have to make sure file doesn’t get clobbered by simultaneous writes
# Modified — 4/27/2004 — DL
# Cleaned up some old declarations, variables, etc - 6/25/04 - DL
# Minor fixes to “cleanup” routine - 6/25/04 - DL
# SCCS Delta 1.3
# $Id: uxpsh,v 1.1 2006/01/16 18:16:02 dlahiri Exp $
#——————————————————————————

#use strict;                    #For now, don’t use strict subs
BEGIN { $Curses::OldCurses = 1; }
use lib “/opt/perl/lib/site_perl/5.8.8/sun4-solaris”;
use Curses;      #PerlMenu needs “Curses”
use perlmenu;    #Main menu package

use Crypt::Simple;
use File::Copy qw/mv cp/;
require “menuutil.pl”;    #For “pause” and “print_nl” routines

my $pwfl = “/var/uxadm/usccpux.txt”;

$SIG{’INT’} = ‘cleanup’;    #Set Signal Handler
$| = 1;                     #Flush after every write to stdout

my $window = &initscr();

&menu_curses_application($window);
&menu_quit_routine(”endwin”);

#Default prefs active at start
my $numbered_flag = 1;            #Numbered menus
my $num_pref      = “numbered”;
my $gopher_pref   = “default”;    #Non-gopherlike arrows/scrolling
my $gopher_flag   = 0;
my $mult_pref     = “single”;     #Single column menus
my $mult_flag     = 0;
my $arrow_pref    = “arrow”;      #Arrow selection indicator menus
my $arrow_flag    = 1;

my $menu_default_top = 0;              # Storage for mainline top item number.
my $menu_default_row = 0;              # Storage for mainline arrow location.
my $menu_default_col = 0;              # Storage for mainline arrow location.
my $row              = my $col = 0;    # Storage for row/col for menuutil.pl
my $title_cnt = 0;    # To trigger different subtitles/bottom titles
my $default;
my $version = “v0.5″;

@input_data   = ();
@display_data = ();
@protect      = ();
$bell         = “\007″;
$default      = “”;
my $null = “/dev/null”;
my $src;
my $dest;
my $username = $ENV{LOGNAME};

while (1) {
&menu_init(
$numbered_flag, “UNIX Password Shell Utility $version”,
0, “”, “– WARNING $username! If you don’t have permission to use this application, \n exit immediately!\n Abusers will be prosecuted to the maximum extent of the Law.\n All actions are logged and monitored. “,
“main_menu_help”
);

&menu_paint_file( “/usr/bin/unix_text”, 0 );
&menu_item( “Exit this shell”,                “exit” );
&menu_item( “Search records for a hostname”, “host_pw” );
&menu_item( “host modify”,                    “mod_host” );
&menu_item( “host add”,                       “add_host” );
&menu_item( “host delete”,                    “del_host” );
my $sel =
&menu_display( “”, $menu_default_row, $menu_default_top,
$menu_default_col );
if ( $sel eq “exit” ) { last; }

if ( $sel eq “%EMPTY%” ) {
die “Not enough screen lines to display passwd shell menu\n”;
}
if ( $sel ne “%UP%” ) {

#Note that this assumes the “action_text” is a subroutine name
&$sel();
}
if ( $sel eq “host_pw” )  { &host_pw(); }
if ( $sel eq “mod_host” ) { &mod_host(); }
if ( $sel eq “add_host” ) { &add_host(); }
if ( $sel eq “del_host” ) { &del_host(); }
}
endwin;
exit;

sub main_menu_help {
my ( $item_text, $item_tag ) = @_;

&top_title(”Password shell — Help Screen for Specific Menu Items”);
&print_nl( “Selection \”$item_text\”", 2 );
if ( $item_tag eq “exit” ) {
&print_nl( “Selecting this item will immediately exit this program.”,
1 );
}
elsif ( $item_tag eq “host_pw” ) {
&print_nl( “Search password for given host string — returns value only if valid hostname in the repository”,
1 );
}
elsif ( $item_tag eq “mod_host” ) {
print_nl( “Modify an existing password entry.”, 1 );
}
elsif ( $item_tag eq “add_host” ) {
print_nl( “Add a password entry to the encrypted repository.”, 1 );
}
elsif ( $item_tag eq “del_host” ) {
print_nl( “Delete a password entry from the ecrypted repository.”, 1 );
}
pause(”(Press any key to exit help)”);
}

sub host_pw {
my $sel;
while (1) {
my $prow = $row;
my $pcol = $col + 2;

#Init a numbered menu with a title
&menu_init( $numbered_flag,
“Search database for host string entered here” );

#Add item to return to main menu
&menu_item( “Exit”, “exit” );

&menu_item( “Host string search”, “get_host_str” );

$sel =
menu_display( “”, $menu_default_row, $menu_default_top,
$menu_default_col );
if ( ( $sel eq “%UP%” ) || ( $sel eq “exit” ) ) { return; }
if ( $sel eq “get_host_str” ) {
print_nl( “    Enter the hostname you’re looking for”, 1 );
&print_nl( “  Supply a null value to exit.”, 2 );
$prow = $row;
$pcol = $col + 2;
my $hostname = &menu_getstr( $prow, $pcol, “Enter hostname: “,
0, $default, 20, 0 );
last if ( $hostname eq “” );
decreep($hostname);
}
&clear_screen();
}
}

sub mod_host {
my $sel;
while (1) {
my $prow = $row;
my $pcol = $col + 2;

#Init a numbered menu with a title
&menu_init( $numbered_flag,
“Modify a host/password entry in the encrypted file” );

#Add item to return to main menu
&menu_item( “Exit”, “exit” );

&menu_item( “Modify host/passwd entry”, “mod_host_str” );

$sel =
menu_display( “”, $menu_default_row, $menu_default_top,
$menu_default_col );
if ( ( $sel eq “%UP%” ) || ( $sel eq “exit” ) ) { return; }
if ( $sel eq “mod_host_str” ) {
print_nl( “    Enter the hostname you want to modify”, 1 );
&print_nl( “  Supply a null value to exit.”, 2 );
$prow = $row;
$pcol = $col + 2;
my $hostname = &menu_getstr( $prow, $pcol, “Enter hostname: “,
0, $default, 20, 0 );
print_nl( “    Enter the new password”, 1 );
$prow = $row;
$pcol = $col + 2;
my $password = &menu_getstr( $prow, $pcol, “Enter new password: “,
0, $default, 20, 0 );
last if ( $hostname eq “” );
modit( $hostname, $password );
}
&clear_screen();
}
}

sub add_host {
my $sel;
while (1) {
my $prow = $row;
my $pcol = $col + 2;

#Init a numbered menu with a title
&menu_init( $numbered_flag,
“Add a host/password entry in the encrypted repository” );

#Add item to return to main menu
&menu_item( “Exit”, “exit” );

&menu_item( “Add host/passwd entry”, “add_host_str” );

$sel =
menu_display( “”, $menu_default_row, $menu_default_top,
$menu_default_col );
if ( ( $sel eq “%UP%” ) || ( $sel eq “exit” ) ) { return; }
if ( $sel eq “add_host_str” ) {
print_nl( “    Enter the hostname you want to add”, 1 );
&print_nl( “  Supply a null value to exit.”, 2 );
$prow = $row;
$pcol = $col + 2;
my $hostname = &menu_getstr( $prow, $pcol, “Enter hostname: “,
0, $default, 20, 0 );
print_nl( “    Enter the password for the given hostname”, 1 );
$prow = $row;
$pcol = $col + 2;
my $password = &menu_getstr( $prow, $pcol, “Enter password: “,
0, $default, 20, 0 );
last if ( $hostname eq “” );
addit( $hostname, $password );
}
&clear_screen();
}
}

sub del_host {
my $sel;
while (1) {
my $prow = $row;
my $pcol = $col + 2;

#Init a numbered menu with a title
&menu_init( $numbered_flag,
“Delete an existing entry from encrypted repository” );

#Add item to return to main menu
&menu_item( “Exit”, “exit” );

&menu_item( “delete host/passwd entry”, “del_host_str” );

$sel =
menu_display( “”, $menu_default_row, $menu_default_top,
$menu_default_col );
if ( ( $sel eq “%UP%” ) || ( $sel eq “exit” ) ) { return; }
if ( $sel eq “del_host_str” ) {
print_nl( “    Enter the hostname you want to delete”, 1 );
&print_nl( “  Supply a null value to exit.”, 2 );
$prow = $row;
$pcol = $col + 2;
my $hostname = &menu_getstr( $prow, $pcol, “Enter host name: “,
0, $default, 20, 0 );
last if ( $hostname eq “” );
delit($hostname);
}
&clear_screen();
}
}

sub handleit {
my $string = shift;
if ( !&display_entry($string) ) {
&new_line(1);
&pause(
“  Login \”$string\” not found - Press any key to continue $bell”);
$default = $string;
}
else { $default = “”; }
}

sub decreep {
require ‘menuutil.pl’;
my $host = shift;
my @arr;
chomp $host;
open( SRCHPWFL, “< $pwfl” ) || die “Unable to open $pwfl: $!\n”;
for (<SRCHPWFL>) {
if ( $_ =~ m/^$host/i ) {
@arr = split( ‘:’, $_ );
my $password = decrypt( $arr[1] );
menu_display(”Password for $arr[0] is : $password”);
}
}
close(SRCHPWFL);
}

sub modit {
my ( $host, $password ) = @_;
my @arr;
my $date = qx/date ‘+%m%d%Y%H%M’/;
chomp $date;
$src  = $pwfl;
$dest = $pwfl . “.” . $date;
chomp $dest;
cp( “$src”, “$dest” );
open( MOD1PWFL, “< $dest” ) || menu_display(”Unable to open $dest: $!\n”);
open( MODPWFL,  “>> $src” ) || menu_display(”Unable to open $src: $!\n”);
system(”> $src”);    #Zero out the actual password file
flock(MODPWFL, 2) or menu_display(”Unable to get an exclusive write lock on $src: $! \n”);

if ( ( !-z $host ) && ( !-z $password ) ) {
my $encrypted = encrypt($password);
for (<MOD1PWFL>) {
print MODPWFL “$_” and next if /^#/;
my @lsplit = split(’:', $_);
if ( $lsplit[0] eq $host ) {
$lsplit[1] = $encrypted;
print MODPWFL “# $username Modified $host entry — $date \n”;
print MODPWFL “$host:$lsplit[1]\n”;
}
else {
print MODPWFL “$_”;
}
}
}
close(MOD1PWFL);
close(MODPWFL);
}

sub addit {
my ( $host, $password ) = @_;
my $date = qx/date ‘+%m%d%Y%H%M’/;
chomp $date;
$src  = $pwfl;
$dest = $pwfl . “.” . $date;
chomp $dest;
cp( “$src”, “$dest” );
open(READPWFL, “< $dest”);
open( ADDPWFL, “>> $src” ) || menu_display(”Unable to open $src : $!\n”);
flock(ADDPWFL, 2) or menu_display(”Unable to get an exclusive write lock on: $src \n”);

if ( ( !-z $host ) && ( !-z $password ) ) {
my $encrypted = encrypt($password);
for (<READPWFL>) {
next  if /^#/o;
my @lsplit = split(’:',$_);
if ( $lsplit[0]  eq $host ) {
menu_display(”$host already has an entry in : $src \n”);
return 1;
}
}
if ($? != 1) {
print ADDPWFL “# $username added entry for $host - $date \n”;
print ADDPWFL “$host:$encrypted\n”;
}
}
close(ADDPWFL);
}

sub delit {
my $host = shift;
my @arr;
chomp $host;
my $date = qx/date ‘+%m%d%Y%H%M’/;
chomp $date;
$src  = $pwfl;
$dest = $pwfl . “.” . $date;
chomp $dest;
cp( “$src”, “$dest” );
open( DEL1PWFL, “< $dest” ) || menu_display(”Unable to open $dest: $!\n”);
open( DELPWFL,  “>> $src” ) || menu_display(”Unable to open $src: $!\n”);
system(”> $src”);    #Zero out actual pw file
flock(DELPWFL, 2) or menu_display(”Unable to get an exclusive write lock on : $src \n”);

for (<DEL1PWFL>) {
print DELPWFL “$_” and next  if /^#/;
my @lsplit = split(’:',$_);
if ( $lsplit[0] ne  $host ) {
print DELPWFL “$_” ;
}
else {
print DELPWFL “# $username deleted entry for $host - $date \n”;
}
}
close(DEL1PWFL);
close(DELPWFL);
}

sub cleanup {
&clear_screen;
wrefresh;
endwin;
exit;
}

sub inwait {
ReadMode(’cbreak’);
if ( defined( $char = ReadKey(-1) ) ) {
print $char;
}
else {
print “No input waiting\n”;
}
ReadMode(’normal’);
}

I had presented this piece in a Lunch and Learn session with my team.

Solaris Virtualization

Download latest flash image

http://www.sun.com/download/products.xml?id=46fc425e

Download to usb drive.

I had to rename the file “[]” were put into the name on download.

I made a directory named images

Moved renamed file (FFXCP1071.tar.gz) into images directory.

(The usb port is on the XSCF card in rear of system)

Load image into system memory:

XSCF>getflashimage file:///media/usb_msd/images/FFXCP1071.tar.gz

Answer: Y (Loads image into memory)

Install image:

XSCF>flashupdate -c update -m xcp -s 1071

Answer: Y

We noticed that our systems were not booting up the normal way. The only way to boot was to “trick” the system into booting into single-user mode and then exiting out to the multi-user run-level.

This was a problem happening only on systems running Solaris 10, with Veritas Volume Manager 5.0 MP1. So naturally we leaned towards tackling this first as a possible VxVM problem (introduced by some patch during the patch cycle). And the patch rev we had decided to apply had the Veritas Storage Foundation 5.0 MP1 RP4 in it.

Again, to reiterate the problem description –

Description:

Systems running Solaris 10 and Veritas Volume Manager tend to hang when booted with a normal init 6 (ie while booting into Run Level 3).

Workaround:

Boot the system in run level S (single user mode) and then exit out of it to boot into multi-user mode.

Details:

Opened a case with Veritas and sent them VRTSexplorer outputs and copies of our messages file from problem host nodeA.

After several iterations of generating explorers and veritas configuration information, Veritas still didn’t have anything substantial they could pin this issue on. They asked me to generate a crash dump of the hanging system (usually should be able to coredump a running host by breaking it and running “sync” from the OBP).

After repeated attempts at generating the core, I was unable to do so. It seems like the system hangs before the dump device is initialized/configured by the OS.

Using the work around, I enabled the solaris deadman timer (which incidentally we should have on all our servers). This involves setting the following line in the /etc/system file —

* set snooping = 1

What the deadman timer does is sents a high-priority hardware interrupt to each CPU (or Core or strand depending on the platform) and updates a counter upon successful response by the CPU. In case of a system being hung (due to hardware issues especially), this count might not increase with an increase in clock tick (interrupts aresent to CPUs every tick). When this counter is not incremented, the kernel panics and kills itself.

This didn’t work because we didn’t encounter that kind of a problem (but is a good idea to have enabled nonetheless).

Another alternative for panicking a system is as follows

Boot the server in kernel debug mode (in SPARC systems it is done by running boot -k from the OBP). This loads the kmdb module into the kernel as it boots up. Breaking the system while in kernel debug mode will not drop it into the OBP but instead launch an mdb interface.

One can generate a coredump by running $ < i="" tried="" that="" but="" no="">

[17]> $ <>
panic[cpu17]/thread=2a101a3fca0: BAD TRAP: type=9 rp=2a101a3f760 addr=0 mmu_fsr=0

sched: trap type = 0×9
pid=0, pc=0×0, sp=0×2a101a3f001, tstate=0×1606, context=0×0
g1-g7: 1011be4, 1, 1870000, 20040, 1860c00, 0, 2a101a3fca0

000002a101a3f480 unix:die+9c (9, 2a101a3f760, 0, 0, 2a101a3f540, 1901ec0)
%l0-3: 0000000000000000 0000000000000009 0000000000000009 0000000000000000
%l4-7: 0000000000000000 00000000018f1800 0000000000001606 0000000001097000
000002a101a3f560 unix:trap+6cc (2a101a3f760, 10000, 0, 0, 30003dac000, 2a101a3fca0)
%l0-3: 0000000000000000 0000000001859480 0000000000000009 0000000000000000
%l4-7: 0000000000000000 00000000018f1800 0000000000001606 0000000000010200
000002a101a3f6b0 unix:ktl0+64 (0, 20040, 1860c00, 1, 1260ce4, 1901ec0)
%l0-3: 0000030003dac000 0000000000000088 0000000000001606 0000000001020354
%l4-7: 0000000000000000 00000000018f1800 000000000000000c 000002a101a3f760
000002a101a3f800 unix:debug_enter+108 (0, a, a, 1826400, 0, 1870000)
%l0-3: 0000000001834060 00000048df61937c 0000000000000000 0000000000000000
%l4-7: 0000000000000000 00000000018f1800 0000000000000000 0000000001011b30
000002a101a3f8d0 unix:abort_seq_softintr+94 (1826400, 18f1800, 30003dac000, 2a101a3fd78, 1, 189f800)
%l0-3: 0000000001834060 00000048df61937c 0000000000000000 0000000000000000
%l4-7: 0000000000000000 00000000018f1800 0000000000000000 0000000001011b30

syncing file systems… done
skipping system dump - no dump device configured
rebooting…

SC Alert: Host System has Reset

SC Alert: Failed to send email alert for recent event.

SC Alert: Indicator SYS/ACT is now STANDBY BLINK

So back to the kernel debug mode again. This time I decided to try and investigate the kernel (you can do this against a coredump as well).

[17]> ::msgbuf
MESSAGE
/pci@780/pci@0/pci@9/scsi@0 (mpt0):
mpt0 supports power management.
/pci@780/pci@0/pci@9/scsi@0 (mpt0):
DMA restricted to lower 4GB due to errata
/pci@780/pci@0/pci@9/scsi@0 (mpt0):
mpt0 Firmware version v1.9.0.0 (IR)
/pci@780/pci@0/pci@9/scsi@0 (mpt0):
mpt0: IOC Operational.
/pci@780/pci@0/pci@9/scsi@0 (mpt0):
mpt0: Initiator WWNs: 0×5080020000262858-0×508002000026285b
PCI-device: scsi@0, mpt0
mpt0 is /pci@780/pci@0/pci@9/scsi@0
sd1 at mpt0: target 0 lun 0
sd1 is /pci@780/pci@0/pci@9/scsi@0/sd@0,0
/pci@780/pci@0/pci@9/scsi@0/sd@0,0 (sd1) online
root on /pci@780/pci@0/pci@9/scsi@0/disk@0,0:a fstype ufs
px1 at root: 0×7c0 0×0
px1 is /pci@7c0
PCI Express-device: pci@0, pxb_plx5
pxb_plx5 is /pci@7c0/pci@0
PCI-device: pci@1, pxb_plx6
pxb_plx6 is /pci@7c0/pci@0/pci@1
PCI-device: pci@0, px_pci0
px_pci0 is /pci@7c0/pci@0/pci@1/pci@0
PCI-device: ide@8, uata0
uata0 is /pci@7c0/pci@0/pci@1/pci@0/ide@8
WARNING: px1: spurious interrupt from ino 0×4
uata-0#0

sd2 at uata0: target 0 lun 0
sd2 is /pci@7c0/pci@0/pci@1/pci@0/ide@8/sd@0,0
sd3 at mpt0: target 1 lun 0
sd3 is /pci@780/pci@0/pci@9/scsi@0/sd@1,0
/pci@780/pci@0/pci@9/scsi@0/sd@1,0 (sd3) online
sd4 at mpt0: target 2 lun 0
sd4 is /pci@780/pci@0/pci@9/scsi@0/sd@2,0
/pci@780/pci@0/pci@9/scsi@0/sd@2,0 (sd4) online
sd5 at mpt0: target 3 lun 0
sd5 is /pci@780/pci@0/pci@9/scsi@0/sd@3,0
/pci@780/pci@0/pci@9/scsi@0/sd@3,0 (sd5) online
iscsi0 at root
iscsi0 is /iscsi
root remounted on /pseudo/vxio@0:0 fstype ufs
virtual-device: cnex0
cnex0 is /virtual-devices@100/channel-devices@200
PCI-device: isa@2, ebus0
ebus0 is /pci@7c0/pci@0/pci@1/pci@0/isa@2
pseudo-device: dld0
dld0 is /pseudo/dld@0
PCI-device: usb@5, ohci0
ohci0 is /pci@7c0/pci@0/pci@1/pci@0/usb@5
PCI-device: usb@6, ohci1
ohci1 is /pci@7c0/pci@0/pci@1/pci@0/usb@6
cpu0: UltraSPARC-T1 (cpuid 0 clock 1000 MHz)
cpu1: UltraSPARC-T1 (cpuid 1 clock 1000 MHz)
cpu1 initialization complete - online
cpu2: UltraSPARC-T1 (cpuid 2 clock 1000 MHz)
cpu2 initialization complete - online
cpu3: UltraSPARC-T1 (cpuid 3 clock 1000 MHz)
cpu3 initialization complete - online
cpu4: UltraSPARC-T1 (cpuid 4 clock 1000 MHz)
cpu4 initialization complete - online
cpu5: UltraSPARC-T1 (cpuid 5 clock 1000 MHz)
cpu5 initialization complete - online
cpu6: UltraSPARC-T1 (cpuid 6 clock 1000 MHz)
cpu6 initialization complete - online
cpu7: UltraSPARC-T1 (cpuid 7 clock 1000 MHz)
cpu7 initialization complete - online
cpu8: UltraSPARC-T1 (cpuid 8 clock 1000 MHz)
cpu8 initialization complete - online
cpu9: UltraSPARC-T1 (cpuid 9 clock 1000 MHz)
cpu9 initialization complete - online
cpu10: UltraSPARC-T1 (cpuid 10 clock 1000 MHz)
cpu10 initialization complete - online
cpu11: UltraSPARC-T1 (cpuid 11 clock 1000 MHz)
cpu11 initialization complete - online
cpu12: UltraSPARC-T1 (cpuid 12 clock 1000 MHz)
cpu12 initialization complete - online
cpu13: UltraSPARC-T1 (cpuid 13 clock 1000 MHz)
cpu13 initialization complete - online
cpu14: UltraSPARC-T1 (cpuid 14 clock 1000 MHz)
cpu14 initialization complete - online
cpu15: UltraSPARC-T1 (cpuid 15 clock 1000 MHz)
cpu15 initialization complete - online
cpu16: UltraSPARC-T1 (cpuid 16 clock 1000 MHz)
cpu16 initialization complete - online
cpu17: UltraSPARC-T1 (cpuid 17 clock 1000 MHz)
cpu17 initialization complete - online
cpu18: UltraSPARC-T1 (cpuid 18 clock 1000 MHz)
cpu18 initialization complete - online
cpu19: UltraSPARC-T1 (cpuid 19 clock 1000 MHz)
cpu19 initialization complete - online
cpu20: UltraSPARC-T1 (cpuid 20 clock 1000 MHz)
cpu20 initialization complete - online
cpu21: UltraSPARC-T1 (cpuid 21 clock 1000 MHz)
cpu21 initialization complete - online
cpu22: UltraSPARC-T1 (cpuid 22 clock 1000 MHz)
cpu22 initialization complete - online
cpu23: UltraSPARC-T1 (cpuid 23 clock 1000 MHz)
cpu23 initialization complete - online
USB 1.10 device (usb3eb,3301) operating at full speed (USB 1.x) on USB 1.10 root
hub: hub@1, hubd1 at bus address 2
hubd1 is /pci@7c0/pci@0/pci@1/pci@0/usb@6/hub@1
/pci@7c0/pci@0/pci@1/pci@0/usb@6/hub@1 (hubd1) online
PCI-device: pci@0,2, px_pci1
px_pci1 is /pci@7c0/pci@0/pci@1/pci@0,2
PCI-device: pci@1, pxb_plx1
pxb_plx1 is /pci@780/pci@0/pci@1
PCI-device: pci@2, pxb_plx2
pxb_plx2 is /pci@780/pci@0/pci@2
PCI-device: pci@8, pxb_plx3
pxb_plx3 is /pci@780/pci@0/pci@8
PCI-device: pci@2, pxb_plx7
pxb_plx7 is /pci@7c0/pci@0/pci@2
PCI-device: pci@8, pxb_plx8
pxb_plx8 is /pci@7c0/pci@0/pci@8
PCI-device: pci@9, pxb_plx9
pxb_plx9 is /pci@7c0/pci@0/pci@9
NOTICE: e1000g0 registered
Intel(R) PRO/1000 Network Connection, Driver Ver. 5.1.11
NOTICE: pciex8086,105e - e1000g[0] : Adapter copper link is down.
NOTICE: e1000g1 registered
Intel(R) PRO/1000 Network Connection, Driver Ver. 5.1.11
NOTICE: pciex8086,105e - e1000g[1] : Adapter copper link is down.
NOTICE: e1000g3 registered
Intel(R) PRO/1000 Network Connection, Driver Ver. 5.1.11
NOTICE: pciex8086,105e - e1000g[3] : Adapter copper link is down.
NOTICE: pciex8086,105e - e1000g[0] : Adapter 1000Mbps full duplex copper link is
up.
NOTICE: pciex8086,105e - e1000g[1] : Adapter 1000Mbps full duplex copper link is
up.
pseudo-device: devinfo0
devinfo0 is /pseudo/devinfo@0
NOTICE: pciex8086,105e - e1000g[3] : Adapter 1000Mbps full duplex copper link is
up.
NOTICE: VxVM vxdmp V-5-0-34 added disk array DISKS, datype = Disk

NOTICE: VxVM vxdmp V-5-3-1700 dmpnode 300/0×0 has migrated from enclosure FAKE_E
NCLR_SNO to enclosure DISKS

NOTICE: e1000g2 registered
Intel(R) PRO/1000 Network Connection, Driver Ver. 5.1.11

Look at the process table…

[17]> ::ps
S PID PPID PGID SID UID FLAGS ADDR NAME
R 0 0 0 0 0 0×00000001 0000000001859480 sched
R 3 0 0 0 0 0×00020001 000006001172d838 fsflush
R 2 0 0 0 0 0×00020001 000006001172e450 pageout
R 1 0 0 0 0 0×4a004000 000006001172f068 init
R 79 1 78 78 0 0×42010000 0000060017f30028 ssmagent.bin
R 9 1 9 9 0 0×42000000 000006001292f070 svc.configd
R 7 1 7 7 0 0×42000000 000006001172c008 svc.startd
R 54 7 7 7 0 0×4a004000 0000060017e43080 vxvm-sysboot
R 56 54 7 7 0 0×4a004000 0000060017e42468 vxconfigd
R 57 56 57 57 0 0×42020000 0000060017dc8018 vxconfigd

Pick a suspect thread and dig deeper –

[17]> 0000060017e43080::findstack -v
kmdb: thread 60017e43080 isn’t in memory
[17]> 0000060017e43080::walk thread
30003eba480
[17]> 0000060017e43080::walk thread| ::findstack -v
stack pointer for thread 30003eba480: 2a101514ff1
[ 000002a101514ff1 cv_wait_sig_swap_core+0x130() ]
000002a1015150a1 waitid+0×484(0, 60017e42468, 0, 60017e430e8, 0, 1)
000002a101515171 waitsys32+0×10(0, 38, ffbffa80, 83, 39590, 3a57c)
000002a1015152e1 syscall_trap32+0xcc(0, 38, ffbffa80, 83, 39590, 3a57c)
[17]>

[17]> 0000060017e42468::walk thread| ::findstack -v
stack pointer for thread 30003fae200: 2a101dbeff1
[ 000002a101dbeff1 cv_wait_sig_swap_core+0x130() ]
000002a101dbf0a1 waitid+0×484(0, 0, 0, 60017e424d0, 7, 1)
000002a101dbf171 waitsys32+0×10(7, 0, ffbffaa8, 3, 2909e8, 0)
000002a101dbf2e1 syscall_trap32+0xcc(7, 0, ffbffaa8, 3, 2909e8, 0)

[17]> 0000060017dc8018::walk thread | ::findstack -v
stack pointer for thread 30003eb0420: 2a10072ec21
[ 000002a10072ec21 cv_wait+0x38() ]
000002a10072ecd1 delay+0×90(1, 1870000, df39, df3a, 1, 18f0818)
000002a10072ed81 vxio`volkio_flush_cached_io+0xdc(7006ea30, 30003eb0420,
7006789c, 7006e9e8, 0, 0)
000002a10072ee31 vxio`vol_commit_flush_cached_io+0×60(60011503240, 70046d28, 0
, 0, 0, 0)
000002a10072eee1 vxio`vol_ktrans_commit+0×6fc(7006b2f0, 1, 7006789c, 7006d92c
, 60011503240, 0)
000002a10072ef91 vxio`volsioctl_real+0×4c8(70034000, 564f4c86, 0, 100003,
60011403768, 2a10072fadc)
000002a10072f0e1 fop_ioctl+0×20(60012a7f900, 564f4c86, 0, 100003, 60011403768
, 12f1770)
000002a10072f191 ioctl+0×184(1, 600115e8a50, 0, 564f4c86, 0, 564f4c86)
000002a10072f2e1 syscall_trap32+0xcc(1, 564f4c86, 0, 564f4c86, 0, 820770)
[17]>

Found something in the last one, but no immediate red flags. But then I saw this (ssmagent running). SSM Agent is a SNMP agent that is used to monitor our systems and report back to Netcool, our Fault Management server.

[17]> ::ps
S PID PPID PGID SID UID FLAGS ADDR NAME
R 0 0 0 0 0 0×00000001 0000000001859480 sched
R 3 0 0 0 0 0×00020001 000006001172d838 fsflush
R 2 0 0 0 0 0×00020001 000006001172e450 pageout
R 1 0 0 0 0 0×4a004000 000006001172f068 init
R 79 1 78 78 0 0×42010000 0000060017f5e028 ssmagent.bin
R 43 1 42 42 0 0×42020000 0000060017db7078 dhcpagent
R 9 1 9 9 0 0×42000000 0000060012922458 svc.configd
R 7 1 7 7 0 0×42000000 000006001172c008 svc.startd
R 54 7 7 7 0 0×4a004000 000006001172cc20 vxvm-sysboot
R 56 54 7 7 0 0×4a004000 0000060017e10468 vxconfigd
R 57 56 57 57 0 0×42020000 0000060017db4018 vxconfigd
[17]> 0000060017f5e028::walk thread| ::findstack -v
stack pointer for thread 30003fb55a0: 2a101e02961
[ 000002a101e02961 sema_p+0x130() ]
000002a101e02a11 biowait+0×6c(60017dda100, 0, 1870000, 30003dbe000, 2000,
60017dda100)
000002a101e02ac1 ufs`ufs_getpage_miss+0×2ec(60019a0f300, 40000, 4de,
600129cea20, fdba0000, 2a101e03760)
000002a101e02bc1 ufs`ufs_getpage+0×694(300014b7e00, 40000, 1, 0, 1, 3)
000002a101e02d21 fop_getpage+0×44(60019a0f300, 600114659c0, 60011403978, 3,
fdba0000, 3)
000002a101e02df1 segvn_fault+0xb04(8000, 600129cea20, 3, 2000, 40000, 0)
000002a101e02fc1 as_fault+0×4c8(600129cea20, 600129d9200, fdba0000,
60011736320, 189eb00, 0)
000002a101e030d1 pagefault+0×68(fdba14a8, 0, 3, 0, 60017f5e028, 600117362a8)
000002a101e03191 trap+0xd50(2a101e03b90, 10000, 0, 3, fdba14a8, 0)
000002a101e032e1 utl0+0×4c(ff3f40fc, ff3f5a70, 1, 0, ff3f4910, 821)
stack pointer for thread 30003f08820: 2a102005091
[ 000002a102005091 cv_wait_sig_swap_core+0x130() ]
000002a102005141 lwp_park+0×130(0, 1, 30003f089c6, 30003f08820, 0, 100000)
000002a102005231 syslwp_park+0×54(0, 0, 0, 0, ff092010, 1)
000002a1020052e1 syscall_trap32+0xcc(0, 0, 0, 0, ff092010, 1)
stack pointer for thread 30003fb4560: 2a101d9f091
[ 000002a101d9f091 cv_wait_sig_swap_core+0x130() ]
000002a101d9f141 lwp_park+0×130(0, 1, 30003fb4706, 30003fb4560, 0, 100000)
000002a101d9f231 syslwp_park+0×54(0, 0, 0, 0, ff092020, 1)
000002a101d9f2e1 syscall_trap32+0xcc(0, 0, 0, 0, ff092020, 1)
stack pointer for thread 30003ef77c0: 2a1008e9091
[ 000002a1008e9091 cv_wait_sig_swap_core+0x130() ]
000002a1008e9141 lwp_park+0×130(0, 1, 30003ef7966, 30003ef77c0, 0, 100000)
000002a1008e9231 syslwp_park+0×54(0, 0, 0, 0, ff092030, 1)
000002a1008e92e1 syscall_trap32+0xcc(0, 0, 0, 0, ff092030, 1)
stack pointer for thread 30003fd35c0: 2a101db6f91
[ 000002a101db6f91 cv_timedwait_sig+0x16c() ]
000002a101db7041 cv_waituntil_sig+0×8c(60017dc8592, 60017dc8558, 2a101db7ad0,
2, 18f0800, 2)
000002a101db7111 poll_common+0×4e8(60012996580, 60017f5e028, 2a101db7ad0, 0,
fe57bcd0, 2)
000002a101db7201 pollsys+0xf8(fe57bcd0, 1, fe57bd70, 0, 2a101db7ad0, 0)
000002a101db72e1 syscall_trap32+0xcc(fe57bcd0, 1, fe57bd70, 0, fe57bd70, 0)
stack pointer for thread 30003efa460: 2a101df3091
[ 000002a101df3091 cv_wait_sig_swap_core+0x130() ]
000002a101df3141 lwp_park+0×130(0, 1, 30003efa606, 30003efa460, 0, 100000)
000002a101df3231 syslwp_park+0×54(0, 0, 0, 0, ff092050, 1)
000002a101df32e1 syscall_trap32+0xcc(0, 0, 0, 0, ff092050, 1)
stack pointer for thread 30003fe55e0: 2a10089efc1
[ 000002a10089efc1 cv_timedwait_sig+0x16c() ]
000002a10089f071 cv_waituntil_sig+0×8c(30003fe5786, 30003fe5788, 2a10089fa10,
2, 18f0800, 2)
000002a10089f141 lwp_park+0×130(fdffbd50, 0, 30003fe5786, 30003fe55e0, 0,
100000)
000002a10089f231 syslwp_park+0×54(0, fdffbd50, 0, 0, ff092060, 1)
000002a10089f2e1 syscall_trap32+0xcc(0, fdffbd50, 0, 0, ff092060, 1)
stack pointer for thread 30003ff4a20: 2a10083f091
[ 000002a10083f091 cv_wait_sig_swap_core+0x130() ]
000002a10083f141 lwp_park+0×130(0, 0, 30003ff4bc6, 30003ff4a20, 0, 100000)
000002a10083f231 syslwp_park+0×54(0, 0, 0, 0, ff092070, 1)
000002a10083f2e1 syscall_trap32+0xcc(0, 0, 0, 0, ff092070, 1)
[17]>

It wasn’t vxconfigd, but was ssmagent (a Netcool/Micromuse/IBM SNMP monitoring agent) instead that was sitting in a blocked i/o wait state (biowait) and effectively preventing VxVM from starting up the rootdg and mounting the encapsulated volumes.
An svcadm disable ssmagent from the running OS, fixed the problem and system now boots just fine.

Really folks…if you like what you read here…post a few lines. If you hate it…post that too!

Global Zone configuration

Add the oracle user and dba group

vistatst-01:$() # grep dba /etc/groupdba::600:vistatst-01:$() # grep oracle /etc/passwdoracle:x:101:600:Oracle DBA:/u01/app/oracle:/usr/bin/ksh

Configure LUNs for Oracle ASM formatting

c4t6006016030C01C00EC2DC1FE8052DD11d0s2 is ASM disk...Let's inspect it's partition table...       2      5    01          0 104852480 104852479       6      0    00       2560 104849920 104852479c4t6006016030C01C009698C4458152DD11d0s2 is ASM disk...Let's inspect it's partition table...       2      5    01          0 104852480 104852479       6      0    00       2560 104849920 104852479c4t6006016030C01C0046A8487A8152DD11d0s2 is ASM disk...Let's inspect it's partition table...       2      5    01          0 104852480 104852479       6      0    00       2560 104849920 104852479c4t6006016030C01C00CEF5BFAA8152DD11d0s2 is ASM disk...Let's inspect it's partition table...       2      5    01          0 104852480 104852479       6      0    00       2560 104849920 104852479vistatst-01:$() #

Here we have chosen to set up slice 6 as the “disk” with cylinder 0 being excluded by manually partitioning it. The starting sector count (as you can observe is 2560 instead of sector 0).

Make oracle:dba the owner of these LUNS.

vistatst-01:$() # for i in `vxdisk list|grep invalid|awk '{print $1}>lrwxrwxrwx   1 oracle   dba           67 Jul 15 16:50 /dev/rdsk/c4t6006016030C01C00EC2DC1FE8052DD11d0s2 -> ../../devices/scsi_vhci/ssd@g6006016030c01c00ec2dc1fe8052dd11:c,rawlrwxrwxrwx   1 oracle   dba           67 Jul 15 16:50 /dev/rdsk/c4t6006016030C01C009698C4458152DD11d0s2 -> ../../devices/scsi_vhci/ssd@g6006016030c01c009698c4458152dd11:c,rawlrwxrwxrwx   1 oracle   dba           67 Jul 15 16:50 /dev/rdsk/c4t6006016030C01C0046A8487A8152DD11d0s2 -> ../../devices/scsi_vhci/ssd@g6006016030c01c0046a8487a8152dd11:c,rawlrwxrwxrwx   1 oracle   dba           67 Jul 15 16:50 /dev/rdsk/c4t6006016030C01C00CEF5BFAA8152DD11d0s2 -> ../../devices/scsi_vhci/ssd@g6006016030c01c00cef5bfaa8152dd11:c,raw

And change the ownerships of the underlying files…

crw-r-----   1 oracle   dba      118, 426 Aug 14 13:41 devices/scsi_vhci/ssd@g6006016030c01c00ec2dc1fe8052dd11:c,rawcrw-r-----   1 oracle   dba      118, 418 Aug 14 13:41 devices/scsi_vhci/ssd@g6006016030c01c009698c4458152dd11:c,rawcrw-r-----   1 oracle   dba      118, 402 Aug 14 13:41 devices/scsi_vhci/ssd@g6006016030c01c0046a8487a8152dd11:c,rawcrw-r-----   1 oracle   dba      118, 434 Aug 14 13:41 devices/scsi_vhci/ssd@g6006016030c01c00cef5bfaa8152dd11:c,raw

Configure Solaris Non-Global Zone (NGZ)

vistatst-01:$() # zonecfg -z vistatst-01-z01 set limitpriv=default,proc_priocntlvistatst-01:$() # zonecfg -z vistatst-01-z01 exportcreate -bset zonepath=/zonefiles/vistatst-01-z01set autoboot=trueset limitpriv=default,proc_priocntlset ip-type=sharedadd netset address=10.10.10.119set physical=ce0endadd deviceset match=/dev/rdsk/c4t6006016030C01C00EC2DC1FE8052DD11d0s6endadd deviceset match=/dev/dsk/c4t6006016030C01C00EC2DC1FE8052DD11d0s6endadd deviceset match=/dev/rdsk/c4t6006016030C01C009698C4458152DD11d0s6endadd deviceset match=/dev/dsk/c4t6006016030C01C009698C4458152DD11d0s6endadd deviceset match=/dev/rdsk/c4t6006016030C01C0046A8487A8152DD11d0s6endadd deviceset match=/dev/dsk/c4t6006016030C01C0046A8487A8152DD11d0s6endadd deviceset match=/dev/rdsk/c4t6006016030C01C00CEF5BFAA8152DD11d0s6endadd deviceset match=/dev/dsk/c4t6006016030C01C00CEF5BFAA8152DD11d0s6endadd datasetset name=vinmarttstpoolendvistatst-01:$() #vistatst-01:$() # zoneadm -z vistatst-01-z01 reboot

Set up CSSD to start up

If CSSD doesn’t start up at this point — run the following command from $ORACLE_HOME/bin —

#./localconfig resetvinmartst-01:$(bin) # ./localconfig resetSuccessfully accumulated necessary OCR keys.Creating OCR keys for user 'root', privgrp 'root'..Operation successful.Configuration for local CSS has been initializedAdding to inittabAug 14 11:58:36 vinmartst-01 root: Oracle Cluster Synchronization Service starting by user request.Startup will be queued to init within 30 seconds.Checking the status of new Oracle init process...Expecting the CRS daemons to be up within 600 seconds.Aug 14 11:58:36 vinmartst-01 root: Cluster Ready Services completed waiting on dependencies.CSS is active on these nodes.        vinmartst-01CSS is active on all nodes.Oracle CSS service is installed and running under init(1M)

Verify that cssd is running in the process table —

vinmartst-01:$(bin) # ps -ef|grep css  oracle  9120  8207   0 11:58:36 ?           0:00 /u01/app/oracle/product/10.2.0/db_1/bin/ocssd.bin    root  9220  8373   0 11:59:59 console     0:00 grep css

Verify that the LUNs are visible to the NGZ —

vinmartst-01:$(.oracle) # ls -alrt /dev/rdsktotal 4drwxr-xr-x   2 root     root         512 Aug 12 16:34 .drwxr-xr-x  12 root     root        1024 Aug 14 11:57 ..crw-r-----   1 oracle   dba      118, 422 Aug 14 13:46 c4t6006016030C01C009698C4458152DD11d0s6crw-r-----   1 oracle   dba      118, 430 Aug 14 13:55 c4t6006016030C01C00EC2DC1FE8052DD11d0s6crw-r-----   1 oracle   dba      118, 438 Aug 14 14:10 c4t6006016030C01C00CEF5BFAA8152DD11d0s6crw-r-----   1 oracle   dba      118, 406 Aug 14 14:10 c4t6006016030C01C0046A8487A8152DD11d0s6

vinmartst-01:$(.oracle) # ls -lart /dev/dsktotal 4brw-r-----   1 oracle   dba      118, 430 Aug 12 16:34 c4t6006016030C01C00EC2DC1FE8052DD11d0s6brw-r-----   1 oracle   dba      118, 438 Aug 12 16:34 c4t6006016030C01C00CEF5BFAA8152DD11d0s6brw-r-----   1 oracle   dba      118, 422 Aug 12 16:34 c4t6006016030C01C009698C4458152DD11d0s6brw-r-----   1 oracle   dba      118, 406 Aug 12 16:34 c4t6006016030C01C0046A8487A8152DD11d0s6drwxr-xr-x   2 root     root         512 Aug 12 16:34 .drwxr-xr-x  12 root     root        1024 Aug 14 11:57 ..

Run Oracle Installer and do the rest of the stuff that needs to be done!

References

• http://askdba.org/weblog/?p=126

• http://askdba.org/weblog/?p=132

• http://preview.tinyurl.com/5ukyuw

Some of the questions we asked are as follows –

• Why use a log analysis tool?
• What do most shops use
• What does a tool such as splunk buy us (as an IT shop)
• What are it’s benefits and pit-falls?
• What is the cost of ownership?

Why use a log-analysis tool?

The biggest reason to use such a tool would be to move from a Reactive to Proactive Systems Management paradigm

With the number of systems (about 900+ *nix servers in that shop) and the criticality (many systems cost millions of dollars in down-time) of availability of these, it is imperative to find a tool that can actually be used quickly and effortlessly to analyze valuable log information

If such a tool can look at various layers of a “delivered stack” (aka hardware, os, application, network, san, etc), it would be a gold-mine by virtue of being able to link the stack “end-to-end” and by speeding up the analysis process.

What do most shops use?

Most shops I’ve been in do log analysis like this –

a) Don’t do any log analysis unless absolutely required. And if it is required, admins log into the individual servers and parse through the logs using vi (or using a combination of grep/awk/sed if they are script-savvy)

b) Have a centralized ssh (or god forbid! rsh) trusted admin host from where they launch a log parser script that filters specific key words and that gets emailed to a mailbox or to the individual admins’ email boxes

c) have a centralized log host where they run a script akin to the one mentioned above

I’ve worked in shops of varying sizes — from a ISP/Telecom giant who ran 4000+ sun servers to a 50-server tiny sweatshop. Most of the shops I’ve been in fall some where in between (with hosts ranging from 200 - 1000 in number). That’s a lot of hosts to manage and a lot of logging that needs to be parsed.

What does a log-analysis tool buy an IT shop?

You’ve all probably thought about this — a centralized, easy-to-use log analysis tool buys an IT shop valuable time!

So what does Splunk claim to do?

In there own words –

“The Splunk Server indexes IT data from ANY source. No need to configure it for specific formats, write regular expressions or change your logging output. Search mountains of data by time, keywords, type of event, source, host or relationships to other events. “

Some key features of Splunk:

• Universal Indexing
• Can index terabytes of data all from one place
• Capable of indexing approx. 22,000 events/second at density of 150 bytes/event.

How does splunk acquire data?

Access data from any live source:

• Mounted files: NFS/SMB, CIFS/AFP, NAS/SAN, FIFO,
• Remote files: rsync, scp/ftp/rcp,
• Network ports: UDP & TCP, syslog/syslog-ng, log4j/log4php, JMX/JMS, SNMP
• Databases: SQL/ODBC
• Splunk Servers: Access data locally on production hosts and forward it to another Splunk Server over SSL/TCP

The actual evaluation results will be the next article.

http://www.infoworld.com/article/08/04/01/14FE-april-fool-hp-windy_1.html

With Solaris 10 and VCS 5.x, nfsd HAS TO run under VCS control. In order to achieve that, the following needs ton happen (on every node that will host the NFS share) —

Disable/Delete the NFS services from SMF

# svccfg delete -f svc:/network/nfs/server:default# svccfg delete -f svc:/network/nfs/status:default# svccfg delete -f svc:/network/nfs/nlockmgr:default

Manually restart lockd, statd and automountd

# /usr/lib/nfs/lockd# /usr/lib/nfs/statd# /usr/lib/fs/autofs/automount# /usr/lib/autofs/automountd

NOTE: In this example (see below), the NFSgrp is configured only for one node. To add another node, add the node name and number to SystemList and AutoStartList

       group NFSgrp (               SystemList = { hostA = 0 }               AutoStartList = { hostA }        )        DiskGroup nfsDG (               Critical = 0               DiskGroup = testdg        )        Volume nfsVOL (              Critical = 0              Volume = testnfshome              DiskGroup = testdg        )        IP IPres (                Device = bge0               Address = "10.10.10.22"                NetMask = "255.255.255.0"        )        Mount Mountres (                MountPoint = "/nfs/testnfs"                BlockDevice = "/dev/vx/dsk/testdg/testnfshome"                FSType = vxfs                MountOpt = rw                FsckOpt = "-y"        )        NFS NFSres (                Nservers = 16        )        NFSLock NFSLockres (                PathName = "/nfs/testnfs"        )        NIC NICres (                Device = bge0        )        Share Shareres (                PathName = "/nfs/testnfs"                Options = "-o rw -d \"test home dirs\""        )        // IPres requires Shareres        IPres requires NICres        nfsVOL requires nfsDG        Mountres requires nfsVOL        NFSLockres requires Mountres        Shareres requires NFSLockres        Shareres requires NFSres        // resource dependency tree        //        // group NFSgrp        // {        // IP IPres        //      {        //      NIC NICres        //      Share Shareres        //          {        //          NFSLock NFSLockres        //              {        //              Mount Mountres        //                  {        //                  Volume nfsVOL        //                      {        //                      DG nfsDG        //                      }        //                  }        //              }        //          NFS NFSres        //          }        //       }        // }