Leveraging Centralized SSH2 based trusts to monitor network interface status on solaris servers
Since SSH2 key-based trusts have been established in this landscape (at root level), the automation of a variety of tasks becomes easily achievable. The SSH2 key-based trust ensures secure and encrypted transport mechanism (that reinforces security-oriented approach to system administration). Leveraging tools such as sudo (1m) or powerbroker an additional layer of security and auditability can be added.
Using TLRC and ndd_get.sh to collect Network-related information
The following two scripts can be used to make network interface related metrics collections.
tlrc.pl (Test Login Run Command) is a perl script that reads input from a colon-separated text file (of very specific format) or from the command-line and can execute any command on the remote host(s) specified with STDOUT/STDERR logging, etc.
tlrc.pl (test login run command) –
#!/usr/bin/env perl
use Getopt::Std;
use Net::Ping;
my %Args;
getopts( ‘l:i:c:o:n:adT:th’, \%Args );
if ( $Args{h} ) {
&printUsage && exit 0;
}
my $hlist = $Args{i} || “/path/to/inventory.txt”;
my $ssh = “/usr/bin/ssh”;
my $rsh = “/usr/bin/rsh”;
my $p = Net::Ping->new();
my $lid = $Args{l} || “nobody”;
my $outfile = $Args{o} || “tlrc.out”;
my @shlcmd = $Args{c};
my $conprot = $Args{T} or “ssh”;
if ( $conprot = “rsh|remsh|rlogin” ) {
$conprot = “rsh”;
}
else {
&printUsage && exit 1;
}
open( RHL, “< $hlist" ) or die "Unable to open input file $hlist: $! \n"; @rhl = ;
close(RHL);
open( WOF, “|tee $outfile” )
or die “Unable to open output file $outfile for writes: $! \n”;
open( WHL, “>> hlist.tlrc” );
if ( $Args{c} ) {
die “Can’t execute $Args{c} with the \”-t\” switch \n”
if ( ( $Args{t} or $Args{d} ) );
runCmd(@shlcmd);
}
if ( $Args{d} ) {
die “Can’t munge dmesg and run login tests at the same time! \n”
if $Args{t};
&dmesgMunger;
}
if ( $Args{t} ) {
&loginTest;
}
sub printUsage {
print
“Usage: $0 [ -l <> ][ -i
][ -c ][ -n ]|[ -a ]|[ -t ]|[ -h ] \n”;
print
“\t -l — pass the login name you want to use for this session \n
\t -i
— pass the input file (colon-delimited) with list of hosts an
d pingability status \n
\t -c — quoted Command you want to run remotely \n
\t -n — comma delimited list of hosts you want to run remote c
ommand specified with \”cmdstring\” on \
\t -a — specifies all hosts in input file to run remote command specified with
\”cmdstring\” on \
\t -T — specifies the Connection type — ssh or rsh \
\t -t — Optional switch to the -c or -h switches, it will only run testing port
ion of the script \
\t -h — print this message \n”;
}
sub runCmd {
my @cmdstring = @_;
if ( $Args{a} ) {
foreach $line (@rhl) {
next if ( $line =~ m/^#/ );
next if ( $line =~ m/^$/ );
my ( $name, $domain, $ip, $pstate, $canlogin, $contype, $serial,
$hid, $usage )
= split( ‘:’, $line );
chomp( $name, $domain, $ip, $pstate, $canlogin, $contype, $serial,
$hid, $usage );
if ( $pstate == 0 ) {
if ( $canlogin == 0 ) {
if ( $contype == 0 ) {
ssh_cmd( $lid, $name, @cmdstring );
}
elsif ( $contype == 1 ) {
rsh_cmd( $lid, $name, @cmdstring );
}
else {
print “Cannot understand connection type! \n”;
}
}
else {
print “Cannot log into the server! \n”;
}
}
else {
print “$name is unpingable — can’t reach! \n”;
}
}
}
elsif ( $Args{n} ) {
$hlist = $Args{n};
@hostlist = split( ‘ ‘, $hlist );
foreach $name (@hostlist) {
if ( $conprot = “ssh” ) {
ssh_cmd( $lid, $name, @cmdstring );
}
elsif ( $conprot = “rsh” ) {
rsh_cmd( $lid, $name, @cmdstring );
}
else {
die “Unknown Option with \”-T\” switch! \n”;
}
}
}
}
sub ssh_cmd {
my ( $id, $host, @cmd ) = @_;
print “$ssh $id\@$host ‘@cmd’ \n”;
@sshout = qx/$ssh $id\@$host ‘@cmd’/;
#or die “Can’t run cmd : $! \n”;
print WOF “$host \n”;
print WOF “@sshout \n”;
}
sub rsh_cmd {
my ( $id, $host, @cmd ) = @_;
print “$rsh -l $id $host ‘@cmd’ \n”;
@rshout = qx/$rsh -l $id $host ‘@cmd’ /;
#or die “can’t run $rsh -l $id $host ‘@cmd’ : $! \n”;
print WOF “$host \n”;
print WOF “@rshout \n”;
}
sub dmesgMunger {
&getToday;
&runCmd(
“cat /var/adm/messages|grep \”$today\”|egrep -v \”vas|auth\|lw8\|mail.info\|Wait
ing\|Networker savegroup\|local1|checked|wrap|Normal\”|egrep -i \”scsi|disk|err|
fatal|pers|mem|link|fcp|AFT|ASFR|PSYND|ESYND|full|vx_nospace|vxfs|vxvm\”"
);
}
sub getToday {
my ( $sec, $min, $hour, $mday, $mon, $year, $wday, $yday, $isdst ) =
localtime(time);
chomp( $sec, $min, $hour, $mday, $mon, $year, $wday, $yday, $isdst );
$year += 1900;
$mon += 1;
my %months = (
1 => ‘Jan’,
2 => ‘Feb’,
3 => ‘Mar’,
4 => ‘Apr’,
5 => ‘May’,
6 => ‘Jun’,
7 => ‘Jul’,
8 => ‘Aug’,
9 => ‘Sep’,
10 => ‘Oct’,
11 => ‘Nov’,
12 => ‘Dec’,
);
if ( $mday < mday = " $mday" today = "$months{$mon} $mday" line ="~" pstate ="=" npstate =" $p-">ping( $name, 1 );
if ( $npstate == 0 ) {
#$p->close();
print “Running \”$ssh $lid\@$name\”…\n”;
my @sshout =
system( “$ssh”, “-l”, “$lid”, “$name”, “\’exit\’” );
$exitval = $? >> 8;
chomp $exitval;
print WOF
“attempt to log into $name ended with status $exitval \n”;
print WHL “$name:$domain:$ip:$pstate:$exitval\n”;
}
else {
print WOF “Unable to ping $name \n”;
}
}
if ( $pstate == 1 ) {
print
“$hlist says $name is inaccessible.\nBut I will try to ping $name again anyway..
.\n”;
my $npstate = $p->ping( $name, 1 );
if ( $npstate == 0 ) {
#$p->close();
print “Running \”$ssh $lid\@$name\”…\n”;
my @sshout =
system( “$ssh”, “-l”, “$lid”, “$name”, “\’exit;\’” );
$exitval = $? >> 8;
chomp $exitval;
print WOF
“attempt to log into $name ended with status $exitval \n”;
print WHL “$name:$domain:$ip:$pstate:$exitval\n”;
}
else {
print WOF “Unable to ping $name \n”;
}
}
}
close(WOF);
}
inventory.txt (the input file passed to tlrc.pl) –
#HOSTNAME:DOMAIN NAME:IP:PINGABLE(1 == no; 0 == yes):Login(1 == no;0 == yes):Connection(0=ssh:1=telnet/rsh):SERIAL:HOSTID:USAGE(P|NP)
This is a colon-delimited file with fields as listed above. Not all of them are required for running the script, but can be useful in certain cases (eg: hostid, serial #).
ndd_get.sh is a Korn-shell based script that returns the NIC link-related statistics in a comma-separated output format.
#!/usr/bin/env ksh
NDD=/usr/sbin/ndd
ID=`/usr/xpg4/bin/id -u`
HOSTNAME=`/usr/bin/hostname`
printUsage() {
echo “Usage: $0 [ -a ]|[-n -i ]|[-h] \n”;
}
splitter() {
interface=$1
INSTANCE=`echo $interface|awk -F\e ‘{print $2}’`
BASEDEV=`echo $interface|awk -F\e ‘{print $1}’`
ADAPTER=”$BASEDEV”e
}
macipget() {
IF=$1
IFCONFIG=/usr/sbin/ifconfig
IP=`$IFCONFIG $IF|grep inet|awk ‘{print $2}’`
MAC=`$IFCONFIG $IF|grep ether|awk ‘{print $2}’`
}
nddget() {
#set -x
AD=$1
INST=$2
$NDD -set /dev/$AD instance $INST
LSTAT=`$NDD -get /dev/$AD link_status`
LSPEED=`$NDD -get /dev/$AD link_speed`
LMODE=`$NDD -get /dev/$AD link_mode`
IS_100FDX=`$NDD -get /dev/$AD adv_100fdx_cap`
IS_100HDX=`$NDD -get /dev/$AD adv_100hdx_cap`
IS_10FDX=`$NDD -get /dev/$AD adv_10fdx_cap`
IS_10HDX=`$NDD -get /dev/$AD adv_10hdx_cap`
AUTONEG=`$NDD -get /dev/$AD adv_autoneg_cap`
LP_100FDX=`$NDD -get /dev/$AD lp_100fdx_cap`
LP_100FDX=`$NDD -get /dev/$AD lp_100hdx_cap`
LP_10FDX=`$NDD -get /dev/$AD lp_10fdx_cap`
LP_10HDX=`$NDD -get /dev/$AD lp_10hdx_cap`
LP_AUTONEG=`$NDD -get /dev/$AD lp_autoneg_cap`
if [ $LSTAT -eq 0 ]; then
linkstat=”down”
else
linkstat=”up”
fi
if [ $LSPEED -eq 0 ]; then
linkspeed=”10″
else
linkspeed=”100″
fi
if [ $LMODE -eq 0 ]; then
linkmode=”Half Duplex”
else
linkmode=”Full Duplex”
fi
if [ $AUTONEG -eq 0 ]; then
autoneg=”Off”
else
autoneg=”on”
fi
if [ $LP_AUTONEG -eq 0 ]; then
lp_autoneg=”Off”
else
lp_autoneg=”On”
fi
IF=$AD$INST
macipget $IF
print “$HOSTNAME,$IF,$IP,$MAC,$linkstat,$linkspeed,$linkmode,$autoneg,$lp_au
toneg”
}
kstatget() {
#set -x
AD=$1
INST=$2
linkspeed=`/usr/bin/kstat -p $AD|grep -i link_|\
grep “$AD:$INST”|grep link_speed|awk ‘{print $2}’`
is_up=`/usr/bin/kstat -p $AD|grep -i link_|\
grep “$AD:$INST”|grep link_up| awk ‘{print $2}’`
if [ $is_up -eq 1 ]; then
linkstat=”UP”
else
linkstat=”DOWN”
fi
LINK_MODE=`/usr/bin/kstat -p $AD|grep -i link_|\
grep $AD:$INST|grep link_duplex|awk ‘{print $2}’`
case $LINK_MODE in
2) linkmode=”Full Duplex”;;
1) linkmode=”Half Duplex”;;
*) linkmode=”Unknown”;;
esac
$NDD -set /dev/$AD instance $INST
AUTONEG=`$NDD -get /dev/$AD adv_autoneg_cap`
LP_AUTONEG=`/usr/bin/kstat -p $AD|\
grep $AD:$INST|grep lp_cap_autoneg|awk ‘{print $2}’`
if [ $AUTONEG -eq 0 ]; then
autoneg=”Off”
else
autoneg=”On”
fi
if [ $LP_AUTONEG -eq 0 ]; then
lp_autoneg=”Off”
else
lp_autoneg=”On”
fi
IF=$AD$INST
macipget $IF
print “$HOSTNAME,$IF,$IP,$MAC,$linkstat,$linkspeed,$linkmode,$autoneg,$lp_au
toneg”
}
bgekstatget() {
#set -x
AD=$1
INST=$2
linkspeed=`/usr/bin/kstat -m $AD -i $INST -n parameters|\
grep -i link_| grep link_speed|awk ‘{print $2}’`
is_up=`/usr/bin/kstat -m $AD -i $INST -n parameters|\
grep -i link_|grep link_status| awk ‘{print $2}’`
if [ $is_up -eq 1 ]; then
linkstat=”UP”
else
linkstat=”DOWN”
fi
LINK_MODE=`/usr/bin/kstat -m $AD -i $INST -n parameters|\
grep -i link_|grep link_duplex|awk ‘{print $2}’`
case $LINK_MODE in
2) linkmode=”Full Duplex”;;
1) linkmode=”Half Duplex”;;
*) linkmode=”Unknown”;;
esac
AUTONEG=`/usr/bin/kstat -m $AD -i $INST -n parameters|\
grep -i link_|grep autoneg|awk ‘{print $2}’`
LP_AUTONEG=`/usr/bin/kstat -m $AD -i $INST -n parameters|\
grep lp_| grep autoneg |awk ‘{print $2}’`
if [ $AUTONEG -eq 0 ]; then
autoneg=”Off”
else
autoneg=”On”
fi
if [ $LP_AUTONEG -eq 0 ]; then
lp_autoneg=”Off”
else
lp_autoneg=”On”
fi
IF=$AD$INST
macipget $IF
print “$HOSTNAME,$IF,$IP,$MAC,$linkstat,$linkspeed,$linkmode,$autoneg,$lp_au
toneg”
}
dmfeget() {
AD=$1
INST=$2
EADAPT=$AD$INST
#$NDD -set /dev/$EADAPT
# NOte the ndd set is not required since dmfe interfaces are directly
# set up as device files (such as /dev/dmfe0, /dev/dmfe1)
LSTAT=`$NDD -get /dev/$EADAPT link_status`
LSPEED=`$NDD -get /dev/$EADAPT link_speed`
LMODE=`$NDD -get /dev/$EADAPT link_mode`
IS_100FDX=`$NDD -get /dev/$EADAPT adv_100fdx_cap`
IS_100HDX=`$NDD -get /dev/$EADAPT adv_100hdx_cap`
IS_10FDX=`$NDD -get /dev/$EADAPT adv_10fdx_cap`
IS_10HDX=`$NDD -get /dev/$EADAPT adv_10hdx_cap`
AUTONEG=`$NDD -get /dev/$EADAPT adv_autoneg_cap`
LP_AUTONEG=`$NDD -get /dev/$ADAPT lp_autoneg_cap`
if [ $LSTAT -eq 0 ]; then
linkstat=”down”
else
linkstat=”up”
fi
if [ $LSPEED -eq 0 ]; then
linkspeed=”10″
else
linkspeed=”100″
fi
if [ $LMODE -eq 0 ]; then
linkmode=”Half Duplex”
else
linkmode=”Full Duplex”
fi
if [ $AUTONEG -eq 0 ]; then
autoneg=”Off”
else
autoneg=”on”
fi
if [ $LP_AUTONEG -eq 0 ]; then
lp_autoneg=”Off”
else
lp_autoneg=”On”
fi
macipget $EADAPT
print “$HOSTNAME,$EADAPT,$IP,$MAC,$linkstat,$linkspeed,$linkmode,$autoneg,$l
p_autoneg”
}
getParms() {
#set -x
case $ADAPTER in
qfe) nddget $ADAPTER $INSTANCE;;
hme) nddget $ADAPTER $INSTANCE;;
eri) nddget $ADAPTER $INSTANCE;;
ce) kstatget $ADAPTER $INSTANCE;;
bge) bgekstatget $ADAPTER $INSTANCE;;
dmfe) dmfeget $ADAPTER $INSTANCE;;
*) echo “Error: Unknown adapter! \n” &&amp; exit 1;;
esac
}
nicStatAll() {
#set -x
/usr/sbin/ifconfig -a|nawk ‘/UP/{print $1}’|egrep -v “lo0|clprivnet”| \
awk -F: ‘{print $1}’ |sort -nr|uniq > /tmp/iflist;
for interface in `cat /tmp/iflist`
do
if [ $interface = ":*" ]; then
next
fi
# Deprecated code — left behind for old time’s sake
#count=`echo $interface|wc -m|sed -e”s!^[ /t]!!g”`
#count1=`expr $count - 2`
#count2=`expr $count - 1`
#int=`echo $interface|cut -c 1-${count1}`
#dev=/dev/${int}
#inst=`echo $interface|cut -c ${count2}`
case $interface in
eri*) INSTANCE=`echo $interface|awk -F\i ‘{print $2}’`
BASEDEV=`echo $interface|awk -F\i ‘{print $1}’`
ADAPTER=”$BASEDEV”i;;
*) splitter $interface;;
esac
getParms
done
}
if [ $ID -ne 0 ]; then
echo “ERROR: You are not root! Only root can run this script!\n”;
exit 1;
fi
while getopts an:i:h arg
do
case $arg in
a) nicStatAll &&amp; exit 0;;
n) ADAPTER=${OPTARG};;
i) INSTANCE=${OPTARG};;
h) printUsage &&amp; exit 0;;
*) printUsage &&amp; exit 1;;
esac
done
shift $(($OPTIND - 1))
if [ ! -z ${ADAPTER} ]; then
if [ ! -z ${INSTANCE} ]; then
getParms
else
printUsage && exit 1
fi
else
printUsage && exit 1
fi
On the centralized management host (whose SSH2-based Key is trusted by the monitored hosts) run the following command to perform the inventory:
admin:(dev) $ sudo ./tlrc.pl -l root -a \-c "/path/to/nddget.sh -a" \-o ~/logs/ndd_get_today.txt/usr/bin/ssh root@host1 '/path/to/ndd_get.sh -a'/usr/bin/ssh root@host2 '/path/to/ndd_get.sh -a'host1host1,bge2,IP,MAC,UP,100,Full Duplex,On,Onhost1,bge1,IP,MAC,UP,100,Full Duplex,On,Onhost1,bge0,IP,MAC,UP,100,Full Duplex,On,Onhost2host2,bge2,10.228.147.62,0:3:ba:49:45:51,UP,100,Full Duplex,On,Onhost2,bge1,10.228.143.62,0:3:ba:49:45:50,UP,100,Full Duplex,On,Onhost2,bge0,10.228.139.62,0:3:ba:49:45:4f,UP,100,Full Duplex,On,On/usr/bin/rsh -l root host3 '/path/to/ndd_get.sh -a'/usr/bin/rsh -l root host4 '/path/to/ndd_get.sh -a'
host3host3,qfe1,IP,MAC,up,100,Full Duplex,Off,Offhost3,qfe0,IP,MAC,up,100,Full Duplex,Off,Offhost3,ce0,IP,MAC,UP,1000,Full Duplex,On,On
Look at the text output created thus:
admin:(logs) $ more ndd_get_today.txt
host1host1,bge2,IP,MAC,UP,100,Full Duplex,On,Onhost1,bge1,IP,MAC,UP,100,Full Duplex,On,Onhost1,bge0,IP,MAC,UP,100,Full Duplex,On,Onhost2host2,bge2,IP,MAC,UP,100,Full Duplex,On,Onhost2,bge1,IP,MAC,UP,100,Full Duplex,On,Onhost2,bge0,IP,MAC,UP,100,Full Duplex,On,On
Now look at the sudo log file to see if there’s associated logging captured.
admin:(log) $ sudo tail sudo.logSep 5 16:32:36 : lahirdx : TTY=pts/27 ; PWD=/export/home/lahirdx/dev ; USER=root ; COMMAND=/usr/bin/ssh aesdbc1Sep 6 09:49:31 : lahirdx : TTY=pts/30 ; PWD=/export/home/lahirdx/dev ; USER=root ; COMMAND=./tlrc.pl -a -c /export/patches/Scripts/bin/ndd_get.sh -a -o /export/home/lahirdx/logs/ndd_get_9606.txtSep 6 09:49:40 : lahirdx : TTY=pts/30 ; PWD=/export/home/lahirdx/dev ; USER=root ; COMMAND=./tlrc.pl -l root -a -c /export/patches/Scripts/bin/ndd_get.sh -a -o /export/home/lahirdx/logs/ndd_get_9606.txt
NOTE: Look at the full command line, who executed a particular command, when etc getting captured in the logs. Also, it is imperative to ensure that the “/path/to/ndd_get.sh” is the same on all the monitored hosts. This author recommends creating a system V package to deploy commonly used scripts and tools under /opt/tools (or similar directory structure) to ensure standardization of the environment.
![/!\](http://ilscha01-ss-01.uscc.com/wiki/modern/img/alert.png "/!\"){kind=small}
Remember to delete the service maps for NFS using svccfg command
Sun Workstation 9) Televideo 910 10) Televideo 925 11) Wyse Model 50 12) X Terminal Emulator (xterms) 13) OtherType the number of your choice and press Return: 12Configuring network interface addresses: bge1.RPC: Timed out![/!\](http://dwailsun/wiki/modern/img/alert.png "/!\"){kind=small}
Think of a optimal battery of tests that can help us determine whether this virtualized solaris 8 is a viable platform for servers that cannot be migrated…. 
Set up sshd after adding these packages, complete with start up scripts, sshd privsep user id in the system accounts files (passwd and shadow).