Mar 092007
 

Leveraging Centralized SSH2 based trusts to monitor network interface status on solaris servers

Since SSH2 key-based trusts have been established in this landscape (at root level), the automation of a variety of tasks becomes easily achievable. The SSH2 key-based trust ensures secure and encrypted transport mechanism (that reinforces security-oriented approach to system administration). Leveraging tools such as sudo (1m) or powerbroker an additional layer of security and auditability can be added.

Using TLRC and ndd_get.sh to collect Network-related information

The following two scripts can be used to make network interface related metrics collections.

tlrc.pl (Test Login Run Command) is a perl script that reads input from a colon-separated text file (of very specific format) or from the command-line and can execute any command on the remote host(s) specified with STDOUT/STDERR logging, etc.

tlrc.pl (test login run command) --

#!/usr/bin/env perl

use Getopt::Std;
use Net::Ping;

my %Args;

getopts( ‘l:i:c:o:n:adT:th’, \%Args );

if ( $Args{h} ) {
&printUsage && exit 0;
}

my $hlist = $Args{i} || “/path/to/inventory.txt”;
my $ssh = “/usr/bin/ssh”;

my $rsh = “/usr/bin/rsh”;
my $p = Net::Ping->new();
my $lid = $Args{l} || “nobody”;
my $outfile = $Args{o} || “tlrc.out”;
my @shlcmd = $Args{c};
my $conprot = $Args{T} or “ssh”;

if ( $conprot = “rsh|remsh|rlogin” ) {
$conprot = “rsh”;
}
else {
&printUsage && exit 1;
}

open( RHL, “< $hlist" ) or die "Unable to open input file $hlist: $! \n"; @rhl = ;
close(RHL);
open( WOF, “|tee $outfile” )
or die “Unable to open output file $outfile for writes: $! \n”;
open( WHL, “>> hlist.tlrc” );

if ( $Args{c} ) {
die “Can’t execute $Args{c} with the \”-t\” switch \n”
if ( ( $Args{t} or $Args{d} ) );
runCmd(@shlcmd);
}

if ( $Args{d} ) {
die “Can’t munge dmesg and run login tests at the same time! \n”
if $Args{t};
&dmesgMunger;
}

if ( $Args{t} ) {
&loginTest;
}

sub printUsage {
print
“Usage: $0 [ -l <> ][ -i ][ -c ][ -n ]|[ -a ]|[ -t ]|[ -h ] \n”;
print
“\t -l — pass the login name you want to use for this session \n
\t -i — pass the input file (colon-delimited) with list of hosts an
d pingability status \n
\t -c — quoted Command you want to run remotely \n
\t -n — comma delimited list of hosts you want to run remote c
ommand specified with \”cmdstring\” on \
\t -a — specifies all hosts in input file to run remote command specified with
\”cmdstring\” on \
\t -T — specifies the Connection type — ssh or rsh \
\t -t — Optional switch to the -c or -h switches, it will only run testing port
ion of the script \
\t -h — print this message \n”;
}

sub runCmd {

my @cmdstring = @_;

if ( $Args{a} ) {
foreach $line (@rhl) {
next if ( $line =~ m/^#/ );
next if ( $line =~ m/^$/ );
my ( $name, $domain, $ip, $pstate, $canlogin, $contype, $serial,
$hid, $usage )
= split( ‘:’, $line );
chomp( $name, $domain, $ip, $pstate, $canlogin, $contype, $serial,
$hid, $usage );
if ( $pstate == 0 ) {
if ( $canlogin == 0 ) {
if ( $contype == 0 ) {
ssh_cmd( $lid, $name, @cmdstring );
}
elsif ( $contype == 1 ) {
rsh_cmd( $lid, $name, @cmdstring );
}
else {
print “Cannot understand connection type! \n”;
}
}
else {
print “Cannot log into the server! \n”;
}
}
else {
print “$name is unpingable — can’t reach! \n”;
}
}
}
elsif ( $Args{n} ) {
$hlist = $Args{n};
@hostlist = split( ‘ ‘, $hlist );
foreach $name (@hostlist) {
if ( $conprot = “ssh” ) {
ssh_cmd( $lid, $name, @cmdstring );
}
elsif ( $conprot = “rsh” ) {
rsh_cmd( $lid, $name, @cmdstring );
}
else {
die “Unknown Option with \”-T\” switch! \n”;
}
}
}
}

sub ssh_cmd {

my ( $id, $host, @cmd ) = @_;
print “$ssh $id\@$host ‘@cmd’ \n”;
@sshout = qx/$ssh $id\@$host ‘@cmd’/;

#or die “Can’t run cmd : $! \n”;
print WOF “$host \n”;
print WOF “@sshout \n”;
}

sub rsh_cmd {

my ( $id, $host, @cmd ) = @_;
print “$rsh -l $id $host ‘@cmd’ \n”;

@rshout = qx/$rsh -l $id $host ‘@cmd’ /;

#or die “can’t run $rsh -l $id $host ‘@cmd’ : $! \n”;

print WOF “$host \n”;
print WOF “@rshout \n”;
}

sub dmesgMunger {

&getToday;
&runCmd(
“cat /var/adm/messages|grep \”$today\”|egrep -v \”vas|auth\|lw8\|mail.info\|Wait
ing\|Networker savegroup\|local1|checked|wrap|Normal\”|egrep -i \”scsi|disk|err|
fatal|pers|mem|link|fcp|AFT|ASFR|PSYND|ESYND|full|vx_nospace|vxfs|vxvm\””
);
}

sub getToday {
my ( $sec, $min, $hour, $mday, $mon, $year, $wday, $yday, $isdst ) =
localtime(time);
chomp( $sec, $min, $hour, $mday, $mon, $year, $wday, $yday, $isdst );
$year += 1900;
$mon += 1;

my %months = (
1 => ‘Jan’,
2 => ‘Feb’,
3 => ‘Mar’,
4 => ‘Apr’,
5 => ‘May’,
6 => ‘Jun’,
7 => ‘Jul’,
8 => ‘Aug’,
9 => ‘Sep’,
10 => ‘Oct’,
11 => ‘Nov’,
12 => ‘Dec’,
);

if ( $mday < mday = " $mday" today = "$months{$mon} $mday" line ="~" pstate ="=" npstate =" $p-">ping( $name, 1 );
if ( $npstate == 0 ) {

#$p->close();
print “Running \”$ssh $lid\@$name\”…\n”;
my @sshout =
system( “$ssh”, “-l”, “$lid”, “$name”, “\’exit\'” );
$exitval = $? >> 8;
chomp $exitval;
print WOF
“attempt to log into $name ended with status $exitval \n”;
print WHL “$name:$domain:$ip:$pstate:$exitval\n”;
}
else {
print WOF “Unable to ping $name \n”;
}
}
if ( $pstate == 1 ) {
print
“$hlist says $name is inaccessible.\nBut I will try to ping $name again anyway..
.\n”;
my $npstate = $p->ping( $name, 1 );
if ( $npstate == 0 ) {

#$p->close();
print “Running \”$ssh $lid\@$name\”…\n”;
my @sshout =
system( “$ssh”, “-l”, “$lid”, “$name”, “\’exit;\'” );
$exitval = $? >> 8;
chomp $exitval;
print WOF
“attempt to log into $name ended with status $exitval \n”;
print WHL “$name:$domain:$ip:$pstate:$exitval\n”;
}
else {
print WOF “Unable to ping $name \n”;
}
}
}
close(WOF);
}

inventory.txt (the input file passed to tlrc.pl) --

#HOSTNAME:DOMAIN NAME:IP:PINGABLE(1 == no; 0 == yes):Login(1 == no;0 == yes):Connection(0=ssh:1=telnet/rsh):SERIAL:HOSTID:USAGE(P|NP)


This is a colon-delimited file with fields as listed above. Not all of them are required for running the script, but can be useful in certain cases (eg: hostid, serial #).

ndd_get.sh is a Korn-shell based script that returns the NIC link-related statistics in a comma-separated output format.

#!/usr/bin/env ksh

NDD=/usr/sbin/ndd
ID=`/usr/xpg4/bin/id -u`
HOSTNAME=`/usr/bin/hostname`

printUsage() {
echo “Usage: $0 [ -a ]|[-n -i ]|[-h] \n”;
}

splitter() {
interface=$1
INSTANCE=`echo $interface|awk -F\e ‘{print $2}’`
BASEDEV=`echo $interface|awk -F\e ‘{print $1}’`
ADAPTER=”$BASEDEV”e
}

macipget() {
IF=$1
IFCONFIG=/usr/sbin/ifconfig
IP=`$IFCONFIG $IF|grep inet|awk ‘{print $2}’`
MAC=`$IFCONFIG $IF|grep ether|awk ‘{print $2}’`
}

nddget() {
#set -x
AD=$1
INST=$2
$NDD -set /dev/$AD instance $INST
LSTAT=`$NDD -get /dev/$AD link_status`
LSPEED=`$NDD -get /dev/$AD link_speed`
LMODE=`$NDD -get /dev/$AD link_mode`
IS_100FDX=`$NDD -get /dev/$AD adv_100fdx_cap`
IS_100HDX=`$NDD -get /dev/$AD adv_100hdx_cap`
IS_10FDX=`$NDD -get /dev/$AD adv_10fdx_cap`
IS_10HDX=`$NDD -get /dev/$AD adv_10hdx_cap`
AUTONEG=`$NDD -get /dev/$AD adv_autoneg_cap`
LP_100FDX=`$NDD -get /dev/$AD lp_100fdx_cap`
LP_100FDX=`$NDD -get /dev/$AD lp_100hdx_cap`
LP_10FDX=`$NDD -get /dev/$AD lp_10fdx_cap`
LP_10HDX=`$NDD -get /dev/$AD lp_10hdx_cap`
LP_AUTONEG=`$NDD -get /dev/$AD lp_autoneg_cap`
if [ $LSTAT -eq 0 ]; then
linkstat=”down”
else
linkstat=”up”
fi
if [ $LSPEED -eq 0 ]; then
linkspeed=”10″
else
linkspeed=”100″
fi
if [ $LMODE -eq 0 ]; then
linkmode=”Half Duplex”
else
linkmode=”Full Duplex”
fi
if [ $AUTONEG -eq 0 ]; then
autoneg=”Off”
else
autoneg=”on”
fi
if [ $LP_AUTONEG -eq 0 ]; then
lp_autoneg=”Off”
else
lp_autoneg=”On”
fi
IF=$AD$INST
macipget $IF
print “$HOSTNAME,$IF,$IP,$MAC,$linkstat,$linkspeed,$linkmode,$autoneg,$lp_au
toneg”
}

kstatget() {
#set -x
AD=$1
INST=$2

linkspeed=`/usr/bin/kstat -p $AD|grep -i link_|\
grep “$AD:$INST”|grep link_speed|awk ‘{print $2}’`

is_up=`/usr/bin/kstat -p $AD|grep -i link_|\
grep “$AD:$INST”|grep link_up| awk ‘{print $2}’`
if [ $is_up -eq 1 ]; then
linkstat=”UP”
else
linkstat=”DOWN”
fi
LINK_MODE=`/usr/bin/kstat -p $AD|grep -i link_|\
grep $AD:$INST|grep link_duplex|awk ‘{print $2}’`
case $LINK_MODE in
2) linkmode=”Full Duplex”;;
1) linkmode=”Half Duplex”;;
*) linkmode=”Unknown”;;
esac

$NDD -set /dev/$AD instance $INST
AUTONEG=`$NDD -get /dev/$AD adv_autoneg_cap`
LP_AUTONEG=`/usr/bin/kstat -p $AD|\
grep $AD:$INST|grep lp_cap_autoneg|awk ‘{print $2}’`
if [ $AUTONEG -eq 0 ]; then
autoneg=”Off”
else
autoneg=”On”
fi
if [ $LP_AUTONEG -eq 0 ]; then
lp_autoneg=”Off”
else
lp_autoneg=”On”
fi
IF=$AD$INST
macipget $IF
print “$HOSTNAME,$IF,$IP,$MAC,$linkstat,$linkspeed,$linkmode,$autoneg,$lp_au
toneg”

}

bgekstatget() {
#set -x
AD=$1
INST=$2

linkspeed=`/usr/bin/kstat -m $AD -i $INST -n parameters|\
grep -i link_| grep link_speed|awk ‘{print $2}’`

is_up=`/usr/bin/kstat -m $AD -i $INST -n parameters|\
grep -i link_|grep link_status| awk ‘{print $2}’`
if [ $is_up -eq 1 ]; then
linkstat=”UP”
else
linkstat=”DOWN”
fi
LINK_MODE=`/usr/bin/kstat -m $AD -i $INST -n parameters|\
grep -i link_|grep link_duplex|awk ‘{print $2}’`
case $LINK_MODE in
2) linkmode=”Full Duplex”;;
1) linkmode=”Half Duplex”;;
*) linkmode=”Unknown”;;
esac

AUTONEG=`/usr/bin/kstat -m $AD -i $INST -n parameters|\
grep -i link_|grep autoneg|awk ‘{print $2}’`
LP_AUTONEG=`/usr/bin/kstat -m $AD -i $INST -n parameters|\
grep lp_| grep autoneg |awk ‘{print $2}’`
if [ $AUTONEG -eq 0 ]; then
autoneg=”Off”
else
autoneg=”On”
fi
if [ $LP_AUTONEG -eq 0 ]; then
lp_autoneg=”Off”
else
lp_autoneg=”On”
fi

IF=$AD$INST
macipget $IF
print “$HOSTNAME,$IF,$IP,$MAC,$linkstat,$linkspeed,$linkmode,$autoneg,$lp_au
toneg”

}

dmfeget() {

AD=$1
INST=$2
EADAPT=$AD$INST
#$NDD -set /dev/$EADAPT
# NOte the ndd set is not required since dmfe interfaces are directly
# set up as device files (such as /dev/dmfe0, /dev/dmfe1)

LSTAT=`$NDD -get /dev/$EADAPT link_status`
LSPEED=`$NDD -get /dev/$EADAPT link_speed`
LMODE=`$NDD -get /dev/$EADAPT link_mode`
IS_100FDX=`$NDD -get /dev/$EADAPT adv_100fdx_cap`
IS_100HDX=`$NDD -get /dev/$EADAPT adv_100hdx_cap`
IS_10FDX=`$NDD -get /dev/$EADAPT adv_10fdx_cap`
IS_10HDX=`$NDD -get /dev/$EADAPT adv_10hdx_cap`
AUTONEG=`$NDD -get /dev/$EADAPT adv_autoneg_cap`
LP_AUTONEG=`$NDD -get /dev/$ADAPT lp_autoneg_cap`
if [ $LSTAT -eq 0 ]; then
linkstat=”down”
else
linkstat=”up”
fi
if [ $LSPEED -eq 0 ]; then
linkspeed=”10″
else
linkspeed=”100″
fi
if [ $LMODE -eq 0 ]; then
linkmode=”Half Duplex”
else
linkmode=”Full Duplex”
fi
if [ $AUTONEG -eq 0 ]; then
autoneg=”Off”
else
autoneg=”on”
fi
if [ $LP_AUTONEG -eq 0 ]; then
lp_autoneg=”Off”
else
lp_autoneg=”On”
fi
macipget $EADAPT

print “$HOSTNAME,$EADAPT,$IP,$MAC,$linkstat,$linkspeed,$linkmode,$autoneg,$l
p_autoneg”

}

getParms() {
#set -x
case $ADAPTER in
qfe) nddget $ADAPTER $INSTANCE;;
hme) nddget $ADAPTER $INSTANCE;;
eri) nddget $ADAPTER $INSTANCE;;
ce) kstatget $ADAPTER $INSTANCE;;
bge) bgekstatget $ADAPTER $INSTANCE;;
dmfe) dmfeget $ADAPTER $INSTANCE;;
*) echo “Error: Unknown adapter! \n” &&amp;amp; exit 1;;
esac
}

nicStatAll() {
#set -x
/usr/sbin/ifconfig -a|nawk ‘/UP/{print $1}’|egrep -v “lo0|clprivnet”| \
awk -F: ‘{print $1}’ |sort -nr|uniq > /tmp/iflist;
for interface in `cat /tmp/iflist`
do
if [ $interface = “:*” ]; then
next
fi
# Deprecated code — left behind for old time’s sake
#count=`echo $interface|wc -m|sed -e”s!^[ /t]!!g”`
#count1=`expr $count – 2`
#count2=`expr $count – 1`
#int=`echo $interface|cut -c 1-${count1}`
#dev=/dev/${int}
#inst=`echo $interface|cut -c ${count2}`
case $interface in
eri*) INSTANCE=`echo $interface|awk -F\i ‘{print $2}’`
BASEDEV=`echo $interface|awk -F\i ‘{print $1}’`
ADAPTER=”$BASEDEV”i;;
*) splitter $interface;;
esac
getParms
done
}

if [ $ID -ne 0 ]; then
echo “ERROR: You are not root! Only root can run this script!\n”;
exit 1;
fi

while getopts an:i:h arg
do
case $arg in
a) nicStatAll &&amp;amp; exit 0;;
n) ADAPTER=${OPTARG};;
i) INSTANCE=${OPTARG};;
h) printUsage &&amp;amp; exit 0;;
*) printUsage &&amp;amp; exit 1;;
esac
done
shift $(($OPTIND – 1))

if [ ! -z ${ADAPTER} ]; then
if [ ! -z ${INSTANCE} ]; then
getParms
else
printUsage && exit 1
fi
else
printUsage && exit 1
fi

On the centralized management host (whose SSH2-based Key is trusted by the monitored hosts) run the following command to perform the inventory:

admin:(dev) $ sudo ./tlrc.pl -l root -a \<br />-c "/path/to/nddget.sh -a" \<br />-o ~/logs/ndd_get_today.txt<br /><span class="anchor" id="line-45"></span><br />/usr/bin/ssh root@host1 '/path/to/ndd_get.sh -a'<br />/usr/bin/ssh root@host2 '/path/to/ndd_get.sh -a'<br />host1<br />host1,bge2,IP,MAC,UP,100,Full Duplex,On,On<br />host1,bge1,IP,MAC,UP,100,Full Duplex,On,On<br />host1,bge0,IP,MAC,UP,100,Full Duplex,On,On<br /><span class="anchor" id="line-54"></span><br />host2<br />host2,bge2,10.228.147.62,0:3:ba:49:45:51,UP,100,Full Duplex,On,On<br />host2,bge1,10.228.143.62,0:3:ba:49:45:50,UP,100,Full Duplex,On,On<br />host2,bge0,10.228.139.62,0:3:ba:49:45:4f,UP,100,Full Duplex,On,On<br /><span class="anchor" id="line-60"></span><br />/usr/bin/rsh -l root host3 '/path/to/ndd_get.sh -a'<br />/usr/bin/rsh -l root host4 '/path/to/ndd_get.sh -a'<br /><br />host3<br />host3,qfe1,IP,MAC,up,100,Full Duplex,Off,Off<br />host3,qfe0,IP,MAC,up,100,Full Duplex,Off,Off<br />host3,ce0,IP,MAC,UP,1000,Full Duplex,On,On<br /><span class="anchor" id="line-78"></span><br /><truncated><br /><span class="anchor" id="line-92"></span></truncated>

Look at the text output created thus:

admin:(logs) $ more ndd_get_today.txt<br /><br />host1<br />host1,bge2,IP,MAC,UP,100,Full Duplex,On,On<br />host1,bge1,IP,MAC,UP,100,Full Duplex,On,On<br />host1,bge0,IP,MAC,UP,100,Full Duplex,On,On<br /><span class="anchor" id="line-105"></span><br />host2<br />host2,bge2,IP,MAC,UP,100,Full Duplex,On,On<br />host2,bge1,IP,MAC,UP,100,Full Duplex,On,On<br />host2,bge0,IP,MAC,UP,100,Full Duplex,On,On<br /><span class="anchor" id="line-110"></span><br /><truncated><br /><span class="anchor" id="line-123"></span></truncated>

Now look at the sudo log file to see if there’s associated logging captured.

admin:(log) $ sudo tail sudo.log<br /><span class="anchor" id="line-129"></span>Sep  5 16:32:36 : lahirdx : TTY=pts/27 ; PWD=/export/home/lahirdx/dev ;<br /><span class="anchor" id="line-130"></span>    USER=root ; COMMAND=/usr/bin/ssh aesdbc1<br /><span class="anchor" id="line-131"></span>Sep  6 09:49:31 : lahirdx : TTY=pts/30 ; PWD=/export/home/lahirdx/dev ;<br /><span class="anchor" id="line-132"></span>    USER=root ; COMMAND=./tlrc.pl -a -c /export/patches/Scripts/bin/ndd_get.sh<br /><span class="anchor" id="line-133"></span>    -a -o /export/home/lahirdx/logs/ndd_get_9606.txt<br /><span class="anchor" id="line-134"></span>Sep  6 09:49:40 : lahirdx : TTY=pts/30 ; PWD=/export/home/lahirdx/dev ;<br /><span class="anchor" id="line-135"></span>    USER=root ; COMMAND=./tlrc.pl -l root -a -c<br /><span class="anchor" id="line-136"></span>    /export/patches/Scripts/bin/ndd_get.sh -a -o /export/home/lahirdx/logs/ndd_get_9606.txt<br /><br /><span class="anchor" id="line-139"></span>

NOTE: Look at the full command line, who executed a particular command, when etc getting captured in the logs. Also, it is imperative to ensure that the “/path/to/ndd_get.sh” is the same on all the monitored hosts. This author recommends creating a system V package to deploy commonly used scripts and tools under /opt/tools (or similar directory structure) to ensure standardization of the environment.

 Posted by at 6:49 pm