Mar 182014
 

This is an old one. I thought I’d post this before it falls off main memory.

You will need two things for this. I’ll start with the obvious.

  1. A splunk server configured to act as a deployment server (which will result in apps, configurations, etc being deployed centrally)
  2. A centralized “bastion” host, that has root-level ssh-based trust established with your target hosts (ie splunk agents). Another option is to have unprivileged ssh key-based trusts set up, but with local sudo privileges (passwordless) on each of your target hosts.

Once you do that, you can run a modification of this script shown below —

I’d written in for an old shop of mine, where we managed Solaris on SPARC, Solaris on x86 and Redhat Linux. Modify/write functions corresponding to your target OS of choice and it should do pretty much what you need rather easily.

Another thing to watch out for is make sure the version of splunk you are working with hasn’t changed the underlying mechanisms. I had written this for Splunk v4.x at least 3 years back (can’t remember how long ago now – this is one script I didn’t put in version control, so no way for me to tell how old it…and timestamps changed because I copied them from my old laptop, etc).

 

Mar 122014
 

Okay, this has been a long time coming, but thought I’d write this down before I forget all about it.

First a little bit of a rant. In the EMC world, we assign LUNs to hosts and identify them using a hex-id code called the LUN ID. This, in conjunction with the EMC Frame ID, helps us uniquely and “easily” identify the LUNs, especially when you consider that there are usually multiple EMC frames and hundreds or thousands of SAN-attached hosts in most modern enterprise data centers.

Okay, the challenge is when using a disk multipathing solution other than EMC’s expensive power path software, or the exorbitantly expensive Veritas Storage Foundation suite from Symantec.

Most self-respecting modern operating systems have disk multipathing software built in and freely available.  and the focus item for this blog is Solaris. Solaris has a mature and efficient multipathing software called MPxIO, which cleverly creates a pseudo-device corresponding to the complex of the multiple (or single) paths via which a SAN LUN is visible to the Operating system.

The MPxIO driver, then using the LUN’s global unique identifier (GUID) to, perhaps can be argued, rightly identify the LUN uniquely. This is however a challenge since the GUID is a 32-character string (very nicely explained here http://www.slideshare.net/JamesCMcPherson/what-is-a-guid).

When I first proposed to our usual user community – Oracle DBAs, that they would have to use disks named as shown below, I faced outraged howls of indignation. “How can we address these disks in ASM, etc?”

c3t60060160294122002A2ADDB296EADE11d0

To work around, we considered creating an alternate namespace, say /dev/oracle/disk001 and so forth, retaining the major/minor numbers of the underlying devices.  But that would get hairy real quick, especially on those databases where we had multiple terabytes of storage (hundreds of LUNs).

If you are working with CLARiiON arrays, then you will have figure out some other way (than what I’ve shown in this blog post) to map your MPxIO/Solaris disk name to the valid Hex LUNID.

The code in this gist will cover what’s what. A friendly storage admin told me about this, wrt VMAX/Symmetrix LUNs. Apparently EMC generates the GUID (aka UUID) of LUNs differently based on whether it is a CLARiiON/VNX or a Symmetrix/VMAX.

In that, the fields that were extracted corresponding to variables $xlunid1 and so forth, and converted to their hex values, is the pertinent piece of information. Having this information then reduces our need to install symcli on every host so that we can extract the LUN ID that way.

This then will give us the ability to map a MPxIO/Solaris Disk name to an EMC LUN ID.

The full script is here — https://github.com/implicateorder/sysadmin-tools/blob/master/getluninfo.pl