Aug 132018
 

The tool iperf (or iperf3) is popular as a synthetic network load generator that can be used to measure the network throughput between two nodes on a TCP/IP network. This is fine for point-to-point measurements. However, when it comes to distributed computing clusters involving multiple racks of nodes, it becomes challenging to find a tool that can easily simulate traffic that will test the network infrastructure sufficiently. One could of course always set up Hadoop on the cluster (or if Hadoop is already installed), run a TeraGen (MapReduce tool) that will generate heavy network traffic. However, often we are required to test and validate the boundaries and limitations of a network, especially the traffic between nodes of the cluster (which crosses Rack boundaries), often before a software stack can be deployed on the infrastructure. It is useful to be able to use a simple and well known tool (hence iperf3) so that it can provide a common frame of reference, and the tests can easily be reproduced irrespective of the underlying infrastructure (eg: bare-metal vs private cloud vs public cloud).

The objective of this document is, that, using the standard and freely available tool iperf3, we run parallel synthetic network loads to test the limits of a distributed cluster network. In other words, given a known set of iperf3 server instances, randomly run iperf3 client sessions against them simultaneously from multiple hosts. Towards that end, the same hosts can be set up as servers as well as clients (with the caveat that we avoid running a test from any given node against itself).

The methodology introduced in this document can be used to generate synthetic network loads to test East-West network bandwidth.  The diagram below shows the recommended (and standard) network topology for a distributed computing cluster (such as is Hadoop).

The Leaf to Spine uplink is recommended to be such that each node in one rack be able to talk in a non-blocking fashion to each node in another rack. Or, in other words, the E-W oversubscription ratio should be 1:1. If you have 1:1 oversubscription and each node has one 10 gbps NIC, then you should get as close to 10 gbps in bandwidth with these tests when you are running network traffic in parallel across all nodes between two racks.

Practical reasons may result in lower oversubscription ratios of upto 4:1. For really large clusters (more than 500 nodes), this might be higher  (7:1).

Some caveats need to be kept in mind. Hadoop has traditionally been a shared-nothing architecture, and with great emphasis on Data locality. What that means is that the worker nodes of a Hadoop cluster have locally attached drives (SATA is the norm). There are multiple copies of data stored (HDFS defaults to 3 replicas per HDFS block).

However, as network hardware has improved in terms of throughput, it has opened up avenues for flexibility vis-a-vis data locality (40 Gbps, 25 Gbps, 50 Gbps ethernet networks are fast becoming ubiquitous, with 100 Gbps uplink ports becoming more prevalent in enterprise data-centers). As we start seeing more architectures leveraging network based data storage and retrieval in these distributed computing environments, the need to gravitate towards 1:1 oversubscription will also increase correspondingly.

We need the following for this methodology to work —

  • iperf3 installed on all nodes
  • moreutils installed on all nodes
  • A clush setup (clustershell) (a laptop will do).

The clush setup involves creating an entry in the file /etc/clustershell/groups as follows —

 

mc_all:host[102,104,106,108,110,112,114,116,118,120,122,124,126,128,130,132,134,136,138,140].my.company.com host[202,204,206,208,210,212,214,216,218,220,222,224,226,228,230,232,234,236,238,240].my.company.com host[302,304,305-0333].my.company.com

mc_rack1:host[102,104,106,108,110,112,114,116,118,120,122,124,126,128,130,132,134,136,138,140].my.company.com 

mc_rack2 :host[202,204,206,208,210,212,214,216,218,220,222,224,226,228,230,232,234,236,238,240].my.company.com

mc_rack3: host[302,304,305-0333].my.company.com

Install iperf3 and moreutils as follows —

# clush -g mc_all 'sudo yum install -y iperf3 moreutils'

After this is done, launch iperf3 in Daemon mode on all the hosts as follows —

clush -g mc_all -l johndoe sudo iperf3 -sD -p 5001

clush -g mc_all -l johndoe sudo iperf3 -sD -p 5002

clush -g mc_all -l johndoe sudo iperf3 -sD -p 5003

Verify that the iperf3 Daemons are running —

$ clush -g mc_all -l johndoe 'ps -ef|grep iperf3|grep -v grep'

host102.my.company.com: root      10245 1 14 12:50 ? 00:06:42 iperf3 -sD -p 5001

host106.my.company.com: root       9495 1 12 12:50 ? 00:05:36 iperf3 -sD -p 5001

host104.my.company.com: root       9554 1 9 12:50 ? 00:04:20 iperf3 -sD -p 5001

<truncated for readability>

host315.my.company.com: root      33247 1 0 12:57 ? 00:00:00 iperf3 -sD -p 5001

host224.my.company.com: root      33136 1 0 12:57 ? 00:00:00 iperf3 -sD -p 5001

host323.my.company.com: root      33257 1 0 12:57 ? 00:00:00 iperf3 -sD -p 5001

host318.my.company.com: root      32868 1 0 12:57 ? 00:00:00 iperf3 -sD -p 5001

host236.my.company.com: root      33470 1 0 12:57 ? 00:00:00 iperf3 -sD -p 5001

host236.my.company.com: johndoe   33734 33492 22 13:34 ? 00:00:10 iperf3 -c host120.my.company.com -p 5001 -t 60

 

After all the nodes have been setup to run iperf3 server instances in Daemon mode, create multiple nodelist files (in this case, we are testing cross-rack bandwidth, and so set up the nodelist per rack as follows —

$ ls -lrt 

total 32

-rw-r--r--  1 johndoe  staff 806 Aug  9 14:52 nodes.rack3

-rw-r--r--  1 johndoe  staff 520 Aug  9 14:52 nodes.rack2

-rw-r--r--  1 johndoe  staff 520 Aug  9 14:52 nodes.rack1

-rwxr-xr-x  1 johndoe  staff 221 Aug  9 14:52 random_net.sh

-rwxr-xr-x  1 johndoe  staff 221 Aug  9 14:52 randomnet_single.sh

Distribute these files to all nodes of the cluster. Assumptions are that you already have clush setup (so have keys to the kingdom, so to speak — ie an user id on each node which has sudo privileges as root etc).

$ clush -g mc_all -c nodes.rack* --dest=/home/johndoe

$ clush -g mc_all -c randomnet*.sh --dest=/home/johndoe

Each file contains a list of all nodes in the specific rack. Eg:

$ cat nodes.rack1 

host102.my.company.com

host104.my.company.com

host106.my.company.com

host108.my.company.com 

host110.my.company.com

The shell script random_net.sh is intended to randomly select a target node against which to run the iperf test. This is to be run on the client side, say from Rack2 (client) to Rack1 (server) —

#!/bin/sh

# Read nodelist into an array

IFS=$'\r\n' GLOBIGNORE='*' command eval 'nodes=($(cat $1))'

# Create an array of ports on which iperf3 server daemon instances will be running

ports=(5001 5002 5003)

psize=${#ports[@]}

size=${#nodes[@]}

for i in $(seq $size)

do

    index=$(( $RANDOM % size ))

    pindex=$(( $RANDOM % psize ))

    target=${nodes[$index]}

    ptarget=${ports[$pindex]}

    iperf3 -c $target -p $ptarget |ts |tee -a ~/iperf3.log  # Run payload from Client to server

    iperf3 -c $target -R -p $ptarget |ts |tee -a ~/iperf3.log  # Reverse the direction

done

In order to run against a single iperf3 server instance per node, run this script instead

#!/bin/sh

# Read nodelist into an array

IFS=$'\r\n' GLOBIGNORE='*' command eval 'nodes=($(cat $1))'

# Create an array of ports on which iperf3 server daemon instances will be running

size=${#nodes[@]}

for i in $(seq $size)

do

    index=$(( $RANDOM % size ))

    target=${nodes[$index]}

    iperf3 -c $target -p 5001 |ts |tee -a ~/${target}_iperf3.log    # Run payload from Client to server

    iperf3 -c $target -R -p 5001|ts |tee -a  ~/${target}_iperf3.log # Reverse the direction

done

Run the script(s) as follows —

$ clush -g mc_rack2 -l johndoe 'sh random_net.sh nodes.rack1'

Or

$ clush -g mc_rack2 -l johndoe 'sh randomnet_single.sh nodes.rack1'

This example shows the test running in parallel on all nodes of Rack2 by randomly selecting a node in Rack1 as the target. The output of the tests can be parsed to generate usable reports that will help ascertain if there is a bottleneck anywhere.

$ clush -g mc_rack2 -l johndoe 'sh random_net.sh nodes.rack1'

host218.my.company.com: Aug 09 14:31:17 Connecting to host host108.my.company.com, port 5001

host218.my.company.com: Aug 09 14:31:17 [ 4] local 192.168.1.59 port 54962 connected to 192.168.1.14 port 5001

host218.my.company.com: Aug 09 14:31:17 [ ID] Interval Transfer     Bandwidth Retr Cwnd

host218.my.company.com: Aug 09 14:31:17 [ 4]   0.00-1.00 sec 646 MBytes 5.42 Gbits/sec 80   595 KBytes 

host218.my.company.com: Aug 09 14:31:17 [ 4]   1.00-2.00 sec 542 MBytes 4.55 Gbits/sec 50   561 KBytes 

host218.my.company.com: Aug 09 14:31:17 [ 4]   2.00-3.00 sec 574 MBytes 4.81 Gbits/sec 29   577 KBytes 

host204.my.company.com: Aug 09 14:31:17 Connecting to host host140.my.company.com, port 5001

host204.my.company.com: Aug 09 14:31:17 [ 4] local 192.168.1.52 port 47034 connected to 192.168.1.30 port 5001

host204.my.company.com: Aug 09 14:31:17 [ ID] Interval Transfer     Bandwidth Retr Cwnd

host204.my.company.com: Aug 09 14:31:17 [ 4]   0.00-1.00 sec 870 MBytes 7.30 Gbits/sec 38   799 KBytes 

host204.my.company.com: Aug 09 14:31:17 [ 4]   1.00-2.00 sec 626 MBytes 5.25 Gbits/sec 28   454 KBytes 

host204.my.company.com: Aug 09 14:31:17 [ 4]   2.00-3.00 sec 516 MBytes 4.33 Gbits/sec 19   512 KBytes 

host204.my.company.com: Aug 09 14:31:17 [ 4]   3.00-4.00 sec 590 MBytes 4.95 Gbits/sec 19   656 KBytes 

host204.my.company.com: Aug 09 14:31:17 [ 4]   4.00-5.00 sec 581 MBytes 4.88 Gbits/sec 88   649 KBytes 

host204.my.company.com: Aug 09 14:31:17 [ 4]   5.00-6.00 sec 570 MBytes 4.78 Gbits/sec 19   592 KBytes 

host204.my.company.com: Aug 09 14:31:17 [ 4]   6.00-7.00 sec 561 MBytes 4.71 Gbits/sec 41   560 KBytes 

host204.my.company.com: Aug 09 14:31:17 [ 4]   7.00-8.00 sec 589 MBytes 4.94 Gbits/sec 91   563 KBytes 

host204.my.company.com: Aug 09 14:31:17 [ 4]   8.00-9.00 sec 539 MBytes 4.52 Gbits/sec 46   479 KBytes 

host204.my.company.com: Aug 09 14:31:17 [ 4]   9.00-10.00 sec 570 MBytes 4.78 Gbits/sec 68   607 KBytes 

host204.my.company.com: Aug 09 14:31:17 - - - - - - - - - - - - - - - - - - - - - - - - -

host204.my.company.com: Aug 09 14:31:17 [ ID] Interval Transfer     Bandwidth Retr

host204.my.company.com: Aug 09 14:31:17 [ 4]   0.00-10.00 sec 5.87 GBytes 5.04 Gbits/sec 457             sender

host204.my.company.com: Aug 09 14:31:17 [ 4]   0.00-10.00 sec 5.87 GBytes 5.04 Gbits/sec              receiver

host204.my.company.com: Aug 09 14:31:17 

host204.my.company.com: Aug 09 14:31:17 iperf Done.

host218.my.company.com: Aug 09 14:31:17 [ 4]   3.00-4.00 sec 636 MBytes 5.34 Gbits/sec 12   484 KBytes 

host218.my.company.com: Aug 09 14:31:17 [ 4]   4.00-5.00 sec 508 MBytes 4.26 Gbits/sec 65   433 KBytes 

host218.my.company.com: Aug 09 14:31:17 [ 4]   5.00-6.00 sec 384 MBytes 3.22 Gbits/sec 62   566 KBytes 

host218.my.company.com: Aug 09 14:31:17 [ 4]   6.00-7.00 sec 632 MBytes 5.30 Gbits/sec 69   519 KBytes 

host218.my.company.com: Aug 09 14:31:17 [ 4]   7.00-8.00 sec 595 MBytes 4.99 Gbits/sec 30   650 KBytes 

host218.my.company.com: Aug 09 14:31:17 [ 4]   8.00-9.00 sec 564 MBytes 4.73 Gbits/sec 45   478 KBytes 

host218.my.company.com: Aug 09 14:31:17 [ 4]   9.00-10.00 sec 525 MBytes 4.40 Gbits/sec 26   444 KBytes 

host218.my.company.com: Aug 09 14:31:17 - - - - - - - - - - - - - - - - - - - - - - - - -

host218.my.company.com: Aug 09 14:31:17 [ ID] Interval Transfer     Bandwidth Retr

host218.my.company.com: Aug 09 14:31:17 [ 4]   0.00-10.00 sec 5.47 GBytes 4.70 Gbits/sec 468             sender

host218.my.company.com: Aug 09 14:31:17 [ 4]   0.00-10.00 sec 5.47 GBytes 4.70 Gbits/sec              receiver

 

NOTE: This blog documents my personal opinions only and does not reflect my employer’s positions on any subject written about here. 

Mar 182014
 

This is an old one. I thought I’d post this before it falls off main memory.

You will need two things for this. I’ll start with the obvious.

  1. A splunk server configured to act as a deployment server (which will result in apps, configurations, etc being deployed centrally)
  2. A centralized “bastion” host, that has root-level ssh-based trust established with your target hosts (ie splunk agents). Another option is to have unprivileged ssh key-based trusts set up, but with local sudo privileges (passwordless) on each of your target hosts.

Once you do that, you can run a modification of this script shown below —

I’d written in for an old shop of mine, where we managed Solaris on SPARC, Solaris on x86 and Redhat Linux. Modify/write functions corresponding to your target OS of choice and it should do pretty much what you need rather easily.

Another thing to watch out for is make sure the version of splunk you are working with hasn’t changed the underlying mechanisms. I had written this for Splunk v4.x at least 3 years back (can’t remember how long ago now – this is one script I didn’t put in version control, so no way for me to tell how old it…and timestamps changed because I copied them from my old laptop, etc).

 

Mar 122014
 

Okay, this has been a long time coming, but thought I’d write this down before I forget all about it.

First a little bit of a rant. In the EMC world, we assign LUNs to hosts and identify them using a hex-id code called the LUN ID. This, in conjunction with the EMC Frame ID, helps us uniquely and “easily” identify the LUNs, especially when you consider that there are usually multiple EMC frames and hundreds or thousands of SAN-attached hosts in most modern enterprise data centers.

Okay, the challenge is when using a disk multipathing solution other than EMC’s expensive power path software, or the exorbitantly expensive Veritas Storage Foundation suite from Symantec.

Most self-respecting modern operating systems have disk multipathing software built in and freely available.  and the focus item for this blog is Solaris. Solaris has a mature and efficient multipathing software called MPxIO, which cleverly creates a pseudo-device corresponding to the complex of the multiple (or single) paths via which a SAN LUN is visible to the Operating system.

The MPxIO driver, then using the LUN’s global unique identifier (GUID) to, perhaps can be argued, rightly identify the LUN uniquely. This is however a challenge since the GUID is a 32-character string (very nicely explained here http://www.slideshare.net/JamesCMcPherson/what-is-a-guid).

When I first proposed to our usual user community – Oracle DBAs, that they would have to use disks named as shown below, I faced outraged howls of indignation. “How can we address these disks in ASM, etc?”

c3t60060160294122002A2ADDB296EADE11d0

To work around, we considered creating an alternate namespace, say /dev/oracle/disk001 and so forth, retaining the major/minor numbers of the underlying devices.  But that would get hairy real quick, especially on those databases where we had multiple terabytes of storage (hundreds of LUNs).

If you are working with CLARiiON arrays, then you will have figure out some other way (than what I’ve shown in this blog post) to map your MPxIO/Solaris disk name to the valid Hex LUNID.

The code in this gist will cover what’s what. A friendly storage admin told me about this, wrt VMAX/Symmetrix LUNs. Apparently EMC generates the GUID (aka UUID) of LUNs differently based on whether it is a CLARiiON/VNX or a Symmetrix/VMAX.

In that, the fields that were extracted corresponding to variables $xlunid1 and so forth, and converted to their hex values, is the pertinent piece of information. Having this information then reduces our need to install symcli on every host so that we can extract the LUN ID that way.

This then will give us the ability to map a MPxIO/Solaris Disk name to an EMC LUN ID.

The full script is here — https://github.com/implicateorder/sysadmin-tools/blob/master/getluninfo.pl

Jan 022014
 

This is a continuation of my previous post on this topic.

First, a disclaimer –

  • I have focused on CentOS primarily. I will try and update this to accommodate more than one Linux distro (and even consider writing for Solaris-based implementations in the future).

Here’s the skeleton of the cloudera manager setup plugin:

import posixpath

from starcluster import threadpool
from starcluster import clustersetup
from starcluster.logger import log

class ClouderaMgr(clustersetup.ClusterSetup):

        def __init__(self,cdh_mgr_agentdir='/etc/cloudera-scm-agent'):
                self.cdh_mgr_agentdir = cdh_mgr_agentdir
                self.cdh_mgr_agent_conf = '/etc/cloudera-scm-agent/config.ini'
                self.cdh_mgr_repo_conf = '/etc/yum.repos.d/cloudera-cdh4.repo'
                self.cdh_mgr_repo_url = 'http://archive.cloudera.com/cm4/redhat/6/x86_64/cm/cloudera-manager.repo'
                self._pool = None

        @property
        def pool(self):
                if self._pool is None:
                        self._pool = threadpool.get_thread_pool(20, disable_threads=False)
                return self._pool

        def _install_cdhmgr_repo(self,node):
                node.ssh.execute('wget %s' % self.cdh_mgr_repo_url)
                node.ssh.execute('cat /root/cloudera-manager.repo >> %s' % self.cdh_mgr_repo_conf)

        def _install_cdhmgr_agent(self,node):
                node.ssh.execute('yum install -y cloudera-manager-agent')
                node.ssh.execute('yum install -y cloudera-manager-daemons')

        def _install_cdhmgr(self,master):
                master.ssh.execute('/sbin/service cloudera-scn-agent stop')
                master.ssh.execute('/sbin/chkconfig cloudera-scn-agent off')
                master.ssh.execute('/sbin/chkconfig hue off')
                master.ssh.execute('/sbin/chkconfig oozie off')
                master.ssh.execute('/sbin/chkconfig hadoop-httpfs off')
                master.ssh.execute('yum install -y cloudera-manager-server')
                master.ssh.execute('yum install -y cloudera-manager-server-db')
                master.ssh.execute('/sbin/service cloudera-scm-server-db start')
                master.ssh.execute('/sbin/service cloudera-scm-server start')

        def _setup_hadoop_user(self,node,user):
                node.ssh.execute('gpasswd -a %s hadoop' %user)

        def _install_agent_conf(self, node):
                node.ssh.execute('/bin/sed -e"s/server_host=localhost/server_host=master/g" self.cdh_mgr_agent_conf > /tmp/config.ini; mv /tmp/config.ini self.cdh_mgr_agent_conf')

        def _open_ports(self,master):
                ports = [7180,50070,50030]
                ec2 = master.ec2
                for group in master.cluster_groups:
                        for port in ports:
                                has_perm = ec2.has_permission(group, 'tcp', port, port, '0.0.0.0/0')
                                if not has_perm:
                                        ec2.conn.authorize_security_group(group_id=group.id,
                                                                        ip_protocol='tcp',
                                                                        from_port=port,
                                                                        to_port=port,
                                                                        cidr_ip='0.0.0.0/0')

        def run(self,nodes, master, user, user_shell, volumes):
                for node in nodes:
                     self._install_cdhmgr_repo(node)
                     self._install_cdhmgr_agent(node)
                     self._install_agent_conf(node)
                self._install_cdhmgr(master)
                self._open_ports(master)
        def on_add_node(self, node, nodes, master, user, user_shell, volumes):
                for node in nodes:
                     self._install_cdhmgr_repo(node)
                     self._install_cdhmgr_agent(node)
                     self._install_agent_conf(node)

And this plugin can be referenced and executed in the cluster configuration as follows:

[plugin cdhmgr]
setup_class = cdh_mgr.ClouderaMgr

[cluster testcluster]
keyname = testcluster
cluster_size = 2
cluster_user = sgeadmin
cluster_shell = bash
master_image_id = ami-232b034a
master_instance_type = t1.micro
node_image_id =  ami-232b034a
node_instance_type = t1.micro
plugins = wgetter,rpminstaller,repoinstaller,pkginstaller,cdhmgr

Once this step is successfully executed by StarCluster, you will be able to access the cloudera manager web GUI on port 7180 of your master node’s public IP and/or DNS entry.

<...more to follow...>

Dec 132013
 

I ran into an incredible tool known as StarCluster, which is an open-source project from MIT (http://star.mit.edu/cluster/).

StarCluster is built using Sun Microsystem’s N1 Grid Engine software (Sun used it to do deployment for HPC environments). And the folks at MIT developed on a fork of that (SGE – Sun Grid Engine) and StarCluster was formed.

StarCluster has hooks into the AWS API and is used to dynamically and automatically provision multi-node clusters in Amazon’s EC2 cloud. The base StarCluster software provisions a pre-selected virtual machine image (AMI) onto a pre-defined VM type (e.g.: t1.micro, m1.small, etc). The distributed StarCluster software has a hadoop plugin (http://star.mit.edu/cluster/docs/0.93.3/plugins/hadoop.html) which installs the generic Apache Foundation Hadoop stack on the cluster of nodes (or single node) provisioned thereof. It then uses a tool called dumbo (http://klbostee.github.io/dumbo/) to drive the Hadoop Framework. This is great.

But I want to leverage Cloudera’s CDH manager based Hadoop Distribution (CDH?) and want to roll this into the StarCluster framework. Therefore I have to start developing the necessary plugin to implement Cloudera Hadoop on top of StarCluster.

I won’t delve deeper into StarCluster’s inner-workings in this post, as I just started working on it a couple of days ago. I do intend to see how I can leverage this toolkit to add more functionality into it (like integrating Ganglia-based monitoring, etc) in due time.

Why StarCluster and not a Vagrant, Jenkins, Puppet combination?

Tools like StarCluster are meant to do rapid and effective deployments of distributed computing environments on a gigantic scale (N1 grid engine was used to deploy hundreds of physical servers back in the days when Sun still shone on the IT industry). While Vagrant, Puppet, Jenkins etc can be put together to build such a framework, it is in essence easier to use StarCluster (with less effort) to operationalize a Hadoop Cluster on EC2 (and after I spend some more time on the inner-workings, I hope to be able to use it work on other virtualization technologies too – perhaps integrate with an on-premise Xen or vmware or KVM cluster).

Let’s Skim the surface:

How StarCluster works

Let’s start with how StarCluster works (and this pre-supposes a working knowledge of Amazon’s AWS/EC2 infrastructure).

It is in essence a toolkit that you can install on your workstation/laptop and use it to drive configuration and deployment of your distributed clustering environment on EC2.

To set it up, follow the instructions on MIT’s website. It was pretty straightforward on my Mac Book Pro OS ver 10.9 (Mavericks). I already had the Xcode software installed on my Mac. So I cloned the StarCluster Git Distribution and ran through two commands that needed to be run as listed here — http://star.mit.edu/cluster/docs/latest/installation.html

After that, I set up my preliminary configuration file here (~/.starcluster/config) and we were ready to roll. I chose to test the star cluster functionality without and plugins etc by setting up a single node.

It’s important to choose the right AMI (because not all AMIs can be deployed on VM’s of your choice. For instance, I wanted to use the smallest, t1.micro instance to do my testing, since I won’t really be using this for any actual work. I settled on a CentOS 5.4 AMI from RealStack.

After verifying that my single node cluster was being setup correctly, I proceeded to develop a plugin to do Cloudera Hadoop initial setup along with a Cloudera Manager  installation on the Master node. As a result thereof, we can then roll out a cluster, connect to Cloudera Manager and use the Cloudera Manager interface to configure the Hadoop cluster. If further automation is possible with Cloudera Manager, I will work on that as an addendum to this effort.

The Plugins

Following the instructions here, I started building out the framework for my plugin. I called the foundational plugin centos (since my selected OS is centos). The CDH specific one will be called Cloudera.

Following directory structure is set up after StarCluster is installed –

In your home directory, a .starcluster directory, under which we have:

total 8
drwxr-xr-x   9 dwai  dlahiri   306 Dec 12 18:15 logs
drwxr-xr-x   4 dwai  dlahiri   136 Dec 13 10:01 plugins
-rw-r--r--   1 dwai  dlahiri  2409 Dec 13 10:01 config
drwxr-xr-x+ 65 dwai  dlahiri  2210 Dec 13 10:01 ..
drwxr-xr-x   5 dwai  dlahiri   170 Dec 13 10:01 .

Your new plugins will go into the plugins directory listed above (for now).

This is as simple as this (the plugin in essence is a python script – we call it centos.py):

from starcluster.clustersetup import ClusterSetup
from starcluster.logger import log

global repo_to_install
global pkg_to_install

class WgetPackages(ClusterSetup):
	def __init__(self,pkg_to_wget):
		self.pkg_to_wget = pkg_to_wget
		log.debug('pkg_to_wget = %s' % pkg_to_wget)
	def run(self, nodes, master, user, user_shell, volumes):
		for node in nodes:
			log.info("Wgetting %s on %s" % (self.pkg_to_wget, node.alias))
			node.ssh.execute('wget %s' % self.pkg_to_wget)

class RpmInstaller(ClusterSetup):
	def __init__(self,rpm_to_install):
		self.rpm_to_install = rpm_to_install
		log.debug('rpm_to_install = %s' % rpm_to_install)
	def run(self, nodes, master, user, user_shell, volumes):
		for node in nodes:
			log.info("Installing %s on %s" % (self.rpm_to_install, node.alias))
			node.ssh.execute('yum -y --nogpgcheck localinstall %s' %self.rpm_to_install)

class RepoConfigurator(ClusterSetup):
	def __init__(self,repo_to_install):
		self.repo_to_install  = repo_to_install
		log.debug('repo_to_install = %s' % repo_to_install)
	def run(self, nodes, master, user, user_shell, volumes):
		for node in nodes:
			log.info("Installing %s on %s" % (self.repo_to_install, node.alias))
			node.ssh.execute('rpm --import %s' % self.repo_to_install)

class PackageInstaller(ClusterSetup):
	def __init__(self,pkg_to_install):
		self.pkg_to_install  = pkg_to_install
		log.debug('pkg_to_install = %s' % pkg_to_install)
	def run(self, nodes, master, user, user_shell, volumes):
		for node in nodes:
			log.info("Installing %s on %s" % (self.pkg_to_install, node.alias))
			node.ssh.execute('yum -y install %s' % self.pkg_to_install)

Now we can reference the plugin in our configuration file:

[cluster testcluster]
keyname = testcluster
cluster_size = 1
cluster_user = sgeadmin
cluster_shell = bash
node_image_id = ami-0db22764
node_instance_type = t1.micro
plugins = wgetter,rpminstaller,repoinstaller,pkginstaller,sge
permissions = zookeeper.1, zookeeper.2, accumulo.monitor, hdfs, jobtracker, tablet_server, master_server, accumulo_logger, accumulo_tracer, datanode_data, datanode_metadata, tasktrackers, namenode_http_monitor, datanode_http_monitor, accumulo, accumulo_http_monitor

[plugin wgetter]
setup_class = centos.WgetPackages
pkg_to_wget = "http://archive.cloudera.com/cdh4/one-click-install/redhat/5/x86_64/cloudera-cdh-4-0.x86_64.rpm"

[plugin rpminstaller]
setup_class = centos.RpmInstaller
rpm_to_install = cloudera-cdh-4-0.x86_64.rpm

[plugin repoinstaller]
setup_class = centos.RepoConfigurator
repo_to_install = "http://archive.cloudera.com/cdh4/redhat/5/x86_64/cdh/RPM-GPG-KEY-cloudera"

[plugin pkginstaller]
setup_class = centos.PackageInstaller
pkg_to_install = hadoop-0.20-mapreduce-jobtracker

This is not complete by any means – I intend to post the complete plugin + framework in a series of subsequent posts.