StarCluster, Cloudera Manager, EC2 Part 2

This is a continuation of my previous post on this topic. First, a disclaimer – I have focused on CentOS primarily. I will try and update this to accommodate more than one Linux distro (and even consider writing for Solaris-based implementations in the future). Here’s the skeleton of the cloudera manager setup plugin: import posixpath …

StarCluster, Cloudera Manager, EC2 Part 2 Read More »

Cloudera Hadoop, StarCluster and Amazon EC2

I ran into an incredible tool known as StarCluster, which is an open-source project from MIT (http://star.mit.edu/cluster/). StarCluster is built using Sun Microsystem’s N1 Grid Engine software (Sun used it to do deployment for HPC environments). And the folks at MIT developed on a fork of that (SGE – Sun Grid Engine) and StarCluster was …

Cloudera Hadoop, StarCluster and Amazon EC2 Read More »

Using Perl and Graphviz to simplify your busy life

We all have to create topology diagrams, high level visual representations of our thoughts or environments (or software solutions etc) from time to time, for as they say – “A picture is worth a thousand words”. The Graphviz software (http://www.graphviz.org/) provides us a simple and elegant way to do so, which with a little bit …

Using Perl and Graphviz to simplify your busy life Read More »

STREAM(like) benchmark, Solaris and the SPARC T4 processor

About the STREAM benchmark http://blogs.utexas.edu/jdm4372/tag/stream-benchmark/ Here’s what the author has to say about the benchmark itself — What is STREAM? The STREAM benchmark is a simple synthetic benchmark program that measures sustainable memory bandwidth (in MB/s) and the corresponding computation rate for simple vector kernels. /*———————————————————————–*/ /* Program: Stream */ /* Revision: $Id: stream.c,v 5.9 2009/04/11 …

STREAM(like) benchmark, Solaris and the SPARC T4 processor Read More »

Using mdb and DTrace to detect Memory Leaks

It’s been a while since I’ve posted anything. So, here goes: Identifying kernel memory leak on myhost-ldom (after P2V and OS upgrade from Solaris 9 to Solaris 10u10): kmem_track.d sees this: Tracing…If you see more allocs than frees, there is a potential issue… Check against the cache name that is suspect CACHE NAME ALLOCS FREES kmem_bufctl_audit_cache …

Using mdb and DTrace to detect Memory Leaks Read More »

Password Vault in Perl

The password vault tool uses the “Crypt::Simple” perl module (don’t go by the name, it’s got a bunch of stuff happening under the hood) to encrypt anything that is typed into the interface and saves it on a file-system acl’ed back end store.

Only a select group (the System Admins) have ability to access this backend store (as a result the Vault application). The appliction has been written to search, modify, add and delete entries and each action is audited in the backend repository. Using a simply Curses based menu, the app is self-intuitive and requires no training.