Untangling HBA multipath

How can you check what HBA is used for a given device?
If you are using multipath, which device is using which port?

Let's say we need to check /dev/sdc
First, check which scsi_device belongs to it:

# ls -l /sys/class/block/sdc/device/scsi_device/
drwxr-xr-x 3 root root 0 Jul 20 12:22 1:0:0:0/

The "1" above means that the device is seen through host1.
Now we can look at host1:

# cat /sys/class/scsi_host/host1/info
Emulex LPe11002-M4 4Gb 2port FC: PCIe SFF HBA on PCI bus 02 device 00 irq 66 port 1

so, it's the second port of an Emulex adapter.

Mounting a disk image

Dumping a full disk is a quick way to perform a backup:

# dd if=/dev/sdb of=filename.dsk

But once you have such a dump, you can't directly mount it, you must use kpartx:

# kpartx -av filename.dsk
add map loop0p1 (252:0): 0 7830408 linear /dev/loop0 1144

You can see the partitions with fdisk

# fdisk -l /dev/loop0

Disk /dev/loop0: 4009 MB, 4009754624 bytes
128 heads, 22 sectors/track, 2781 cylinders, total 7831552 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xc3072e18

      Device Boot      Start         End      Blocks   Id  System
/dev/loop0p1   *        1144     7831551     3915204    b  W95 FAT32

Now you can mount the partitions that are inside the disk image.

# mount /dev/mapper/loop0p1 /mnt
# df -h
Filesystem           Size  Used Avail Use% Mounted on
/dev/sda5             31G   12G   18G  41% /
udev                 3.9G  8.0K  3.9G   1% /dev
tmpfs                1.6G  960K  1.6G   1% /run
none                 5.0M     0  5.0M   0% /run/lock
none                 3.9G  200K  3.9G   1% /run/shm
/dev/sdb5             20G  1.4G   18G   8% /home
/dev/mapper/loop0p1  3.8G  1.1G  2.7G  29% /mnt

Once you've done, you can umount it with

# umount /mnt
# kpartx -dv filename.dsk
del devmap : loop0p1
loop deleted : /dev/loop0

Note:
Ubuntu doesn't have kpartx installed by default:

# sudo apt-get install kpartx

How to check if a variable contains a string

string="this string contains a test substring"

if [[ "$string" == *test* ]]
then
        print "substring found"
fi 

RAID 5 write performance

Everybody knows that RAID 5 has poor write performance, but how much?
With some spare parts at my disposal, I've done some testing:
Here is a RAID 5 volume, built with 7 SAS drives, each one is 146GB @ 10K rpm.
The controller is an HP Smart Array P400


[root@test /]# dd if=/dev/vg_99/lv_share of=/dev/null bs=1024 count=10240000
10240000+0 records in
10240000+0 records out
10485760000 bytes (10 GB) copied, 40.097 s, 262 MB/s
[root@test /]# dd of=/dev/vg_99/lv_share if=/dev/zero bs=1024 count=10240000
10240000+0 records in
10240000+0 records out
10485760000 bytes (10 GB) copied, 500.826 s, 20.9 MB/s

So, 262 MB/s reading vs 20.9 MB/s writing.

/usr/share/cluster/oracledb.sh

It's a bad idea to modify it.
During an upgrade, yum just overwrites it without notice and without creating any .rpmsave

grub rescue >

Panic! On my dual boot pc, a failed linux upgrade had wiped the linux boot partion.
The Windows partitions is still there, but unbootable: the boot process was struck at a grub error

error: no such partition.

grub rescue >

The MRB had to be rewritten but I had no Windows install CD ready at hand.
By chance, I had a GParted Live CD in the desk drawer.
I've booted it and luckily it had the ms-sys command on board: ms-sys just rewrites a Windows MBR on the HD.

# ms-sys -m /dev/hda

Reboot.
Windows was back up and running.

Xvnc black screen

I was configuring my workstation for remote VNC login following this blog post.
The checklist ends suggesting a reboot command.
If you don't obey, you'll only get a black screen on a VNC connection.
That seems to break the spell: no reboots under linux!

After some investigation it seems that the cause was gdm not accepting tcp connections from Xvnc.
A full reboot is not necessary, just a gdm restart:

Find the gdm instance started by init:

# ps -ef --forest

search the process tree for something like this:

root      3621     1  0 14:35 ?        00:00:00 /usr/sbin/gdm-binary -nodaemon

root      3707  3621  0 14:35 ?        00:00:00  \_ /usr/sbin/gdm-binary -nodaemon

root      3713  3707  0 14:35 tty7     00:00:00      \_ /usr/bin/Xorg :0 -br -audit 0 -auth /var/gdm/:0.Xauth -nolisten tcp vt7

gdm       3734  3707  0 14:35 ?        00:00:00      \_ /usr/libexec/gdmgreeter

the parent gdm here has PID 3621

# kill -1 3621

The console X session will restart, and after that, VNC logins will work.

Update:
On RHEL/CentOS 6, you must use kill -9 to reload gdm, and the vnc-server package is called tigervnc-server

Realtime monitoring of syslog messages

Swatch is perl script that continuously monitor log files and acts upon patterns that may show up.
Based on some previous work, I wrote some scripts to use it as a daemon.

The following packages are needed from the EPEL repositories:
swatch-3.2.3-2.el5.noarch.rpm
perl-Mail-Sendmail-0.79-9.el5.1.noarch.rpm

While these are from the base repositories
perl-DateManip
perl-Date-Calc
perl-TimeDate

Here is the /etc/init.d/swatch

#!/bin/sh
#
# swatch: watch system log
#
# chkconfig: 345 10 99
# description: The Simple WATCHer is an automated monitoring tool \
# that is capable of alerting system administrators \
# of anything that matches the patterns described \
# in the configuration file, whilst constantly searching \
# logfiles using perl.
#
# processname: swatch
# config: /etc/sysconfig/swatch/swatch
# pidfile: /var/run/swatch.pid

CHECK_LOG="undefined_logfile"
SWATCH_CONF="undefined_conf"
SWATCH_BIN="/usr/bin/swatch"
SWATCH_PID_FILE="/var/run/swatch.pid"
SWATCH_SCRIPTDIR="/var/run"
SWATCH_LOG="/var/log/swatch"

. /etc/rc.d/init.d/functions

if [ -f /etc/sysconfig/swatch/swatch ]; then
        . /etc/sysconfig/swatch/swatch
else
        echo "/etc/sysconfig/swatch/swatch does not exists."
        exit 0
fi

if [ ! -x ${SWATCH_BIN} ]; then
        echo "File ${SWATCH_BIN} not installed!"
        exit 0
fi

if [ ! -f ${SWATCH_CONF} ]; then
        echo "File ${SWATCH_CONF} does not exist."
        exit 0
fi

prog=swatch
RETVAL=0

start() {
        echo -n $"Starting $prog: "
        daemon "$SWATCH_BIN --daemon -c $SWATCH_CONF -t $CHECK_LOG --pid-file=$SWATCH_PID_FILE --script-dir=$SWATCH_SCRIPTDIR >>$SWATCH_LOG"
        RETVAL=$?
        if [ $RETVAL = 0 ]; then
                success
        else
                failure
        fi
        echo
        return $RETVAL
}

stop() {
        echo -n $"Stopping $prog: "
        killproc "$SWATCH_BIN"
        RETVAL=$?
        if [ $RETVAL = 0 ]; then
                success
        else
                failure
        fi
        echo
        return $RETVAL
}

case "$1" in
        start)
                start
                ;;
        stop)
                stop
                ;;
        status)
                status $prog
                RETVAL=$?
                ;;
        restart)
                stop
                start
                RETVAL=$?
                ;;
        *)
                echo $"Usage: $0 {start|stop|status|restart}"
                exit 1
                ;;
esac

exit $RETVAL

The config files:
Where to watch: /etc/sysconfig/swatch/swatch

# log to watch
CHECK_LOG="/var/log/messages"

# regular expressions
SWATCH_CONF="/etc/sysconfig/swatch/swatchrc"

What to watch: /etc/sysconfig/swatch/swatchrc

# swatch config
watchfor   /regex_to_watch/
        mail addresses=user\@domain,subject=swatch_alert

And the logrotate stuff: /etc/logrotate.d/swatch

/var/log/swatch {
    postrotate
        /etc/init.d/swatch restart 2> /dev/null > /dev/null || true
    endscript
}

Ubuntu-style Putty

Designing a custom font was a stroke of genius from the Ubuntu team.
It gave instant recognisability to the distribution and it boosted ergonomics as well.

I'm currently using the same font even on Windows in my Putty sessions:

The Ubuntu fonts can be downloaded from http://font.ubuntu.com/
Unzip and copy into the Windows fonts folder.
Set Putty to use "Ubuntu Mono 12" with the Antialiased flag.

Here's the reg file for the above configuration:

--- CUT HERE ---

Windows Registry Editor Version 5.00

[HKEY_CURRENT_USER\Software\SimonTatham\PuTTY\Sessions\Default%20Settings]
"Font"="Ubuntu Mono"
"FontIsBold"=dword:00000000
"FontCharSet"=dword:00000000
"FontHeight"=dword:0000000c
"FontVTMode"=dword:00000004
"UseSystemColours"=dword:00000000
"TryPalette"=dword:00000000
"BoldAsColour"=dword:00000001
"Colour0"="187,187,187"
"Colour1"="255,255,255"
"Colour2"="45,15,44"
"Colour3"="85,85,85"
"Colour4"="0,0,0"
"Colour5"="0,255,0"
"Colour6"="0,0,0"
"Colour7"="85,85,85"
"Colour8"="187,0,0"
"Colour9"="255,85,85"
"Colour10"="0,187,0"
"Colour11"="85,255,85"
"Colour12"="187,187,0"
"Colour13"="255,255,85"
"Colour14"="0,150,187"
"Colour15"="85,170,255"
"Colour16"="187,0,187"
"Colour17"="255,85,255"
"Colour18"="0,187,187"
"Colour19"="85,255,255"
"Colour20"="187,187,187"
"Colour21"="255,255,255"
"BoldFont"=""
"BoldFontIsBold"=dword:01000000
"BoldFontCharSet"=dword:00000007
"BoldFontHeight"=dword:00000000
"WideFont"=""
"WideFontIsBold"=dword:00000003
"WideFontCharSet"=dword:00000000
"WideFontHeight"=dword:0000000a
"WideBoldFont"=""
"WideBoldFontIsBold"=dword:00000000
"WideBoldFontCharSet"=dword:00000000
"WideBoldFontHeight"=dword:343232e8
"ShadowBold"=dword:00000000
"ShadowBoldOffset"=dword:00000001
"FontQuality"=dword:00000003
"ANSIColour"=dword:00000001

--- CUT HERE ---