/usr/share/cluster/oracledb.sh

It's a bad idea to modify it.
During an upgrade, yum just overwrites it without notice and without creating any .rpmsave

Xvnc black screen

I was configuring my workstation for remote VNC login following this blog post.
The checklist ends suggesting a reboot command.
If you don't obey, you'll only get a black screen on a VNC connection.
That seems to break the spell: no reboots under linux!

After some investigation it seems that the cause was gdm not accepting tcp connections from Xvnc.
A full reboot is not necessary, just a gdm restart:

Find the gdm instance started by init:

# ps -ef --forest

search the process tree for something like this:

root      3621     1  0 14:35 ?        00:00:00 /usr/sbin/gdm-binary -nodaemon

root      3707  3621  0 14:35 ?        00:00:00  \_ /usr/sbin/gdm-binary -nodaemon

root      3713  3707  0 14:35 tty7     00:00:00      \_ /usr/bin/Xorg :0 -br -audit 0 -auth /var/gdm/:0.Xauth -nolisten tcp vt7

gdm       3734  3707  0 14:35 ?        00:00:00      \_ /usr/libexec/gdmgreeter

the parent gdm here has PID 3621

# kill -1 3621

The console X session will restart, and after that, VNC logins will work.

Update:
On RHEL/CentOS 6, you must use kill -9 to reload gdm, and the vnc-server package is called tigervnc-server

Realtime monitoring of syslog messages

Swatch is perl script that continuously monitor log files and acts upon patterns that may show up.
Based on some previous work, I wrote some scripts to use it as a daemon.

The following packages are needed from the EPEL repositories:
swatch-3.2.3-2.el5.noarch.rpm
perl-Mail-Sendmail-0.79-9.el5.1.noarch.rpm

While these are from the base repositories
perl-DateManip
perl-Date-Calc
perl-TimeDate

Here is the /etc/init.d/swatch

#!/bin/sh
#
# swatch: watch system log
#
# chkconfig: 345 10 99
# description: The Simple WATCHer is an automated monitoring tool \
# that is capable of alerting system administrators \
# of anything that matches the patterns described \
# in the configuration file, whilst constantly searching \
# logfiles using perl.
#
# processname: swatch
# config: /etc/sysconfig/swatch/swatch
# pidfile: /var/run/swatch.pid

CHECK_LOG="undefined_logfile"
SWATCH_CONF="undefined_conf"
SWATCH_BIN="/usr/bin/swatch"
SWATCH_PID_FILE="/var/run/swatch.pid"
SWATCH_SCRIPTDIR="/var/run"
SWATCH_LOG="/var/log/swatch"

. /etc/rc.d/init.d/functions

if [ -f /etc/sysconfig/swatch/swatch ]; then
        . /etc/sysconfig/swatch/swatch
else
        echo "/etc/sysconfig/swatch/swatch does not exists."
        exit 0
fi

if [ ! -x ${SWATCH_BIN} ]; then
        echo "File ${SWATCH_BIN} not installed!"
        exit 0
fi

if [ ! -f ${SWATCH_CONF} ]; then
        echo "File ${SWATCH_CONF} does not exist."
        exit 0
fi

prog=swatch
RETVAL=0

start() {
        echo -n $"Starting $prog: "
        daemon "$SWATCH_BIN --daemon -c $SWATCH_CONF -t $CHECK_LOG --pid-file=$SWATCH_PID_FILE --script-dir=$SWATCH_SCRIPTDIR >>$SWATCH_LOG"
        RETVAL=$?
        if [ $RETVAL = 0 ]; then
                success
        else
                failure
        fi
        echo
        return $RETVAL
}

stop() {
        echo -n $"Stopping $prog: "
        killproc "$SWATCH_BIN"
        RETVAL=$?
        if [ $RETVAL = 0 ]; then
                success
        else
                failure
        fi
        echo
        return $RETVAL
}

case "$1" in
        start)
                start
                ;;
        stop)
                stop
                ;;
        status)
                status $prog
                RETVAL=$?
                ;;
        restart)
                stop
                start
                RETVAL=$?
                ;;
        *)
                echo $"Usage: $0 {start|stop|status|restart}"
                exit 1
                ;;
esac

exit $RETVAL

The config files:
Where to watch: /etc/sysconfig/swatch/swatch

# log to watch
CHECK_LOG="/var/log/messages"

# regular expressions
SWATCH_CONF="/etc/sysconfig/swatch/swatchrc"

What to watch: /etc/sysconfig/swatch/swatchrc

# swatch config
watchfor   /regex_to_watch/
        mail addresses=user\@domain,subject=swatch_alert

And the logrotate stuff: /etc/logrotate.d/swatch

/var/log/swatch {
    postrotate
        /etc/init.d/swatch restart 2> /dev/null > /dev/null || true
    endscript
}

libfipscheck

Another yum mess:
After an apparently successfull yum update, sshd didn't start with the following error message:

/usr/lib/libfipscheck.so.1: file too short

The file was 0 bytes.
Where does it come from?

# yum provides libfipscheck.so.1

fipscheck-lib-1.2.0-1.el5.i386 : Library files for fipscheck
Repo        : rhel-5.5-x86_64
Matched from:
Other       : libfipscheck.so.1

the package was installed but somewhat corrupted, so I reinstalled it:

# yum reinstall fipscheck-lib

after that, sshd started fine.

Yum headaches

After a yum update on a workstation, I've found the root mailbox full of messages like this:

/etc/cron.daily/0logwatch:

Undefined subroutine &main::TimeBuild called at /etc/cron.daily/0logwatch line 846.

}

{

The yum itself had problems:

# yum

There was a problem importing one of the Python modules

required to run yum. The error leading to this problem was:

   cannot import name RpmUtilsError

Please install a package which provides this module, or

verify that the module is installed correctly.

It's possible that the above module doesn't match the

current version of Python, which is:

2.4.3 (#1, Apr 14 2011, 20:42:01)

[GCC 4.1.2 20080704 (Red Hat 4.1.2-50)]

If you cannot solve this problem yourself, please go to

the yum faq at:

  http://wiki.linux.duke.edu/YumFaq

I've reinstalled yum

rpm -iv yum-3.2.22-37.el5.noarch.rpm rpm-python-4.4.2.3-22.el5.i386.rpm rpm-4.4.2.3-22.el5.i386.rpm

rpm-apidocs-4.4.2.3-22.el5.i386.rpm rpm-build-4.4.2.3-22.el5.i386.rpm rpm-devel-4.4.2.3-22.el5.i386.rpm rpm-libs-4.4.2.3-22.el5.i386.rpm

and yum was back in action, but there were other problems:
with an rpm -qa I looked at the installed packages and they appeared to be less than usual.
For example there was no mkinitrd package installed but the mkinitrd command was right there.
Probably the rpm database was corrupted but an

# rpm --rebuilddb

sorted no different effect.
I did an rpm -qa on a similarly configured workstation and compared it with the rpm -qa of this machine: there were more than 140 rpms missing.
I extracted the missing list and reinstalled all of them with a big yum install command

# yum install \

NetworkManager \

NetworkManager-glib \

NetworkManager-gnome \

a2ps \

at-spi \

at-spi-devel \

autofs \

avahi \

avahi-compat-libdns_sd \
...and 140 more...

Live Linux P2V

Here's how to clone a live, powered-on machine to a VMWare VM.

The test case scenario is a conversion from an HP BL460 to a Version 4 ESX VM, with LSI Logic parallel virtual SCSI controller and one e1000 vnic.
Linux version is Red Hat 5.5 x64
UPDATE: I've made some encouraging experiments with 6.2 also

Important: this method does not guarantee data consistency, as the disk will be copied while active, but it's usually good enought to bring up a bootable copy of the source machine where you can later restore a consistent backup.
On the other hand, this procedure gives way less downtime than a cold clone, and you can test the cloning process without halting the source machine.

Onward I will call the source machine "P" and the destination machine "V".

Take note of the P disk configuration

# fdisk -l /dev/cciss/c0d0

Disk /dev/cciss/c0d0: 146.7 GB, 146778685440 bytes
255 heads, 32 sectors/track, 35132 cylinders
Units = cylinders of 8160 * 512 = 4177920 bytes

           Device Boot      Start         End      Blocks   Id  System
/dev/cciss/c0d0p1               1          25      101984   83  Linux
/dev/cciss/c0d0p2              26       23961    97658880   8e  Linux LVM

This is an HP machine with a SmartArray controller.

Create a new VM, called V, with a disk at least as big a the P one: in this case I've created a 147GB vmdk.
Boot the V VM into rescue mode.

boot: linux rescue

Start the network interface and assign an IP enabled for SSH with the P machine.
Skip searching for existing installations.
All the following commands are to be executed on the V machine.
Now dump the P hard drive to the V vmdk:

# ssh root@P "cat /dev/cciss/c0d0 | gzip -1 | cat" | gzip -d >/dev/sda

note that this compresses the data over the wire, as this usually leads to faster transfers, even more if the disk has many empty sectors.

After completion, an fdisk should display the same partition table of the P machine on the V machine

# fdisk -l /dev/sda

Disk /dev/sda: 157.8 GB, 157840048128 bytes
255 heads, 32 sectors/track, 37779 cylinders
Units = cylinders of 8160 * 512 = 4177920 bytes

           Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1          25      101984   83  Linux
/dev/sda2              26       23961    97658880   8e  Linux LVM


Reboot V into rescue mode again

boot: linux rescue

This time choose to search for existing installations, as it should find the new disk content.

# chroot /mnt/sysimage

Make sure the LVM has seen the new disk

# pvscan

PV /dev/sda2       VG VolGroup00   lvm2 [93.13 GB / 59.13 GB free]

If you don't see the PV, check /etc/lvm/lvm.conf for any filter other than the default one.

Modify the boot modules list taking as an example those of another identically configured VM.
This is the modprobe.conf of a tipical ESX VM.

# vi /etc/modprobe.conf

alias eth0 e1000
alias scsi_hostadapter mptbase
alias scsi_hostadapter1 mptspi
alias scsi_hostadapter2 ata_piix

Rebuild the initial ram disk, as referenced in the grub.conf

# mkinitrd -v -f /boot/initrd-2.6.18-194.el5.img 2.6.18-194.el5

Reinstall grub, just to be safe (it should be already there from the disk cloning process).

# grub-install /dev/sda

Now note that, after reboot, the V machine will start with the same IP address of the original P one.
If it doesn't suit your need, change the IP address in /etc/sysconfig/network-scripts/ifcfg-eth0
Now you can reboot the V machine.

If the M machine was at runlevel 5, the V machine may start with corrupted graphics because of the different video driver of the VM.
Switch to runlevel 3 and reconfigure X with

# system-config-display --reconfig

Now you can proceed with vmware-tools installation as usual on VMs.