2011-11-30

An AWK parser for NMAP

NMAP is quite useful for identifing devices on a network, but its output is quite verbose.
Here's an AWK script that parses the output of NMAP into a csv.
It contains some euristics to extract info that might be useful to identify unknown hosts.

function print_buffer()
{
    gsub(":", "", ip)
    gsub("Service Info: ", "", info)
    gsub("\\|_ Discover OS Version over NetBIOS and SMB: ", "", info)
    gsub("Discover OS Version over NetBIOS and SMB: ", "", info)
    gsub(", ", ";", info)
    print ip","os" "hostname" "info
    ip="unknown"
    info="unknown"
    os=""
    hostname=""
}
BEGIN { ip="IP"; info="Info"; }

/Interesting ports/ { print_buffer(); ip=$4 }
/All [0-9]+ scanned/ { print_buffer(); ip=$6 }

/135\/tcp  open  msrpc[ ]+Microsoft RPC/ { if ( os == "") os="Windows NT" }
/135\/tcp  open  msrpc[ ]+Microsoft Windows RPC/ { if ( os == "") os="Windows NT" }
/IIS webserver 3.0/ { os="Windows NT" }
/IIS webserver 4.0/ { os="Windows NT" }
/IIS webserver 5.0/ { os="Windows 2000" }
/IIS webserver 5.1/ { os="Windows XP" }
/IIS webserver 6.0/ { os="Windows 2003" }
/Windows 98 netbios-ssn/ { os="Windows 98" }
/SMB: Windows Vista/ { os="Windows Vista" }
/IIS httpd 3/ { os="Windows NT" }
/Windows XP microsoft-ds/ { os="Windows XP" }
/SMB: Windows XP/ { os="Windows XP" }
/Windows 2000 microsoft-ds/ { os="Windows 2000" }
/Windows 2003 microsoft-ds/ { os="Windows 2003" }
/VMware Authentication Daemon/ { os="VMWare ESX" }
/Apache httpd/ { if ( os == "") os=$7 }
/HP webadmin/ { os="HP" }

/ePolicy Orchestrator \(Computername:/ { hostname=$9 }
/Ultr@VNC \(Name/ { hostname=$6 }
/WinVNC \(Server/ { hostname=$6 }
/Lotus Domino server \(CN=/ { hostname=$7 }

/Service Info/ { info=$0 }
/Discover OS/ { info=info" "$0 }
/telnet/ { info=info" "$4 }
/ ssh / { info=info" unix" }

END { print_buffer() }


Here's a sample output:

xx.xx.xx.10,Windows 2000  OS: Windows |  Windows 2000
xx.xx.xx.12,Windows NT gandalf; OS: Windows |  Windows NT 4.0
xx.xx.xx.13,Windows NT  OS: Windows |  Windows NT 4.0
xx.xx.xx.26,Windows NT bilbo; Host: bilbo.contoso.local; OS: Windows |  Windows NT 4.0
xx.xx.xx.30,  Device: printer
xx.xx.xx.31,  Devices: print server;printer
xx.xx.xx.66,Windows 2000  OS: Windows |  Windows 2000
xx.xx.xx.70,Windows NT aragorn; OS: Windows |  Windows NT 4.0
xx.xx.xx.80,Windows 2000  Host: saruman; OS: Windows |  Windows 2000
xx.xx.xx.82,Windows 2003 pippin; OS: Windows |  Windows Server 2003 3790 Service Pack 2
xx.xx.xx.153,  Device: switch

2011-11-27

Boot disk transplant

Bare hands, MacGyver-style, boot disk transplant.

I've used a network attached linux machine as a temporary storage, but a local USB disk will do as fine.
The new disk should be as least as big as the old one.

First take note of the original disk configuration.
You will need the partition table:

# fdisk -l

Disk /dev/sda: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          13      104391   83  Linux
/dev/sda2              14        1305    10377990   8e  Linux LVM

The volume group configuration (note the PE and the LE sizes)

# vgdisplay -v
    Finding all volume groups
  /dev/hda: open failed: No medium found
    Finding volume group "VolGroup00"
  --- Volume group ---
  VG Name               VolGroup00
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  3
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               2
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               9.88 GB
  PE Size               32.00 MB
  Total PE              316
  Alloc PE / Size       316 / 9.88 GB
  Free  PE / Size       0 / 0
  VG UUID               CJqD4r-kRQM-Cj9e-XwOr-6dn8-TkPH-MfTTyZ

  --- Logical volume ---
  LV Name                /dev/VolGroup00/LogVol00
  VG Name                VolGroup00
  LV UUID                d4iToX-8m0c-jf9s-sMMN-VDpO-brE8-4kWhf2
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                7.91 GB
  Current LE             253
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0

  --- Logical volume ---
  LV Name                /dev/VolGroup00/LogVol01
  VG Name                VolGroup00
  LV UUID                kEzo7d-AHuv-xiPD-3Qbo-v8yY-6w1D-DFMIwN
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                1.97 GB
  Current LE             63
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:1

  --- Physical volumes ---
  PV Name               /dev/sda2
  PV UUID               7bPvQd-H8yE-v00N-asPT-WQ92-6oCS-bBh6nj
  PV Status             allocatable
  Total PE / Free PE    316 / 0

And the fstab:

# cat /mnt/sysimage/etc/fstab
/dev/VolGroup00/LogVol00 /                       ext3    defaults        1 1
LABEL=/boot             /boot                   ext3    defaults        1 2
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
/dev/VolGroup00/LogVol01 swap                    swap    defaults        0 0

you will have to rebuild the logical volumes that belong to the transplanting disk

Boot the machine in rescue mode to do a full backup

boot: linux rescue

Start the network interface and do not mount any filesystem.
Backup the filesystems on /dev/sda and send them to the second machine via network.

#gzip -1 </dev/VolGroup00/LogVol00 | ssh user@secondmachine "cat >root.gz"
#gzip -1 </dev/sda1 | ssh user@secondmachine "cat >boot.gz"

Turn off the first machine, remove the old hd and put the new one.
Reboot again into linux rescue, with networking and without searching for existing installations.

Use fdisk to rebuild the same layout, with identical partitions, cylinder boundaries and partition Id (8e for LVM)
Now rebuild the LVM

#lvm pvcreate /dev/sda2

Note the PE size:

#lvm vgcreate -s 32M VolGroup00 /dev/sda2

and the LV size

#lvm lvcreate -l 253 -n LogVol00 VolGroup00
#lvm lvcreate -l 63 -n LogVol01 VolGroup00

Rebuild the swap partition

#mkswap /dev/VolGroup00/LogVol01

and write back the previous backups:

#ssh user@secondmachine "cat boot.gz" | gzip -d > /dev/sda1
#ssh user@secondmachine "cat root.gz" | gzip -d > /dev/VolGroup00/LogVol00

Reboot once again in rescue mode

#chroot /mnt/sysimage 
#grub-install /dev/sda

And finally reboot with your new disk.
Note that if the new disk is bigger than the old one, you will have some unpartitioned space left.
You can partition it with fdisk, pvcreate, and add it to LVM.

The broken LVM boot mirror

I was experimenting with LVM mirroring of the root partition and I've discovered that it won't fail gracefully.
After successfully setting up a mirror of LogVol00 to a second hd, I've rebooted the machine without the second hard disk.
I was expecting a normal boot from the first hd, which was the one with the grub and the boot partition.
I've got a kernel panic instead.
To recover the system I had to boot in rescue mode

boot: linux rescue

The rescue mode won't find any linux partition even if asked to search for them.
So back to the shell:

lvm pvscan

The system complained about an unknown device within VolGroup00

lvm vgscan
lvm lvscan

All logical volumes were marked as inactive

So, force the volume group activation

lvm vgchange -ay --partial VolGroup00

Now we can remove the missing mirror copy from the volume group

lvm vgreduce --removemissing --force VolGroup00

and the mirror logical volume will be converted back to linear.
Reboot.

2011-11-25

How to rename the root volume group

This was tested on a RedHat 5.4
Boot the machine with installation CD into rescue mode

boot: linux rescue

at the request of mounting the existing installation, choose skip
Activate the volume group with

#lvm pvscan
#lvm vgscan

Now you can rename the volume group

#lvm vgrename VolGroup00 VolGroup99

Now you must change all the references to VolGroup00 on the filesytem to the new volume group name.
Activate the new volume group

#lvm vgchange -ay VolGroup99

Mount the root and the boot filesystems

#mkdir -p /mnt/sysimage
#mount /dev/VolGroup99/LogVol00 /mnt/sysimage

Mount the boot filesystems (note that the device name may vary according to your actual boot device)

#mount /dev/sda1 /mnt/sysimage/boot

Edit the fstab to look like this

#vi /mnt/sysimage/etc/fstab



/dev/VolGroup99/LogVol00 /                      ext3    defaults        1 1
LABEL=/boot             /boot                   ext3    defaults        1 2
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
/dev/VolGroup99/LogVol01 swap                   swap    defaults        0 0

and the grub configuration

#vi /mnt/sysimage/boot/grub/grub.conf




# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /boot/, eg.
#          root (hd0,0)
#          kernel /vmlinuz-version ro root=/dev/VolGroup99/LogVol00
#          initrd /initrd-version.img
#boot=/dev/sda
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title Red Hat Enterprise Linux Server (2.6.18-164.el5)
        root (hd0,0)
        kernel /vmlinuz-2.6.18-164.el5 ro root=/dev/VolGroup99/LogVol00 rhgb quiet
        initrd /initrd-2.6.18-164.el5.img

Now reboot once again in rescue mode.
This time choose to mount the filesystem onto /mnt/sysimage (not in read only)
Finally, rebuild the initrd image, as it yet contains the old fstab, matching your kernel version

#chroot /mnt/sysimage
#mkinitrd -v -f /boot/initrd-2.6.18-164.el5.img 2.6.18-164.el5

Reboot into your newly renamed system.

Note: if you don't reboot into rescue mode before rebuilding the initial ram disk, the mkinitrd will fail as it won't find the new /dev/VolGroup99 pseudo files

2011-11-06

The most important field of web forms

How many times, while registering on some web site, you had to enter your email address twice?
What's the point? It's not a password and the field is not masked, so you can see if you typed it wrong.
Anyway, I always do a copy/paste from the first field. I guess many others do the same, thus obviating the whole check.
Even XKCD agrees...


2011-11-05

How much the sync mount option of the ext3 filesystem slows you down?

Literature says that if you want assured write consistency on your filesystem you should mount it with the sync option.
This way every write is immediately committed to the disk at the cost of a performance hit. But how much?
While working on a broader project I had the chance to test the write performance of sync mounted filesystem vs the same filesystem mounted with full caching and journaling enabled.
The filesystem was on single LUN on an enterprise class storage with 4Gbit/s fibre channel connection to the host.
Throughput was measured looking at the block-out counter of a vmstat 1 command.

Results are:
  30MB/s with sync on
250MB/s with sync off