Ubuntu Server 12.04.2 KVM Host Setup

#!/bin/bash
#KVM Virtual machine host package installation:

apt-get install ubuntu-virt-server

#set scheduler on all sd? devices

find /dev/ -name sd? -type b | cut --delimiter='/' -f3 | while read -r; do echo "deadline" > /sys/block/$REPLY/queue/scheduler; done && \
find /dev/ -name sd? -type b | cut --delimiter='/' -f3 | while read -r; do echo -n "$REPLY:";cat /sys/block/$REPLY/queue/scheduler; done

#configure default io scheduler to deadline
sed -i 's/GRUB_CMDLINE_LINUX=""/GRUB_CMDLINE_LINUX="elevator=deadline"/g' /etc/default/grub && sudo update-grub

#configure fstab mount options to relatime(assumes ext4 file systems)

sed -i -r 's/ext4(\s+)errors/ext4\1relatime,errors/g' /etc/fstab

#remount all filesystems with relatime using UUID
cat /etc/fstab | grep relatime | cut -f1 --delimiter=' ' | cut -f2 --delimiter='=' | xargs --verbose mount -o remount -U

#install NTP and configure to sync time with at least 2 local NTP servers if available

apt-get install -y ntp && service ntp stop
read -p "Enter IP/URL of primary time server:" primary_ntp_server && read -p "Enter IP/URL of backup time server:" backup_ntp_server
cat <<EOF >/etc/ntp.conf

driftfile /var/lib/ntp/ntp.drift
statistics loopstats peerstats clockstats
filegen loopstats file loopstats type day enable
filegen peerstats file peerstats type day enable
filegen clockstats file clockstats type day enable

server $primary_ntp_server iburst

server $backup_ntp_server
restrict -4 default kod notrap nomodify nopeer noquery
restrict -6 default kod notrap nomodify nopeer noquery
restrict 127.0.0.1
restrict ::1

EOF
service ntp start

#configure bridge networking with eth0 (assumes eth0 has static ip configured)

sed -i -e '/^[[:space:]]*#/!s/eth0/br0/g' -e '/^[[:space:]]*[^#]*iface br0 inet/a \
\tbridge_ports eth0 \
\tbridge_fd 9 \
\tbridge_hello 2 \
\tbridge_maxage 12 \
\tbridge_stp off'  /etc/network/interfaces

invoke-rc.d networking restart

Ubuntu Server 12.04.2 KVM Guest Setup

preparing the template image:
#Setup io scheduler and serial console for virsh
sed -i -r 's/GRUB_CMDLINE_LINUX=".*"/GRUB_CMDLINE_LINUX="elevator=deadline console=ttyS0,38400n8 console=tty0"/g' /etc/default/grub && update-grub

#Setup serial terminal console config
cat <<EOF >/etc/init/ttyS0.conf
# ttyS0 - getty
#
# This service maintains a getty on ttyS0 from the point the system is
# started until it is shut down again.

start on stopped rc RUNLEVEL=[2345]
stop on runlevel [!2345]

respawn
exec /sbin/getty -L 38400 ttyS0 vt102
EOF

#Setup fstab mount options to relatime(assumes ext4 file systems)
sed -i -r 's/ext4(\s+)errors/ext4\1relatime,errors/g' /etc/fstab


#!/bin/bash
if [[ $EUID -ne 0 ]]; then
echo "script must be run as root. run with sudo" 2>&1
exit 1
fi
old_name=`hostname`
echo "Current hostname:$old_name"
echo -n "Enter new server name:"
read new_name
echo "$new_name">/etc/hostname
hostname $new_name
sed -i "s/$old_name/$new_name/g" /etc/hosts

#regenerate udev persistent net rules on reboot
echo "Removing /etc/udev/rules.d/70-persistent-net.rules"
rm /etc/udev/rules.d/70-persistent-net.rules

echo "Regenerating ssh server keys"
rm /etc/ssh/ssh_host_*
dpkg-reconfigure openssh-server

echo "*** You should probably change the password and reboot ***"

QT5 on xUbuntu 12.04.2 LTS

The latest version of xVideoServiceThief (2.5) switched to using the QT5 library.

To get it to work, I needed to setup QT5 on a xUbuntu 12.04.2 LTS which I did by running the following commands:

sudo apt-get install python-software-properties && \
sudo apt-add-repository ppa:canonical-qt5-edgers/qt5-proper && \
sudo apt-get update && \
sudo apt-get install libqt5gui5 libqt5webkit5 libqt5script5

Let me know in the comments if this helped you or if there were any issues.

Cloudstack 4.1 on Ubuntu Server

My objective was to setup a Cloudstack 4.1 management node which will also server an NFS server for both primary and secondary storage.

The series of commands below were run on an Ubuntu Server 12.04.2(precise) system. During installation of the operating system, only the OpenSSH server was enabled in the task selection screen. I also made sure to name the server with a fully qualified domain name i.e. cloudstack.henyo.com.

#setup cloudstack source

echo "deb http://cloudstack.apt-get.eu/ubuntu precise 4.1" > /etc/apt/sources.list.d/cloudstack.list

#setup percona source

cat <<EOF >/etc/apt/sources.list.d/percona.list

deb http://repo.percona.com/apt precise main

deb-src http://repo.percona.com/apt precise main

EOF

#install the cloudstack apt-key

wget -O - http://cloudstack.apt-get.eu/release.asc|apt-key add -

#install stuff
apt-get update && \
apt-get install -y --force-yes openntpd nfs-kernel-server cloudstack-management percona-server-server-5.5 percona-server-client-5.5

cat <<EOF >/etc/mysql/conf.d/cloudstack.cnf

[mysqld]

innodb_rollback_on_timeout=1

innodb_lock_wait_timeout=600

max_connections=350

log-bin=mysql-bin

binlog-format = 'ROW'

EOF

service mysql restart

#setup nfs

mkdir -p /export/primary /export/secondary /var/primary /var/secondary

cat <<EOF >>/etc/fstab

/var/primary    /export/primary   none    bind  0  0

/var/secondary  /export/secondary none    bind  0  0

EOF

mount --bind /var/primary/ /export/primary/

mount --bind /var/secondary/ /export/secondary/

echo "/export         *(rw,async,no_root_squash,no_subtree_check)" >> /etc/exports

exportfs -a

service nfs-kernel-server restart

#setup cloudstack database

cloudstack-setup-databases cloud:test@localhost --deploy-as=root:root

#setup management server

echo 'Defaults:cloud !requiretty' >> /etc/sudoers.d/cloudstack

cloudstack-setup-management

reboot

I hope you found the post useful. You can subscribe via email or subscribe via a feed reader to get relevant updates from this blog. Have a nice day.

Disabling BBU Auto Learn with megacli

We are currently facing performance problems with mysql and I remember reading about RAID BBU Learning causing huge write performance drops. So I wanted to check if the RAID controller on our Master database had this configured.

Step#1: Find out what is the brand/model of the RAID controller installed on the server

I can of course ask accounting to pull-up the delivery receipt to find the brand/model but where is the fun in that. Googling led me to the following commands:


sudo lspci | grep -i raid
04:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 2108 [Liberator] (rev 04)

sudo lshw -class storage

description: RAID bus controller

       product: MegaRAID SAS 2108 [Liberator]

       vendor: LSI Logic / Symbios Logic

       physical id: 0

       bus info: pci@0000:04:00.0

       logical name: scsi4

       version: 04

       width: 64 bits

       clock: 33MHz

       capabilities: storage pm pciexpress vpd msi msix bus_master cap_list rom

       configuration: driver=megaraid_sas latency=0

       resources: irq:26 ioport:d800(size=256) memory:fae7c000-fae7ffff memory:faec0000-faefffff memory:fae80000-faebffff

Step#2: Install the megacli package to be able to query the raid card for its status

Downloaded the megacli package from the LSI website. But they only provide RPMs so I had to convert them with:

sudo alien -k MegaCli-8.07.06-1.noarch.rpm

Then installed with:

sudo dpkg -i megacli_8.07.06-1_all.deb

To find out where the files got installed do:

sudo dpkg -c megacli_8.07.06-1_all.deb

Step#3: Use megacli to probe for BBU status and information

Running the command results to something unexpected:

./MegaCli64 -adpCount

Controller Count: 0.

Exit Code: 0x00

megacli cant find the raid adapter. It seems that megacli has an issue with kernels >= 3.0 and can be remedied with:

sudo setarch x86_64 --uname-2.6 /opt/MegaRAID/MegaCli/MegaCli64 -adpCount

Controller Count: 1.

Exit Code: 0x01

So to find out about the BBU:

sudo setarch x86_64 --uname-2.6 /opt/MegaRAID/MegaCli/MegaCli64 -AdpBbuCmd -GetBbuStatus -aALL

BBU status for Adapter: 0

BatteryType: iBBU

Voltage: 3972 mV

Current: 0 mA

Temperature: 24 C

Battery State: Optimal

BBU Firmware Status:

  Charging Status              : None

  Voltage                                 : OK

  Temperature                             : OK

  Learn Cycle Requested                  : No

  Learn Cycle Active                      : No

  Learn Cycle Status                      : OK

  Learn Cycle Timeout                     : No

  I2c Errors Detected                     : No

  Battery Pack Missing                    : No

  Battery Replacement required            : No

  Remaining Capacity Low                  : No

  Periodic Learn Required                 : No

  Transparent Learn                       : No

  No space to cache offload               : No

  Pack is about to fail & should be replaced : No

  Cache Offload premium feature required  : No

  Module microcode update required        : No

GasGuageStatus:

  Fully Discharged        : No

  Fully Charged           : Yes

  Discharging             : Yes

  Initialized             : Yes

  Remaining Time Alarm    : No

  Discharge Terminated    : No

  Over Temperature        : No

  Charging Terminated     : No

  Over Charged            : No

  Relative State of Charge: 97 %

  Charger System State: 49168

  Charger System Ctrl: 0

  Charging current: 0 mA

  Absolute state of charge: 53 %

  Max Error: 2 %

Exit Code: 0x00

sudo setarch x86_64 --uname-2.6 /opt/MegaRAID/MegaCli/MegaCli64 -AdpBbuCmd -GetBbuCapacityInfo -aALL

BBU Capacity Info for Adapter: 0

  Relative State of Charge: 97 %

  Absolute State of charge: 53 %

  Remaining Capacity: 641 mAh

  Full Charge Capacity: 664 mAh

  Run time to empty: Battery is not being discharged.  

  Average time to empty: Battery is not being discharged.  

  Estimated Time to full recharge: Battery is not being charged.  

  Cycle Count: 37

Max Error = 2 %

Remaining Capacity Alarm = 120 mAh

Remining Time Alarm = 10 Min

Exit Code: 0x00

sudo setarch x86_64 --uname-2.6 /opt/MegaRAID/MegaCli/MegaCli64 -AdpBbuCmd -GetBbuProperties -aALL

BBU Properties for Adapter: 0

  Auto Learn Period: 30 Days

  Next Learn time: Sun Feb 17 19:37:09 2013

  Learn Delay Interval:0 Hours

  Auto-Learn Mode: Enabled

Exit Code: 0x00

Auto learn mode should be disabled and scheduled during off-peak time.

#!/bin/bash

TMPFILE=$(mktemp -p /tmp bbu.relearn.XXXXXXXXXX) || exit 1

echo "autoLearnMode=1" > $TMPFILE

setarch x86_64 --uname-2.6 /opt/MegaRAID/MegaCli/MegaCli64 -AdpBbuCmd -SetBbuProperties -f $TMPFILE -aALL

sudo setarch x86_64 --uname-2.6 /opt/MegaRAID/MegaCli/MegaCli64 -AdpBbuCmd -GetBbuProperties -aALL

rm $TMPFILE

script for safe write back:

#!/bin/bash
sudo setarch x86_64 --uname-2.6 /opt/MegaRAID/MegaCli/MegaCli64 -LDSetProp ADRA -Lall -aALL
sudo setarch x86_64 --uname-2.6 /opt/MegaRAID/MegaCli/MegaCli64 -LDSetProp -Cached -Lall -aALL
sudo setarch x86_64 --uname-2.6 /opt/MegaRAID/MegaCli/MegaCli64 -LDSetProp DisDskCache -Lall -aALL
sudo setarch x86_64 --uname-2.6 /opt/MegaRAID/MegaCli/MegaCli64 -LDSetProp NoCachedBadBBU -Lall -aALL
sudo setarch x86_64 --uname-2.6 /opt/MegaRAID/MegaCli/MegaCli64 -LDSetProp WB -Lall -aALL
sudo setarch x86_64 --uname-2.6 /opt/MegaRAID/MegaCli/MegaCli64 -LDInfo -Lall -aALL


script to force write back without BBU protection:

#!/bin/bash
sudo setarch x86_64 --uname-2.6 /opt/MegaRAID/MegaCli/MegaCli64 -LDSetProp ADRA -Lall -aALL
sudo setarch x86_64 --uname-2.6 /opt/MegaRAID/MegaCli/MegaCli64 -LDSetProp -Cached -Lall -aALL
sudo setarch x86_64 --uname-2.6 /opt/MegaRAID/MegaCli/MegaCli64 -LDSetProp DisDskCache -Lall -aALL
sudo setarch x86_64 --uname-2.6 /opt/MegaRAID/MegaCli/MegaCli64 -LDSetProp CachedBadBBU -Lall -aALL
sudo setarch x86_64 --uname-2.6 /opt/MegaRAID/MegaCli/MegaCli64 -LDSetProp WB -Lall -aALL
sudo setarch x86_64 --uname-2.6 /opt/MegaRAID/MegaCli/MegaCli64 -LDInfo -Lall -aALL


Useful Links:
http://linux.dell.com/files/whitepapers/solaris/Managing_PERC6_0714.pdf
http://hwraid.le-vert.net/wiki/LSIMegaRAIDSAS
http://yo61.com/dell-drac-bbu-auto-learn-tests-kill-disk-performance.html


I hope you found the post useful. You can subscribe via email or subscribe via a feed reader to get relevant updates from this blog. Have a nice day.

Transferring a huge file over a LAN

We need to transfer several huge files from an old server to a new server. I plan to use netcat to do a raw transfer followed by rsync to ensure data integrity. I'll report back on the results.


On the destination server:
nc -l 1234 > ubuntu-12.04.1-server-i386.iso


On the source server:
time cat ubuntu-12.04.1-server-i386.iso | nc $DESTINATION_IP 1234


On the source server:

cat >rsyncd.conf <<EOF
#rsyncd.conf
[temp]
path=/var/lib/libvirt/images/
read only = yes
list = yes
use chroot = no
EOF

rsync --config=rsyncd.conf --daemon --no-detach --port=1234


On the destination server:
time rsync -av --inplace --progress rsync://apollo@192.168.7.167:1234/temp/ubuntu-12.04.1-server-i386.iso ubuntu-12.04.1-server-i386.iso

Update#1:

time cat ubuntu-12.04.1-server-i386.iso | nc 192.168.7.198 1234

real 0m58.130s
user 0m0.072s
sys 0m2.016s

ubuntu-12.04.1-server-i386.iso is 646MB in size which gives a transfer rate of 11.13MB/s which is NOT what I expected from a gigabit connection.



I hope you found the post useful. You can subscribe via email or subscribe via a feed reader to get relevant updates from this blog. Have a nice day.

Safely Rebooting a Hanging Server

When all else fails, go to the physical server console and hit:
Alt-SysRq R (wait for 21 secs)
Alt-SysRq E (wait for 21 secs)
Alt-SysRq I (wait for 21 secs)
Alt-SysRq S (wait for 21 secs)
Alt-SysRq U (wait for 21 secs)
Alt-SysRq B

Then cross your fingers as the server should reboot

I hope you found the post useful. You can subscribe via email or subscribe via a feed reader to get relevant updates from this blog. Have a nice day.

Trying Ubuntu 12.04LTS Precise Pangolin

I migrated my desktop from Ubuntu 10.04LTS to Ubuntu 12.04LTS.

A few things I needed to add after installation:

• Chromium as my default browser
• adobe-flash to be able to play flash games on facebook
• openjdk-6-jdk for work stuff
• icedtea-plugin to be able to run charting applet on citisec
• pidgin for messaging
• git for more work
• smartgit for routine pushing and comitting to git repos
• smartsvn for routine pushing and comitting to git repos
• STS for work stuff
• DB Visualizer for work stuff
• Virtual Box for desktop virtualization
• Dia for diagramming
• synergy to share one keyboard between my desktop and laptop
• dropbox for personal file sharing
• pencil for mockups
• smplayer for watching videos
• vlc for videos that do not work with smplayer
• git gui for simple git commits
• myqsl-server for development work
• mysql workbench for managing mysql database
• postgresql for development work
• pgadmin for managing postgres database

Should I be replacing stuff on the list with something else? What other stuff have I forgotten to install?

I hope you found the post useful. You can subscribe via email or subscribe via a feed reader to get relevant updates from this blog. Have a nice day.

Protecting ProxmoxVE VMs with IPCop on KVM

I found out a few weeks ago that protecting openvz virtual machines running ubuntu was a bit problematic as I was unable to run UFW on them.

I was able to setup IPCop v2.04 on KVM which will act as a virtual firewall for all of the virtual machines in our ProxmoxVE 2.0 cluster

I hope you found the post useful. You can subscribe via email or subscribe via a feed reader to get relevant updates from this blog. Have a nice day.

Building a good Software Startup

If I were to build a good software startup today, I'd run with the following ideas.

Hire only good people who get things DONE.
Even if they are more expensive in terms of compensation.
Quality trumps numbers. No doubt about it.

Development Team

• 1 Lead Developer
• 1 Senior Developer
• 2 Junior Developers
• 2 Business Analyst/QA

Operations Team

• 1 Lead Systems Admin
• 1 Senior System Admin
• 6 Junior System Admins

Give your team a quiet working environment with ergonomic furniture and equipment

Give your team good internet connection

• the faster they can download their installers, the earlier they can start working on code
• the faster they can look up the solution on google or stackoverflow, the faster they can proceed to the next problem

Give your team good computer hardware

• at least dual core processors with virtualization support
• at least 8GB memory
• at least 2 hard drives for mirror setup
• at least 2 screens

Make sure to have enough environments to test, deploy and support the system properly.

• Development Environment
• Staging Environment
• Support Environment
• Production Environment

Make use of Free or Open Source systems to manage the development process.
Buy a commercial product only if you can't find a solution that fits.

Project/Issue Tracking with Source Code Management Options:

• Indefero
• Gitorious
• Gitlab
• Redmine+gitolite

Continuous Integration Server Options:

• Jenkins
• Team City

Code Quality:

• Sonar

Setup google apps for for email, calendar and other stuff

As an example, here is how I would set it up on three physical servers:

Server#1 running ProxmoxVE
- Firewall/Proxy Server/Package Cache(apt,yum,mvn) VM
- Redmine+gitolite VM
- Jenkins VM
- Sonar VM
- NFS/Samba Server VM

Server#2 running ProxmoxVE
- Development Environment VM
- Staging Environment VM
- Support Environment VM

Server#3 for storage of backups
- nexentastor or freenas or centos+zfsonlinux

I hope you found the post useful. You can subscribe via email or subscribe via a feed reader to get relevant updates from this blog. Have a nice day.

ProxmoxVE 2: Configuring DRBD

I am running ProxmoxVE 2.0-35/d07f49c3 on two nodes connected using a cross cable on gigabit nics.
host1 ip: 10.0.10.201
host2 ip: 10.0.10.202

Check if drbd module is available

modprobe -l | grep drbd

Check if drbd module is loaded

lsmod | grep drbd

If not, load it with

modprobe -v drbd

Check the module version

modinfo drbd | grep version

I installed the appropriate drbd8-utils package on both nodes

dpkg -i drbd8-utils_8.3.10-0_amd64.deb

I built my own deb from source. You can find my build here.

I shrank the data volume on both of
I made use of the extents I freed up by shrinking the data volume to create an lvm volume that will be used for drbd.

lvcreate --verbose --extents 50000 --name lv4drbd pve

Next I modified /etc/drbd.d/global_common.conf so that it only contains:

global {
    usage-count no;
}
common {
    protocol C;
    syncer {
        rate 30M; 
    }
}

Next I created a new resouce file /etc/drbd.d/drbd0.res

resource drbd0 {
    protocol C;
    startup {
        wfc-timeout  0;
        degr-wfc-timeout 60;
        become-primary-on both;
    }
    net {
        cram-hmac-alg sha1;
        shared-secret "drbd0-secret";
        allow-two-primaries;
        after-sb-0pri discard-zero-changes;
        after-sb-1pri discard-secondary;
        after-sb-2pri disconnect;
    }
    on host1 {
        device /dev/drbd0;
        disk /dev/pve/lv4drbd;
        address 10.0.10.201:7780;
        meta-disk internal;
    }
    on host2 {
        device /dev/drbd0;
        disk /dev/pve/lv4drbd;
        address 10.0.10.202:7780;
        meta-disk internal;
    }
}

I added the IP address of the other host into each of the /etc/hosts files.

Next I started drbd on both nodes with

/etc/init.d/drbd start

I then initialized drbd metatdata on both nodes with

drbdadm create-md drbd0

Next I brought up the device on both nodes with

drbdadm up drbd0

I checked the status on both nodes with

drbd-overview

Next I started synchronization(should be instantaneous) from one node with

drbdadm -- --clear-bitmap new-current-uuid drbd0

Lastly, I restarted the drbd service on both nodes to enable Primary/Primary operation

/etc/init.d/drbd stop
/etc/init.d/drbd start

I hope you found the post useful. You can subscribe via email or subscribe via a feed reader to get relevant updates from this blog. Have a nice day.

ProxmoxVE 2.0: Shrinking the data volume

After installing ProxmoxVE 2.0rc1, a data volume was created which makes use of most of the space on the drive.

I wanted to carve out a separate volume for some other use so I needed to shrink this data volume:

First, I unmounted the logical volume with:

umount /var/lib/vz

Then I proceeded to shrink the file system to the minimum size possible with:

resize2fs -Mf /dev/pve/data

Then I proceeded to reduce the size of the logical volume with:

lvreduce --verbose --extents -100000 /dev/pve/data

Then I proceeded to grow the file system back to occupy the now reduced volume with:

resize2fs -f /dev/pve/data

Lastly, I remounted the file system with:

mount /dev/pve/data /var/lib/vz

I hope you found the post useful. You can subscribe via email or subscribe via a feed reader to get relevant updates from this blog. Have a nice day.

dailyLooper bash script

We needed to regenerate some data for the data warehouse so I had to run an extraction script once for each day of the month.

I wrote a simple bash script to make it easier to run a command or other scripts once for every day of a target month and year.

Code can be found below but may also be downloaded from here.

#!/bin/bash
usage(){
    echo "Usage: dailyLooper <year> <month> <script to run>"
    echo "Each date in the given month will be passed into the script in yyyy-mm-dd format"
}


CURRENT_MONTH=$(date +%m)
TARGET_YEAR=$1
TARGET_MONTH=$2

if [ "$#" -lt 3 ]; then
    usage
    exit 1
fi

if [ "$3" = "" ]; then
    SCRIPT="echo"
else
    SCRIPT=$3    
fi

#returns the date yesterday if target month is the current month
#otherwise returns the last date for the target month
lastDayOfMonth(){
    local _year=$1
    local _month=$2   
    if [ $_month -lt $CURRENT_MONTH ]; then
        echo $(date -d "${TARGET_YEAR}-${TARGET_MONTH}-01 + 1 month - 1 day" +%d)
    else
        echo $(date -d "yesterday" +%d)
    fi    
}


main(){
    local _lastDay=$(lastDayOfMonth $TARGET_YEAR $TARGET_MONTH)
    i=1
    while [ "$i" -le $_lastDay ]
    do
        local _date=$(date -d "${TARGET_YEAR}-${TARGET_MONTH}-${i}" +%Y-%m-%d)
        $SCRIPT $_date
        ((i++))
    done #end day of month loop
}

main

I hope you found the post useful. You can subscribe via email or subscribe via a feed reader to get relevant updates from this blog. Have a nice day.

Cheap and Safe File Storage on Linux

I needed to setup a Samba file server using off the shelf desktop components.
It will be used in a small office with 5-10 clients.

Here are the relevant hardware specs:
Processor: i7
Memory: 16GB
Drives: 4 x 1TB sata drives
OS Installed: Ubuntu Sever 10.04LTS

The primary requirement is for the files to be fairly safe i.e. if the server crashes in the middle of saving a file then on restart, the file should return to the consistent state before the save operation. Performance is secondary to safety.

My initial plan was to use one drive for the OS and configure 3 drives in an mdraid level 5. After some research, I came across several references that state that mdraid level 10 works on odd numbered drives. It will work on 3 drives and even on 2. This looked like something I should try. I then had to decide what filesystem to use. I initially wanted to use xfs as I've heard and read a lot of good things about it.

After studying how to setup both mdraid and xfs, I started a thread on the linux-raid mailing list asking for best practice guidelines and guidance on using xfs with mdraid 10n2.

Several days of additional research guided by the comments on the mailing list thread resulted in a lot of learnings for me:

• raid10n2 really works on 3 drives, I had my doubts at first but now I finally get how it can work with just three drives
• drive sector size can be either 512b or 4k which can lead to mis-aligned partitions and have a negative impact on performance
• ext4 provides the option: data=journal which makes it safer than xfs at the cost of some performance

The table below shows the average of the benchmark results I got. Yes, I know bonnie++ only tests sequential write/rewrite/read and some file operations and is not necessarily representative of the workload but IMHO, it is better than nothing. This was how things were setup for the benchmark:

• a 50GB primary partition was created on each device located 1GB from the start of the disk for partition alignment
• the md raid device is created on top of the primary partition created on each device
• the chunk was set to 64k
• the results below are for raid5 and raid10n2 only but I also tried raid10f2 and raid10n2
• raid 10f2 had better read performance but slower write compared to raid 10n2
• raid 10o2 performed quite closely to raid 10n2
• the benchmark was run three times and the average was taken

Bonnie 1.96		Sequential Output				Sequential Input		Random Seeks
64k chunk	Size	Block		Rewrite		Block
ext4/mdraid		K/sec	% CPU	K/sec	% CPU	K/sec	% CPU	/sec	% CPU
raid5 data=ordered	32G	144430	11	68725	9.67	249364	21	280.33	9.33
raid5 data=journal	32G	25077	4.67	21605	5	267283	22.33	293.67	6
raid10n2 data=ordered	32G	137365	8	69940	9	209180	16	365.03	11.67
raid10n2 data=journal	32G	56249	6	37491	6	206548	16	389.00	8

For those interested, you can find/get the script I used here.

I modified this script to run my tests.

You can find the raw results here.

You can also find the debug info here.

I am considering re-running the tests but this time with iozone instead of bonnie++ so that I can get scores for random read and write.

Next, I plan to do pull-the-plug testing on ext4,data=journal on top of raid10n2 while doing streaming writes of a huge file over samba. I plan to use 2 different video files greater than 3GB. First I will copy file1 to the samba share and time it. I will then compute a checksum for file1 both locally and on the samba server. The computed checksum should be equal. Next, I will compute a checksum for file2 locally. It should be different from the value computed for file1. I will then move file1 to another directory and rename file2 to the the same name as file1. I will then attempt to copy the new file1 to the samba server. Halfway into the copy, I will pull the plug of the samba server. Next I will restart the samba server and recompute the checksum of file1. It should still be equal to the computed copy before.

If the system behaves as expected, I will go with ext4,data=journal on top of raid10n2.


I hope you found the post useful. You can subscribe via email or subscribe via a feed reader to get relevant updates from this blog. Have a nice day.

Check disk write-caching

I'm investigating the performance of file systems and found myself needing to check if write-caching was enabled for my disk.

hdparm -W /dev/sda

To turn off write-caching, do

hdparm -W0 /dev/sda

I hope you found the post useful. You can subscribe via email or subscribe via a feed reader to get relevant updates from this blog. Have a nice day.

ProxmoxVE 2.0rc1 Notes:Setup

This is Part#1 of a series of posts on my experience with Proxmox 2.0rc1.

I was tasked to look for a suitable solution to build an internal cloud/virtualization infrastructure and I've decided to spend some time with ProxmoxVE 2.0rc1.

The list below is the hardware available to me:
2 desktop computers
Intel i3-530
4GB DDR3
1TB Hard drive

Things I want to try:
- benchmark disk/file IO performance of host system
- benchmark dis/file IO performance of guest system
- openvz live migration
- kvm live migration
- HA storage for the VM using DRBD
- ProxmoxVE HA i.e. guest starting on secondary node when primary node goes down
- openvz ubuntu server 10.04LTS guest running tomcat6 on sun-java6-jdk
- kvm ubuntu server 10.04LTS guest running tomcat6 on sun-java6-jdk

Installing ProxmoxVE 2.0rc1 on the hardware was fairly straight forward.
The installer took over the whole 1TB drive and partitioned it on its own.

running fdisk -l /dev/sda && df -Th revealed:
- /dev/sda1 is 495MB ext3 mounted on /boot
- dev/sda2 is 999.5GB LVM

running pvscan && vgscan revealed:
- /dev/sda2 was configured as an lvm2 physcial volume
- a volume group named pve was created by proxmox

running lvscan && mount revealed:
- a 4GB /dev/pve/swap volume was created for swap
- a 96GB /dev/pve/root ext3 volume mounted as the root file system
- a 899GB /dev/pve/data ext3 volume mounted on /var/lib/vz

Next I updated ProxmoxVE 2.0rc1 which is Debian based.
The host is behind a proxy server so to update I ran:

export http_proxy=http://10.0.0.1:3128

apt-get update && apt-get -y upgrade

I proceeded to install some useful tools:
apt-get install screen dstat

Next, I configured NTP to sync to a time server on the local network

Next I downloaded an openvz template for ubuntu 10.04LTS and proceeded to create and run a guest.

I ran across an annoying bug while accessing the guest using the console. Tab completion was not working which made it difficult to work on the web console. I found a work around and made a separate post for the web console tab fix

Stay tuned for my next post in this series: Setting up DRBD backed storage even with just one hard disk

I hope you found the post useful. You can subscribe via email or subscribe via a feed reader to get relevant updates from this blog. Have a nice day.

ProxmoxVE 2.0rc1 web console tab fix

I ran across an annoying bug while accessing the guest using the console. Tab completion was not working which made it difficult to work on the web console. I found a work around which modifies /usr/share/pve-manager/ext4/pvemanagerlib.js with:

cp /usr/share/pve-manager/ext4/pvemanagerlib.js /root/pvemanagerlib.js.bak

cat /usr/share/pve-manager/ext4/pvemanagerlib.js | \
sed "s/'Show Controls', value: 'No'/'Show Controls', value: 'Yes'/" \
> /usr/share/pve-manager/ext4/pvemanagerlib.js


I hope you found the post useful. You can subscribe via email or subscribe via a feed reader to get relevant updates from this blog. Have a nice day.

Mount a Proxmox KVM raw image in an LVM over DRBD

Hardware:
2 desktop computers
Intel i3-530
4GB DDR3
1TB Hard drive

Setup:
- 2 node cluster of ProxmoxVE 2.0-35/d07f49c3(rc1)
- LVM volume group named drbd_storage created on top of a drbd block device
- Installed Ubuntu Server 10.04LTS in a kvm container named ubuntu-kvm-template
- proxmox created a logical volume in drbd_storage named vm-108-disk-1

I want to mount the disk so that I can check something out. Apparently it is not that simple.

I eventually figured it out thru trial and error of suggestions across serveral sources.

losetup /dev/loop0 /dev/drbd_storage/vm-108-disk-1
fdisk -l /dev/loop0
apt-get install kpartx
kpartx -av /dev/loop0
pvscan
vgscan
lvscan
vgchange -ay
lvscan
mkdir -p /tmp/108
mount /dev/ubuntu-kvm-template/root /tmp/108
umount /mnt/108
lvchange -an /dev/ubuntu-kvm-template/*
kpartx -dv /dev/loop0
losetup -d /dev/loop0


I hope you found the post useful. You can subscribe via email or subscribe via a feed reader to get relevant updates from this blog. Have a nice day.

Finding the largest sized directory

One of our servers keeps running out of disk space. To address this, we needed to find out which folders had the largest size.

On ubuntu server 10.04LTS, this is how we found the directories with the greatest size:

du -m /var/log | sort -n -r | head -n 10


I hope you found the post useful. You can subscribe via email or subscribe via a feed reader to get relevant updates from this blog. Have a nice day.

Creating a Latin1 PostgreSQL database in Ubuntu 10.04LTS

I've encountered this problem several times already and I've finally stumbled upon a solution.

The Problem:

We were tasked to move a postgresql database from an centos based server to a new ubuntu based server. The database was in latin1 encoding.

On the new ubuntu based server, creating a database with latin1 encoding results in an error.
Command: createdb -E LATIN1 --template template0 trial
Result:
createdb: database creation failed: ERROR:  encoding LATIN1 does not match locale en_PH.utf8
DETAIL: The chosen LC_CTYPE setting requires encoding UTF8

The Fix:

First we needed to modify /var/lib/locales/supported.d/local and add the line:
en_PH.ISO-8859-1 ISO-8859-1

Next we had to regenerate the locale files with:
sudo dpkg-reconfigure locales

Afterwards we were able to create a latin1 database with:
createdb -E LATIN1 --template template0 --locale en_PH.ISO-8859-1 trial

I hope you found the post useful. You can subscribe via email or subscribe via a feed reader to get relevant updates from this blog. Have a nice day.