Ubuntu Server 12.04.2 KVM Host Setup

#!/bin/bash
#KVM Virtual machine host package installation:

apt-get install ubuntu-virt-server

#set scheduler on all sd? devices

find /dev/ -name sd? -type b | cut --delimiter='/' -f3 | while read -r; do echo "deadline" > /sys/block/$REPLY/queue/scheduler; done && \
find /dev/ -name sd? -type b | cut --delimiter='/' -f3 | while read -r; do echo -n "$REPLY:";cat /sys/block/$REPLY/queue/scheduler; done

#configure default io scheduler to deadline
sed -i 's/GRUB_CMDLINE_LINUX=""/GRUB_CMDLINE_LINUX="elevator=deadline"/g' /etc/default/grub && sudo update-grub

#configure fstab mount options to relatime(assumes ext4 file systems)

sed -i -r 's/ext4(\s+)errors/ext4\1relatime,errors/g' /etc/fstab

#remount all filesystems with relatime using UUID
cat /etc/fstab | grep relatime | cut -f1 --delimiter=' ' | cut -f2 --delimiter='=' | xargs --verbose mount -o remount -U

#install NTP and configure to sync time with at least 2 local NTP servers if available

apt-get install -y ntp && service ntp stop
read -p "Enter IP/URL of primary time server:" primary_ntp_server && read -p "Enter IP/URL of backup time server:" backup_ntp_server
cat <<EOF >/etc/ntp.conf

driftfile /var/lib/ntp/ntp.drift
statistics loopstats peerstats clockstats
filegen loopstats file loopstats type day enable
filegen peerstats file peerstats type day enable
filegen clockstats file clockstats type day enable

server $primary_ntp_server iburst

server $backup_ntp_server
restrict -4 default kod notrap nomodify nopeer noquery
restrict -6 default kod notrap nomodify nopeer noquery
restrict 127.0.0.1
restrict ::1

EOF
service ntp start

#configure bridge networking with eth0 (assumes eth0 has static ip configured)

sed -i -e '/^[[:space:]]*#/!s/eth0/br0/g' -e '/^[[:space:]]*[^#]*iface br0 inet/a \
\tbridge_ports eth0 \
\tbridge_fd 9 \
\tbridge_hello 2 \
\tbridge_maxage 12 \
\tbridge_stp off'  /etc/network/interfaces

invoke-rc.d networking restart

Ubuntu Server 12.04.2 KVM Guest Setup

preparing the template image:
#Setup io scheduler and serial console for virsh
sed -i -r 's/GRUB_CMDLINE_LINUX=".*"/GRUB_CMDLINE_LINUX="elevator=deadline console=ttyS0,38400n8 console=tty0"/g' /etc/default/grub && update-grub

#Setup serial terminal console config
cat <<EOF >/etc/init/ttyS0.conf
# ttyS0 - getty
#
# This service maintains a getty on ttyS0 from the point the system is
# started until it is shut down again.

start on stopped rc RUNLEVEL=[2345]
stop on runlevel [!2345]

respawn
exec /sbin/getty -L 38400 ttyS0 vt102
EOF

#Setup fstab mount options to relatime(assumes ext4 file systems)
sed -i -r 's/ext4(\s+)errors/ext4\1relatime,errors/g' /etc/fstab


#!/bin/bash
if [[ $EUID -ne 0 ]]; then
echo "script must be run as root. run with sudo" 2>&1
exit 1
fi
old_name=`hostname`
echo "Current hostname:$old_name"
echo -n "Enter new server name:"
read new_name
echo "$new_name">/etc/hostname
hostname $new_name
sed -i "s/$old_name/$new_name/g" /etc/hosts

#regenerate udev persistent net rules on reboot
echo "Removing /etc/udev/rules.d/70-persistent-net.rules"
rm /etc/udev/rules.d/70-persistent-net.rules

echo "Regenerating ssh server keys"
rm /etc/ssh/ssh_host_*
dpkg-reconfigure openssh-server

echo "*** You should probably change the password and reboot ***"

QT5 on xUbuntu 12.04.2 LTS

The latest version of xVideoServiceThief (2.5) switched to using the QT5 library.

To get it to work, I needed to setup QT5 on a xUbuntu 12.04.2 LTS which I did by running the following commands:

sudo apt-get install python-software-properties && \
sudo apt-add-repository ppa:canonical-qt5-edgers/qt5-proper && \
sudo apt-get update && \
sudo apt-get install libqt5gui5 libqt5webkit5 libqt5script5

Let me know in the comments if this helped you or if there were any issues.

Cloudstack 4.1 on Ubuntu Server

My objective was to setup a Cloudstack 4.1 management node which will also server an NFS server for both primary and secondary storage.

The series of commands below were run on an Ubuntu Server 12.04.2(precise) system. During installation of the operating system, only the OpenSSH server was enabled in the task selection screen. I also made sure to name the server with a fully qualified domain name i.e. cloudstack.henyo.com.

#setup cloudstack source

echo "deb http://cloudstack.apt-get.eu/ubuntu precise 4.1" > /etc/apt/sources.list.d/cloudstack.list

#setup percona source

cat <<EOF >/etc/apt/sources.list.d/percona.list

deb http://repo.percona.com/apt precise main

deb-src http://repo.percona.com/apt precise main

EOF

#install the cloudstack apt-key

wget -O - http://cloudstack.apt-get.eu/release.asc|apt-key add -

#install stuff
apt-get update && \
apt-get install -y --force-yes openntpd nfs-kernel-server cloudstack-management percona-server-server-5.5 percona-server-client-5.5

cat <<EOF >/etc/mysql/conf.d/cloudstack.cnf

[mysqld]

innodb_rollback_on_timeout=1

innodb_lock_wait_timeout=600

max_connections=350

log-bin=mysql-bin

binlog-format = 'ROW'

EOF

service mysql restart

#setup nfs

mkdir -p /export/primary /export/secondary /var/primary /var/secondary

cat <<EOF >>/etc/fstab

/var/primary    /export/primary   none    bind  0  0

/var/secondary  /export/secondary none    bind  0  0

EOF

mount --bind /var/primary/ /export/primary/

mount --bind /var/secondary/ /export/secondary/

echo "/export         *(rw,async,no_root_squash,no_subtree_check)" >> /etc/exports

exportfs -a

service nfs-kernel-server restart

#setup cloudstack database

cloudstack-setup-databases cloud:test@localhost --deploy-as=root:root

#setup management server

echo 'Defaults:cloud !requiretty' >> /etc/sudoers.d/cloudstack

cloudstack-setup-management

reboot

I hope you found the post useful. You can subscribe via email or subscribe via a feed reader to get relevant updates from this blog. Have a nice day.

Disabling BBU Auto Learn with megacli

We are currently facing performance problems with mysql and I remember reading about RAID BBU Learning causing huge write performance drops. So I wanted to check if the RAID controller on our Master database had this configured.

Step#1: Find out what is the brand/model of the RAID controller installed on the server

I can of course ask accounting to pull-up the delivery receipt to find the brand/model but where is the fun in that. Googling led me to the following commands:


sudo lspci | grep -i raid
04:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 2108 [Liberator] (rev 04)

sudo lshw -class storage

description: RAID bus controller

       product: MegaRAID SAS 2108 [Liberator]

       vendor: LSI Logic / Symbios Logic

       physical id: 0

       bus info: pci@0000:04:00.0

       logical name: scsi4

       version: 04

       width: 64 bits

       clock: 33MHz

       capabilities: storage pm pciexpress vpd msi msix bus_master cap_list rom

       configuration: driver=megaraid_sas latency=0

       resources: irq:26 ioport:d800(size=256) memory:fae7c000-fae7ffff memory:faec0000-faefffff memory:fae80000-faebffff

Step#2: Install the megacli package to be able to query the raid card for its status

Downloaded the megacli package from the LSI website. But they only provide RPMs so I had to convert them with:

sudo alien -k MegaCli-8.07.06-1.noarch.rpm

Then installed with:

sudo dpkg -i megacli_8.07.06-1_all.deb

To find out where the files got installed do:

sudo dpkg -c megacli_8.07.06-1_all.deb

Step#3: Use megacli to probe for BBU status and information

Running the command results to something unexpected:

./MegaCli64 -adpCount

Controller Count: 0.

Exit Code: 0x00

megacli cant find the raid adapter. It seems that megacli has an issue with kernels >= 3.0 and can be remedied with:

sudo setarch x86_64 --uname-2.6 /opt/MegaRAID/MegaCli/MegaCli64 -adpCount

Controller Count: 1.

Exit Code: 0x01

So to find out about the BBU:

sudo setarch x86_64 --uname-2.6 /opt/MegaRAID/MegaCli/MegaCli64 -AdpBbuCmd -GetBbuStatus -aALL

BBU status for Adapter: 0

BatteryType: iBBU

Voltage: 3972 mV

Current: 0 mA

Temperature: 24 C

Battery State: Optimal

BBU Firmware Status:

  Charging Status              : None

  Voltage                                 : OK

  Temperature                             : OK

  Learn Cycle Requested                  : No

  Learn Cycle Active                      : No

  Learn Cycle Status                      : OK

  Learn Cycle Timeout                     : No

  I2c Errors Detected                     : No

  Battery Pack Missing                    : No

  Battery Replacement required            : No

  Remaining Capacity Low                  : No

  Periodic Learn Required                 : No

  Transparent Learn                       : No

  No space to cache offload               : No

  Pack is about to fail & should be replaced : No

  Cache Offload premium feature required  : No

  Module microcode update required        : No

GasGuageStatus:

  Fully Discharged        : No

  Fully Charged           : Yes

  Discharging             : Yes

  Initialized             : Yes

  Remaining Time Alarm    : No

  Discharge Terminated    : No

  Over Temperature        : No

  Charging Terminated     : No

  Over Charged            : No

  Relative State of Charge: 97 %

  Charger System State: 49168

  Charger System Ctrl: 0

  Charging current: 0 mA

  Absolute state of charge: 53 %

  Max Error: 2 %

Exit Code: 0x00

sudo setarch x86_64 --uname-2.6 /opt/MegaRAID/MegaCli/MegaCli64 -AdpBbuCmd -GetBbuCapacityInfo -aALL

BBU Capacity Info for Adapter: 0

  Relative State of Charge: 97 %

  Absolute State of charge: 53 %

  Remaining Capacity: 641 mAh

  Full Charge Capacity: 664 mAh

  Run time to empty: Battery is not being discharged.  

  Average time to empty: Battery is not being discharged.  

  Estimated Time to full recharge: Battery is not being charged.  

  Cycle Count: 37

Max Error = 2 %

Remaining Capacity Alarm = 120 mAh

Remining Time Alarm = 10 Min

Exit Code: 0x00

sudo setarch x86_64 --uname-2.6 /opt/MegaRAID/MegaCli/MegaCli64 -AdpBbuCmd -GetBbuProperties -aALL

BBU Properties for Adapter: 0

  Auto Learn Period: 30 Days

  Next Learn time: Sun Feb 17 19:37:09 2013

  Learn Delay Interval:0 Hours

  Auto-Learn Mode: Enabled

Exit Code: 0x00

Auto learn mode should be disabled and scheduled during off-peak time.

#!/bin/bash

TMPFILE=$(mktemp -p /tmp bbu.relearn.XXXXXXXXXX) || exit 1

echo "autoLearnMode=1" > $TMPFILE

setarch x86_64 --uname-2.6 /opt/MegaRAID/MegaCli/MegaCli64 -AdpBbuCmd -SetBbuProperties -f $TMPFILE -aALL

sudo setarch x86_64 --uname-2.6 /opt/MegaRAID/MegaCli/MegaCli64 -AdpBbuCmd -GetBbuProperties -aALL

rm $TMPFILE

script for safe write back:

#!/bin/bash
sudo setarch x86_64 --uname-2.6 /opt/MegaRAID/MegaCli/MegaCli64 -LDSetProp ADRA -Lall -aALL
sudo setarch x86_64 --uname-2.6 /opt/MegaRAID/MegaCli/MegaCli64 -LDSetProp -Cached -Lall -aALL
sudo setarch x86_64 --uname-2.6 /opt/MegaRAID/MegaCli/MegaCli64 -LDSetProp DisDskCache -Lall -aALL
sudo setarch x86_64 --uname-2.6 /opt/MegaRAID/MegaCli/MegaCli64 -LDSetProp NoCachedBadBBU -Lall -aALL
sudo setarch x86_64 --uname-2.6 /opt/MegaRAID/MegaCli/MegaCli64 -LDSetProp WB -Lall -aALL
sudo setarch x86_64 --uname-2.6 /opt/MegaRAID/MegaCli/MegaCli64 -LDInfo -Lall -aALL


script to force write back without BBU protection:

#!/bin/bash
sudo setarch x86_64 --uname-2.6 /opt/MegaRAID/MegaCli/MegaCli64 -LDSetProp ADRA -Lall -aALL
sudo setarch x86_64 --uname-2.6 /opt/MegaRAID/MegaCli/MegaCli64 -LDSetProp -Cached -Lall -aALL
sudo setarch x86_64 --uname-2.6 /opt/MegaRAID/MegaCli/MegaCli64 -LDSetProp DisDskCache -Lall -aALL
sudo setarch x86_64 --uname-2.6 /opt/MegaRAID/MegaCli/MegaCli64 -LDSetProp CachedBadBBU -Lall -aALL
sudo setarch x86_64 --uname-2.6 /opt/MegaRAID/MegaCli/MegaCli64 -LDSetProp WB -Lall -aALL
sudo setarch x86_64 --uname-2.6 /opt/MegaRAID/MegaCli/MegaCli64 -LDInfo -Lall -aALL


Useful Links:
http://linux.dell.com/files/whitepapers/solaris/Managing_PERC6_0714.pdf
http://hwraid.le-vert.net/wiki/LSIMegaRAIDSAS
http://yo61.com/dell-drac-bbu-auto-learn-tests-kill-disk-performance.html


I hope you found the post useful. You can subscribe via email or subscribe via a feed reader to get relevant updates from this blog. Have a nice day.

Transferring a huge file over a LAN

We need to transfer several huge files from an old server to a new server. I plan to use netcat to do a raw transfer followed by rsync to ensure data integrity. I'll report back on the results.


On the destination server:
nc -l 1234 > ubuntu-12.04.1-server-i386.iso


On the source server:
time cat ubuntu-12.04.1-server-i386.iso | nc $DESTINATION_IP 1234


On the source server:

cat >rsyncd.conf <<EOF
#rsyncd.conf
[temp]
path=/var/lib/libvirt/images/
read only = yes
list = yes
use chroot = no
EOF

rsync --config=rsyncd.conf --daemon --no-detach --port=1234


On the destination server:
time rsync -av --inplace --progress rsync://apollo@192.168.7.167:1234/temp/ubuntu-12.04.1-server-i386.iso ubuntu-12.04.1-server-i386.iso

Update#1:

time cat ubuntu-12.04.1-server-i386.iso | nc 192.168.7.198 1234

real 0m58.130s
user 0m0.072s
sys 0m2.016s

ubuntu-12.04.1-server-i386.iso is 646MB in size which gives a transfer rate of 11.13MB/s which is NOT what I expected from a gigabit connection.



I hope you found the post useful. You can subscribe via email or subscribe via a feed reader to get relevant updates from this blog. Have a nice day.

Problem with grails, maven and spring security

Environment:
Ubuntu 12.04 3.2.0-20-generic-pae kernel
java version "1.6.0_24"
OpenJDK Runtime Environment (IcedTea6 1.11.1) (6b24-1.11.1-3ubuntu3)
OpenJDK Server VM (build 20.0-b12, mixed mode)

I installed STS 2.9.1 RELEASE which includes apache-maven-3.0.3

I placed the apache-maven-3.0.3 in my PATH environment variable

I created a project with:
mvn archetype:generate \
-DarchetypeGroupId=org.grails \
-DarchetypeArtifactId=grails-maven-archetype \
-DarchetypeVersion=1.3.7 \
-Dversion=1.0.0alpha \
-DgroupId=com.henyo.foobar -DartifactId=foobar

mvn proceeded to download the internet and then created the project folder with the pom file
I modified the pom file to set the source and target to 1.6 as described in the grails manual

I then did:
mvn initialize
which created the grails directory structure

I tried:
mvn grails:run-app
which resulted in the app successfully running

I then tried running just the unit tests with:
mvn grails:exec -Dcommand=test-app -Dargs="--unit"
which resulted in successful build

I then installed the spring security core plugin with:
mvn grails:install-plugin -DpluginName=spring-securit-core

I then tried using the s2-quickstart script:
mvn grails:exec -Dcommand=s2-quickstart -Dargs="com.henyo.foobar.model User Role"
which resulted in a build FAILURE:
[ERROR] Failed to execute goal org.grails:grails-maven-plugin:1.3.7:exec (default-cli) on project foobar: Unable to start Grails: java.lang.reflect.InvocationTargetException: org/springframework/security/core/Authentication: org.springframework.security.core.Authentication -> [Help 1]

It seems that the grails maven integration still needs a lot of work. I suspect this is a dependency resolution issue.

Uploaded a sample project that has this issue: http://dl.dropbox.com/u/5541070/foobar.tar.gz

I hope you found the post useful. You can subscribe via email or subscribe via a feed reader to get relevant updates from this blog. Have a nice day.

Trying Ubuntu 12.04LTS Precise Pangolin

I migrated my desktop from Ubuntu 10.04LTS to Ubuntu 12.04LTS.

A few things I needed to add after installation:

• Chromium as my default browser
• adobe-flash to be able to play flash games on facebook
• openjdk-6-jdk for work stuff
• icedtea-plugin to be able to run charting applet on citisec
• pidgin for messaging
• git for more work
• smartgit for routine pushing and comitting to git repos
• smartsvn for routine pushing and comitting to git repos
• STS for work stuff
• DB Visualizer for work stuff
• Virtual Box for desktop virtualization
• Dia for diagramming
• synergy to share one keyboard between my desktop and laptop
• dropbox for personal file sharing
• pencil for mockups
• smplayer for watching videos
• vlc for videos that do not work with smplayer
• git gui for simple git commits
• myqsl-server for development work
• mysql workbench for managing mysql database
• postgresql for development work
• pgadmin for managing postgres database

Should I be replacing stuff on the list with something else? What other stuff have I forgotten to install?

I hope you found the post useful. You can subscribe via email or subscribe via a feed reader to get relevant updates from this blog. Have a nice day.

dailyLooper bash script

We needed to regenerate some data for the data warehouse so I had to run an extraction script once for each day of the month.

I wrote a simple bash script to make it easier to run a command or other scripts once for every day of a target month and year.

Code can be found below but may also be downloaded from here.

#!/bin/bash
usage(){
    echo "Usage: dailyLooper <year> <month> <script to run>"
    echo "Each date in the given month will be passed into the script in yyyy-mm-dd format"
}


CURRENT_MONTH=$(date +%m)
TARGET_YEAR=$1
TARGET_MONTH=$2

if [ "$#" -lt 3 ]; then
    usage
    exit 1
fi

if [ "$3" = "" ]; then
    SCRIPT="echo"
else
    SCRIPT=$3    
fi

#returns the date yesterday if target month is the current month
#otherwise returns the last date for the target month
lastDayOfMonth(){
    local _year=$1
    local _month=$2   
    if [ $_month -lt $CURRENT_MONTH ]; then
        echo $(date -d "${TARGET_YEAR}-${TARGET_MONTH}-01 + 1 month - 1 day" +%d)
    else
        echo $(date -d "yesterday" +%d)
    fi    
}


main(){
    local _lastDay=$(lastDayOfMonth $TARGET_YEAR $TARGET_MONTH)
    i=1
    while [ "$i" -le $_lastDay ]
    do
        local _date=$(date -d "${TARGET_YEAR}-${TARGET_MONTH}-${i}" +%Y-%m-%d)
        $SCRIPT $_date
        ((i++))
    done #end day of month loop
}

main

I hope you found the post useful. You can subscribe via email or subscribe via a feed reader to get relevant updates from this blog. Have a nice day.

Cheap and Safe File Storage on Linux

I needed to setup a Samba file server using off the shelf desktop components.
It will be used in a small office with 5-10 clients.

Here are the relevant hardware specs:
Processor: i7
Memory: 16GB
Drives: 4 x 1TB sata drives
OS Installed: Ubuntu Sever 10.04LTS

The primary requirement is for the files to be fairly safe i.e. if the server crashes in the middle of saving a file then on restart, the file should return to the consistent state before the save operation. Performance is secondary to safety.

My initial plan was to use one drive for the OS and configure 3 drives in an mdraid level 5. After some research, I came across several references that state that mdraid level 10 works on odd numbered drives. It will work on 3 drives and even on 2. This looked like something I should try. I then had to decide what filesystem to use. I initially wanted to use xfs as I've heard and read a lot of good things about it.

After studying how to setup both mdraid and xfs, I started a thread on the linux-raid mailing list asking for best practice guidelines and guidance on using xfs with mdraid 10n2.

Several days of additional research guided by the comments on the mailing list thread resulted in a lot of learnings for me:

• raid10n2 really works on 3 drives, I had my doubts at first but now I finally get how it can work with just three drives
• drive sector size can be either 512b or 4k which can lead to mis-aligned partitions and have a negative impact on performance
• ext4 provides the option: data=journal which makes it safer than xfs at the cost of some performance

The table below shows the average of the benchmark results I got. Yes, I know bonnie++ only tests sequential write/rewrite/read and some file operations and is not necessarily representative of the workload but IMHO, it is better than nothing. This was how things were setup for the benchmark:

• a 50GB primary partition was created on each device located 1GB from the start of the disk for partition alignment
• the md raid device is created on top of the primary partition created on each device
• the chunk was set to 64k
• the results below are for raid5 and raid10n2 only but I also tried raid10f2 and raid10n2
• raid 10f2 had better read performance but slower write compared to raid 10n2
• raid 10o2 performed quite closely to raid 10n2
• the benchmark was run three times and the average was taken

Bonnie 1.96		Sequential Output				Sequential Input		Random Seeks
64k chunk	Size	Block		Rewrite		Block
ext4/mdraid		K/sec	% CPU	K/sec	% CPU	K/sec	% CPU	/sec	% CPU
raid5 data=ordered	32G	144430	11	68725	9.67	249364	21	280.33	9.33
raid5 data=journal	32G	25077	4.67	21605	5	267283	22.33	293.67	6
raid10n2 data=ordered	32G	137365	8	69940	9	209180	16	365.03	11.67
raid10n2 data=journal	32G	56249	6	37491	6	206548	16	389.00	8

For those interested, you can find/get the script I used here.

I modified this script to run my tests.

You can find the raw results here.

You can also find the debug info here.

I am considering re-running the tests but this time with iozone instead of bonnie++ so that I can get scores for random read and write.

Next, I plan to do pull-the-plug testing on ext4,data=journal on top of raid10n2 while doing streaming writes of a huge file over samba. I plan to use 2 different video files greater than 3GB. First I will copy file1 to the samba share and time it. I will then compute a checksum for file1 both locally and on the samba server. The computed checksum should be equal. Next, I will compute a checksum for file2 locally. It should be different from the value computed for file1. I will then move file1 to another directory and rename file2 to the the same name as file1. I will then attempt to copy the new file1 to the samba server. Halfway into the copy, I will pull the plug of the samba server. Next I will restart the samba server and recompute the checksum of file1. It should still be equal to the computed copy before.

If the system behaves as expected, I will go with ext4,data=journal on top of raid10n2.


I hope you found the post useful. You can subscribe via email or subscribe via a feed reader to get relevant updates from this blog. Have a nice day.

Check disk write-caching

I'm investigating the performance of file systems and found myself needing to check if write-caching was enabled for my disk.

hdparm -W /dev/sda

To turn off write-caching, do

hdparm -W0 /dev/sda

I hope you found the post useful. You can subscribe via email or subscribe via a feed reader to get relevant updates from this blog. Have a nice day.

Mount a Proxmox KVM raw image in an LVM over DRBD

Hardware:
2 desktop computers
Intel i3-530
4GB DDR3
1TB Hard drive

Setup:
- 2 node cluster of ProxmoxVE 2.0-35/d07f49c3(rc1)
- LVM volume group named drbd_storage created on top of a drbd block device
- Installed Ubuntu Server 10.04LTS in a kvm container named ubuntu-kvm-template
- proxmox created a logical volume in drbd_storage named vm-108-disk-1

I want to mount the disk so that I can check something out. Apparently it is not that simple.

I eventually figured it out thru trial and error of suggestions across serveral sources.

losetup /dev/loop0 /dev/drbd_storage/vm-108-disk-1
fdisk -l /dev/loop0
apt-get install kpartx
kpartx -av /dev/loop0
pvscan
vgscan
lvscan
vgchange -ay
lvscan
mkdir -p /tmp/108
mount /dev/ubuntu-kvm-template/root /tmp/108
umount /mnt/108
lvchange -an /dev/ubuntu-kvm-template/*
kpartx -dv /dev/loop0
losetup -d /dev/loop0


I hope you found the post useful. You can subscribe via email or subscribe via a feed reader to get relevant updates from this blog. Have a nice day.

Finding the largest sized directory

One of our servers keeps running out of disk space. To address this, we needed to find out which folders had the largest size.

On ubuntu server 10.04LTS, this is how we found the directories with the greatest size:

du -m /var/log | sort -n -r | head -n 10


I hope you found the post useful. You can subscribe via email or subscribe via a feed reader to get relevant updates from this blog. Have a nice day.

Creating a Latin1 PostgreSQL database in Ubuntu 10.04LTS

I've encountered this problem several times already and I've finally stumbled upon a solution.

The Problem:

We were tasked to move a postgresql database from an centos based server to a new ubuntu based server. The database was in latin1 encoding.

On the new ubuntu based server, creating a database with latin1 encoding results in an error.
Command: createdb -E LATIN1 --template template0 trial
Result:
createdb: database creation failed: ERROR:  encoding LATIN1 does not match locale en_PH.utf8
DETAIL: The chosen LC_CTYPE setting requires encoding UTF8

The Fix:

First we needed to modify /var/lib/locales/supported.d/local and add the line:
en_PH.ISO-8859-1 ISO-8859-1

Next we had to regenerate the locale files with:
sudo dpkg-reconfigure locales

Afterwards we were able to create a latin1 database with:
createdb -E LATIN1 --template template0 --locale en_PH.ISO-8859-1 trial

I hope you found the post useful. You can subscribe via email or subscribe via a feed reader to get relevant updates from this blog. Have a nice day.

Changing hostname on Ubuntu Server 10.04LTS

I found out the hard way that it's changing the name of an Ubuntu Server 10.04LTS was not as straight forward as I initially thought. To save you and me the trouble and for future reference, I document how I did it here.

hostname NEW_NAME
echo NEW_NAME > /etc/hostname
cat /etc/hosts | sed s/OLD_NAME/NEW_NAME/g > /etc/hosts.new
mv /etc/hosts /etc/hosts.old
mv /etc/hosts.new /etc/hosts

I hope you found the post useful. You can subscribe via email or subscribe via a feed reader to get relevant updates from this blog. Have a nice day.

Monitoring Tomcat6 with VisualVM on Ubuntu

One of our applications running on Tomcat6 was encountering an issue which was drastically affecting the delivery of services to our clients. To diagnose this, we needed to monitor the state of the virtual machine running Tomcat6. This is what I had to do:

Assumptions:
- tomcat6 installed on an ubuntu server instance
- firewall on the server instance is enabled
- server has non public ip address: 10.0.0.1
- client machine installed with java6 which includes visualvm

1) download catalina-jmx-remote.jar and copy it into the tomcat6 lib folder. At the time of this writing, I found it here: http://apache.opensourceresources.org/tomcat/tomcat-6/v6.0.33/bin/extras/

2) modify /etc/defaults/tomcat6, add the following line to JAVA_OPTS

-Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Djava.rmi.server.hostname=10.0.0.1

3) modify /etc/tomcat6/server.xml, add the following line inside the server block near the other listeners:

<Listener className="org.apache.catalina.mbeans.JmxRemoteLifecycleListener"
rmiRegistryPortPlatform="8086" rmiServerPortPlatform="8086" />

4) enable access to port 8086

ufw allow 8086

5) restart tomcat6

service tomcat6 restart

6) start visualvm on the client machine and add a jmx connection: 10.0.0.1:8086

I hope you found the post useful. You can subscribe via email or subscribe via a feed reader to get relevant updates from this blog. Have a nice day.

Setting up Apache2 and Tomcat6 on Ubuntu 10.04

This is a short post on setting up Apache2 and Tomcat6

Install Java

apt-get -y install python-software-properties
add-apt-repository "deb http://archive.canonical.com/ lucid partner"
apt-get update
apt-get install -y sun-java6-jdk

Install tomcat6 with apache2 support

apt-get -y install tomcat6 libtcnative-1 libapache2-mod-jk

Setup worker.properties with the following values:
workers.tomcat_home=/usr/share/tomcat6
workers.java_home=/usr/lib/jvm/java-6-sun

nano /etc/libapache2-mod-jk/workers.properties

Modify /etc/apache2/mods-available/jk.load

cat >>/etc/apache2/mods-available/jk.load <<EOF
JkWorkersFile /etc/libapache2-mod-jk/workers.properties
JkLogFile /var/log/apache2/mod_jk.log
JkLogLevel error
EOF

Setup virtual host

cat >/etc/apache2/sites-available/test.henyo.com <<EOF
NameVirtualHost *
<VirtualHost *>
        ServerName test.henyo.com
        JkMount /* ajp13_worker
</VirtualHost>
EOF

Activate the site

a2ensite test.henyo.com

Setup tomcat for ajp13

nano /etc/tomcat6/server.xml

Restart tomcat and apache2

service tomcat6 restart
service apache2 restart

Install and configure UFW

apt-get install ufw
ufw allow ssh
ufw allow http
ufw enable

Install and configure Tripwire

apt-get install tripwire

I hope you found the post useful. You can subscribe via email or subscribe via a feed reader to get relevant updates from this blog. Have a nice day.

LXC(Linux Containers) on Ubuntu 10.04

I was able to get LXC(Linux Containers) to work on an Ubuntu 10.04 host system. Linux containers provide lightweight virtualization that lets you isolate processes and resources without the need to provide instruction interpretation mechanisms and other complexities of full virtualization. So if you need to quickly spin-up a virtual server to test a new version of rails or any other software then this post might help you do that.

For the impatient ones, I wrote a bash script which initializes a file system as well as create and runs a container instance. It can be found here and here. You still have to make changes to the host system yourself though before using the scripts.

To install the required tools do:

sudo apt-get install lxc bridge-utils debootstrap

Next is to setup cgroups with:

sudo mkdir /cgroup
sudo echo "none /cgroup cgroup defaults 0 0" >> /etc/fstab
sudo mount /cgroup

Then proceed to setup the network for bridge networking with:

sudo apt-get remove –purge network-manager network-manager-gnome

and then modify /etc/network/interfaces so it contains:

auto lo
iface lo inet loopback

auto br0
iface br0 inet static
address 192.168.0.10
netmask 255.255.255.0
broadcast 192.168.0.255
gateway 192.168.0.1
bridge_ports eth0
bridge_stp off
bridge_maxwait 5
post-up /usr/sbin/brctl setfd br0 0

then we setup the nameserver

sudo echo "nameserver 8.8.8.8" > /etc/resolv.conf

then reboot the host machine with:

sudo reboot

Log-in again and check that the network is still working with:

ping 8.8.8.8

and also

ping archive.ubuntu.com

Next we setup a root file system template that will be used by the containers. The following steps can be done easily using this script.

To start, we build a lucid root file system

sudo mkdir /lxc
sudo cd /lxc
sudo debootstrap --variant=minbase --arch i386 lucid rootfs_template

The next commands below will refer to placeholders so make sure to adjust the commands and replace them with the following:
$ROOTFS --> /lxc/rootfs_template
$MIRROR --> http://archive.ubuntu.com/ubuntu

to continue, we fix the device files

rm -rf $ROOTFS/dev
mkdir $ROOTFS/dev
mknod -m 666 $ROOTFS/dev/null c 1 3
mknod -m 666 $ROOTFS/dev/zero c 1 5
mknod -m 666 $ROOTFS/dev/random c 1 8
mknod -m 666 $ROOTFS/dev/urandom c 1 9
mkdir -m 755 $ROOTFS/dev/pts
mkdir -m 1777 $ROOTFS/dev/shm
mknod -m 666 $ROOTFS/dev/tty c 5 0
mknod -m 666 $ROOTFS/dev/tty0 c 4 0
mknod -m 666 $ROOTFS/dev/tty1 c 4 1
mknod -m 666 $ROOTFS/dev/tty2 c 4 2
mknod -m 666 $ROOTFS/dev/tty3 c 4 3
mknod -m 666 $ROOTFS/dev/tty4 c 4 4
mknod -m 600 $ROOTFS/dev/console c 5 1
mknod -m 666 $ROOTFS/dev/full c 1 7
mknod -m 600 $ROOTFS/dev/initctl p
mknod -m 666 $ROOTFS/dev/ptmx c 5 2

next we modify /etc/apt/sources.list so that it contains:

deb $MIRROR lucid main universe multiverse
deb $MIRROR lucid-security main universe multiverse

next we install and update the packages, run the following as root:

chroot $ROOTFS apt-get install --force-yes -y gpgv
chroot $ROOTFS apt-get update
chroot $ROOTFS apt-get install -y language-pack-en
chroot $ROOTFS locale-gen en_US.UTF-8
chroot $ROOTFS /usr/sbin/update-locale LANG="en_US.UTF-8" LANGUAGE="en_US.UTF-8" LC_ALL="en_US.UTF-8" LC_CTYPE="C"
chroot $ROOTFS apt-get install -y \
    adduser \
    apt-utils \
    console-setup \
    iproute \
    iptables \
    nano \
    netbase \
    openssh-blacklist \
    openssh-blacklist-extra \
    openssh-server \
    iputils-ping \
    rsyslog \
    sudo \
    ufw \
    vim
chroot $ROOTFS apt-get clean

as root user, setup mtab with:

chroot $ROOTFS ln -s /proc/mounts /etc/mtab

as root user, next we append the following to $ROOTFS/etc/environment:

LANG="en_US.UTF-8"
LANGUAGE="en_US.UTF-8"
LC_ALL="en_US.UTF-8"
LC_CTYPE="C"

as root user, next remove unneeded files from $ROOTFS/etc/init of the template system with:

cd $ROOTFS/etc/init
rm -f rc-sysinit.conf console* control* hwclock* module* mount* network-interface* plymouth* procps* tty{4,5,6}.conf udev* upstart*

as root user, create a $ROOTFS/etc/init/rc-sysinit file with the following content:

#!/bin/bash
rm -f $(find /var/run -name '*pid')
rm -f /var/lock/apache/*
exit 0

as root user, make it executable with:

chmod a+x $ROOTFS/etc/init/rc-sysinit

as root user, modify ssh.conf to contain:

# ssh - OpenBSD Secure Shell server
#
# The OpenSSH server provides secure shell access to the system.

description     "OpenSSH server"

start on startup
stop on runlevel S

expect fork
respawn
respawn limit 10 5
umask 022
# replaces SSHD_OOM_ADJUST in /etc/default/ssh
oom never

pre-start script
    test -x /usr/sbin/sshd || { stop; exit 0; }
    test -e /etc/ssh/sshd_not_to_be_run && { stop; exit 0; }
    test -c /dev/null || { stop; exit 0; }

    mkdir -p -m0755 /var/run/sshd
end script

# if you used to set SSHD_OPTS in /etc/default/ssh, you can change the
# 'exec' line here instead
exec /usr/sbin/sshd

as root user, we create an archive of the template so we can reuse it:

cd /lxc
tar zcvf rootfs_template.tgz rootfs_template

To make things a lot easier, you can just use the bash script I wrote which can be downloaded from here. Save it in /lxc and make it executable and run it with:

sudo chmod +x /lxc/create-rootfs.sh
sudo cd /lxc
sudo ./create-rootfs.sh

To configure and run a container, download another bash script I wrote here and save it in /lxc.
Make it executable and run it with:

sudo chmod +x /lxc/create-container.sh
cd /lxc
sudo ./create-container.sh

I hope you found the post useful. You can subscribe via email or subscribe via a feed reader to get relevant updates from this blog. Have a nice day.

Setting up CouchDB in Ubuntu Server 10.04 LTS on VirtualBox

In my day job, I was assigned the task of building a system that could handle 2 million rows of sales data a day. We are seriously considering putting couchdb to use. To even begin to evaluate couchdb, I of course needed to play around with it in a convenient test environment. Fortunately, as documented in my previous post on setting up an ubuntu machine, I have VirtualBox installed and ready.

To setup a couchdb environment, I first needed to create a virtual machine that runs Ubuntu Server 10.04 LTS. To begin, I downloaded the installer for Ubuntu Server 10.04 LTS from the Ubuntu website. I started VirtualBox and went thru the New Virtual Machine Wizard. I named my virtual machine as couchdb-server and accepted most of the defaults. Before starting the virtual machine, I adjusted the Storage settings of the virtual machine and configured the CD/DVD device to mount the Ubuntu Server 10.04 LTS installer(ISO). Starting the virtual machine will bring up the menu for the Ubuntu Server 10.04 LTS installer. I went thru the installation using the defaults. When asked what services to install, I just chose the OpenSSH server.

With the installation finished, I logged in and installed couchdb using:

sudo aptitude install couchdb

After a few screen fulls of text, couchdb was successfully installed. I then added a few lines to /etc/networking/interfaces to enable me to access the virtual server from the outside.

auto eth1
iface eth1 inet dhcp

Next, I turned off the virtual server using:

sudo shutdown -h now

I then modified the Network settings of the virtual machine using VirtualBox. I enabled Network Adapter 2 and set the Attached to drop down to Host-only Adapter.

I then powered up the virtual machine and after logging in, I checked the configured network devices using:

ifconfig

This should show eth1 to be configured and in my case, its IP address is 192.168.56.101

I then started couchdb with:

sudo /etc/init.d/couchdb start

To test couchdb, I opened a browser and accessed:

http://192.168.56.101:5984/_utils

I hope you found the post useful. You can subscribe via email or subscribe via a feed reader to get relevant updates from this blog. Have a nice day.

Farewell Ubuntu 9.04, Hello Ubuntu 10.10

Ubuntu 9.04 has just reached EOL as I was insistently prompted by the updater.
I've now moved to Ubuntu 10.10 and it looks and feels okay. This post is about setting up a machine for Development work.

Using Ubuntu software center has allowed me to easily install the stuff I need day-to-day:

Internet Related:
Chromium
Thunderbird
Pidgin
Mail Notification


Multimedia:
VLC
Adobe Flash 10

Development Related:
Sun Java6
Netbeans
MySQL Database Server
MySQL Administrator
MySQL Query Browser
pgAdmin
7zip

Stuff I had to install manually:
Virtual Box(downloaded the ubuntu 10.10 installer from their site)

Git(terminal usage required)

sudo apt-get install git git-gui

SmartGit(downloaded installer from their site)

PostgreSQL(terminal usage required)

sudo apt-get install postgresql

gedit-plugins(terminal usage required)

sudo apt-get install gedit-plugins

gedit-grails-bundle(terminal usage required)

cd ~/Downloads
git clone https://github.com/aeischeid/gedit-grails-bundle.git
cd gedit-grails-bundle/grails-gedit
./install.sh

gedit-gmate(terminal usage required)

sudo apt-add-repository ppa:ubuntu-on-rails/ppa
sudo apt-get update
sudo apt-get install gedit-gmate

wine1.3(terminal usage required)

sudo add-apt-repository ppa:ubuntu-wine/ppa
sudo apt-get update
sudo apt-get install wine1.3

unrar(terminal usage required)

sudo apt-get install unrar

dropbox
went to their website and downloaded the linux installer for ubuntu


I hope you found the post useful. I hope to update this post when I add some other usefull stuff on this work machine. You can subscribe via email or subscribe via a feed reader to get relevant updates from this blog. Have a nice day.