Building a good Software Startup

If I were to build a good software startup today, I'd run with the following ideas.

Hire only good people who get things DONE.
Even if they are more expensive in terms of compensation.
Quality trumps numbers. No doubt about it.

Development Team

• 1 Lead Developer
• 1 Senior Developer
• 2 Junior Developers
• 2 Business Analyst/QA

Operations Team

• 1 Lead Systems Admin
• 1 Senior System Admin
• 6 Junior System Admins

Give your team a quiet working environment with ergonomic furniture and equipment

Give your team good internet connection

• the faster they can download their installers, the earlier they can start working on code
• the faster they can look up the solution on google or stackoverflow, the faster they can proceed to the next problem

Give your team good computer hardware

• at least dual core processors with virtualization support
• at least 8GB memory
• at least 2 hard drives for mirror setup
• at least 2 screens

Make sure to have enough environments to test, deploy and support the system properly.

• Development Environment
• Staging Environment
• Support Environment
• Production Environment

Make use of Free or Open Source systems to manage the development process.
Buy a commercial product only if you can't find a solution that fits.

Project/Issue Tracking with Source Code Management Options:

• Indefero
• Gitorious
• Gitlab
• Redmine+gitolite

Continuous Integration Server Options:

• Jenkins
• Team City

Code Quality:

• Sonar

Setup google apps for for email, calendar and other stuff

As an example, here is how I would set it up on three physical servers:

Server#1 running ProxmoxVE
- Firewall/Proxy Server/Package Cache(apt,yum,mvn) VM
- Redmine+gitolite VM
- Jenkins VM
- Sonar VM
- NFS/Samba Server VM

Server#2 running ProxmoxVE
- Development Environment VM
- Staging Environment VM
- Support Environment VM

Server#3 for storage of backups
- nexentastor or freenas or centos+zfsonlinux

I hope you found the post useful. You can subscribe via email or subscribe via a feed reader to get relevant updates from this blog. Have a nice day.

ProxmoxVE 2: Configuring DRBD

I am running ProxmoxVE 2.0-35/d07f49c3 on two nodes connected using a cross cable on gigabit nics.
host1 ip: 10.0.10.201
host2 ip: 10.0.10.202

Check if drbd module is available

modprobe -l | grep drbd

Check if drbd module is loaded

lsmod | grep drbd

If not, load it with

modprobe -v drbd

Check the module version

modinfo drbd | grep version

I installed the appropriate drbd8-utils package on both nodes

dpkg -i drbd8-utils_8.3.10-0_amd64.deb

I built my own deb from source. You can find my build here.

I shrank the data volume on both of
I made use of the extents I freed up by shrinking the data volume to create an lvm volume that will be used for drbd.

lvcreate --verbose --extents 50000 --name lv4drbd pve

Next I modified /etc/drbd.d/global_common.conf so that it only contains:

global {
    usage-count no;
}
common {
    protocol C;
    syncer {
        rate 30M; 
    }
}

Next I created a new resouce file /etc/drbd.d/drbd0.res

resource drbd0 {
    protocol C;
    startup {
        wfc-timeout  0;
        degr-wfc-timeout 60;
        become-primary-on both;
    }
    net {
        cram-hmac-alg sha1;
        shared-secret "drbd0-secret";
        allow-two-primaries;
        after-sb-0pri discard-zero-changes;
        after-sb-1pri discard-secondary;
        after-sb-2pri disconnect;
    }
    on host1 {
        device /dev/drbd0;
        disk /dev/pve/lv4drbd;
        address 10.0.10.201:7780;
        meta-disk internal;
    }
    on host2 {
        device /dev/drbd0;
        disk /dev/pve/lv4drbd;
        address 10.0.10.202:7780;
        meta-disk internal;
    }
}

I added the IP address of the other host into each of the /etc/hosts files.

Next I started drbd on both nodes with

/etc/init.d/drbd start

I then initialized drbd metatdata on both nodes with

drbdadm create-md drbd0

Next I brought up the device on both nodes with

drbdadm up drbd0

I checked the status on both nodes with

drbd-overview

Next I started synchronization(should be instantaneous) from one node with

drbdadm -- --clear-bitmap new-current-uuid drbd0

Lastly, I restarted the drbd service on both nodes to enable Primary/Primary operation

/etc/init.d/drbd stop
/etc/init.d/drbd start

I hope you found the post useful. You can subscribe via email or subscribe via a feed reader to get relevant updates from this blog. Have a nice day.

ProxmoxVE 2.0: Shrinking the data volume

After installing ProxmoxVE 2.0rc1, a data volume was created which makes use of most of the space on the drive.

I wanted to carve out a separate volume for some other use so I needed to shrink this data volume:

First, I unmounted the logical volume with:

umount /var/lib/vz

Then I proceeded to shrink the file system to the minimum size possible with:

resize2fs -Mf /dev/pve/data

Then I proceeded to reduce the size of the logical volume with:

lvreduce --verbose --extents -100000 /dev/pve/data

Then I proceeded to grow the file system back to occupy the now reduced volume with:

resize2fs -f /dev/pve/data

Lastly, I remounted the file system with:

mount /dev/pve/data /var/lib/vz

I hope you found the post useful. You can subscribe via email or subscribe via a feed reader to get relevant updates from this blog. Have a nice day.

FizzBuzz in Javascript

Joel and company probably won't hire me since he only gets rockstar developers but here is my attempt at FizzBuzz

<script>
    for(var i=1;i<=100;i++){
        if(i%3 != 0  && i%5 != 0)
            document.write(i);
        if(i%3 == 0)
            document.write('Fizz');            
        if(i%5 == 0)
            document.write('Buzz');                    
        document.write(' ');
    } //end for loop
</script>

I hope you found the post useful. You can subscribe via email or subscribe via a feed reader to get relevant updates from this blog. Have a nice day.

dailyLooper bash script

We needed to regenerate some data for the data warehouse so I had to run an extraction script once for each day of the month.

I wrote a simple bash script to make it easier to run a command or other scripts once for every day of a target month and year.

Code can be found below but may also be downloaded from here.

#!/bin/bash
usage(){
    echo "Usage: dailyLooper <year> <month> <script to run>"
    echo "Each date in the given month will be passed into the script in yyyy-mm-dd format"
}


CURRENT_MONTH=$(date +%m)
TARGET_YEAR=$1
TARGET_MONTH=$2

if [ "$#" -lt 3 ]; then
    usage
    exit 1
fi

if [ "$3" = "" ]; then
    SCRIPT="echo"
else
    SCRIPT=$3    
fi

#returns the date yesterday if target month is the current month
#otherwise returns the last date for the target month
lastDayOfMonth(){
    local _year=$1
    local _month=$2   
    if [ $_month -lt $CURRENT_MONTH ]; then
        echo $(date -d "${TARGET_YEAR}-${TARGET_MONTH}-01 + 1 month - 1 day" +%d)
    else
        echo $(date -d "yesterday" +%d)
    fi    
}


main(){
    local _lastDay=$(lastDayOfMonth $TARGET_YEAR $TARGET_MONTH)
    i=1
    while [ "$i" -le $_lastDay ]
    do
        local _date=$(date -d "${TARGET_YEAR}-${TARGET_MONTH}-${i}" +%Y-%m-%d)
        $SCRIPT $_date
        ((i++))
    done #end day of month loop
}

main

I hope you found the post useful. You can subscribe via email or subscribe via a feed reader to get relevant updates from this blog. Have a nice day.

Cheap and Safe File Storage on Linux

I needed to setup a Samba file server using off the shelf desktop components.
It will be used in a small office with 5-10 clients.

Here are the relevant hardware specs:
Processor: i7
Memory: 16GB
Drives: 4 x 1TB sata drives
OS Installed: Ubuntu Sever 10.04LTS

The primary requirement is for the files to be fairly safe i.e. if the server crashes in the middle of saving a file then on restart, the file should return to the consistent state before the save operation. Performance is secondary to safety.

My initial plan was to use one drive for the OS and configure 3 drives in an mdraid level 5. After some research, I came across several references that state that mdraid level 10 works on odd numbered drives. It will work on 3 drives and even on 2. This looked like something I should try. I then had to decide what filesystem to use. I initially wanted to use xfs as I've heard and read a lot of good things about it.

After studying how to setup both mdraid and xfs, I started a thread on the linux-raid mailing list asking for best practice guidelines and guidance on using xfs with mdraid 10n2.

Several days of additional research guided by the comments on the mailing list thread resulted in a lot of learnings for me:

• raid10n2 really works on 3 drives, I had my doubts at first but now I finally get how it can work with just three drives
• drive sector size can be either 512b or 4k which can lead to mis-aligned partitions and have a negative impact on performance
• ext4 provides the option: data=journal which makes it safer than xfs at the cost of some performance

The table below shows the average of the benchmark results I got. Yes, I know bonnie++ only tests sequential write/rewrite/read and some file operations and is not necessarily representative of the workload but IMHO, it is better than nothing. This was how things were setup for the benchmark:

• a 50GB primary partition was created on each device located 1GB from the start of the disk for partition alignment
• the md raid device is created on top of the primary partition created on each device
• the chunk was set to 64k
• the results below are for raid5 and raid10n2 only but I also tried raid10f2 and raid10n2
• raid 10f2 had better read performance but slower write compared to raid 10n2
• raid 10o2 performed quite closely to raid 10n2
• the benchmark was run three times and the average was taken

Bonnie 1.96		Sequential Output				Sequential Input		Random Seeks
64k chunk	Size	Block		Rewrite		Block
ext4/mdraid		K/sec	% CPU	K/sec	% CPU	K/sec	% CPU	/sec	% CPU
raid5 data=ordered	32G	144430	11	68725	9.67	249364	21	280.33	9.33
raid5 data=journal	32G	25077	4.67	21605	5	267283	22.33	293.67	6
raid10n2 data=ordered	32G	137365	8	69940	9	209180	16	365.03	11.67
raid10n2 data=journal	32G	56249	6	37491	6	206548	16	389.00	8

For those interested, you can find/get the script I used here.

I modified this script to run my tests.

You can find the raw results here.

You can also find the debug info here.

I am considering re-running the tests but this time with iozone instead of bonnie++ so that I can get scores for random read and write.

Next, I plan to do pull-the-plug testing on ext4,data=journal on top of raid10n2 while doing streaming writes of a huge file over samba. I plan to use 2 different video files greater than 3GB. First I will copy file1 to the samba share and time it. I will then compute a checksum for file1 both locally and on the samba server. The computed checksum should be equal. Next, I will compute a checksum for file2 locally. It should be different from the value computed for file1. I will then move file1 to another directory and rename file2 to the the same name as file1. I will then attempt to copy the new file1 to the samba server. Halfway into the copy, I will pull the plug of the samba server. Next I will restart the samba server and recompute the checksum of file1. It should still be equal to the computed copy before.

If the system behaves as expected, I will go with ext4,data=journal on top of raid10n2.


I hope you found the post useful. You can subscribe via email or subscribe via a feed reader to get relevant updates from this blog. Have a nice day.

Check disk write-caching

I'm investigating the performance of file systems and found myself needing to check if write-caching was enabled for my disk.

hdparm -W /dev/sda

To turn off write-caching, do

hdparm -W0 /dev/sda

I hope you found the post useful. You can subscribe via email or subscribe via a feed reader to get relevant updates from this blog. Have a nice day.

Globe UnliAllTxt with PlaySMS

We are using PlaySMS on top of SMS Server Tools 3(http://smstools3.kekekasvi.com/index.php?p=) for our SMS gateway.

We wanted to take advantage of the Unlimited All Text Promo of Globe. I couldn't get the USB modem to register for the promo.

Registering thru a phone with the same SIM works. After reading the documentation of SMS Server Tools, I finally got it to work.

First I had to set the correct SMSC for the modem.

smsc = 639170000130

Next, I had to manually craft an SMS file that will be sent to the shorcode:8888.

To: s8888
Modem: GLOBE

UALLPLUS25

I hope you found the post useful. You can subscribe via email or subscribe via a feed reader to get relevant updates from this blog. Have a nice day.

ProxmoxVE 2.0rc1 Notes:Setup

This is Part#1 of a series of posts on my experience with Proxmox 2.0rc1.

I was tasked to look for a suitable solution to build an internal cloud/virtualization infrastructure and I've decided to spend some time with ProxmoxVE 2.0rc1.

The list below is the hardware available to me:
2 desktop computers
Intel i3-530
4GB DDR3
1TB Hard drive

Things I want to try:
- benchmark disk/file IO performance of host system
- benchmark dis/file IO performance of guest system
- openvz live migration
- kvm live migration
- HA storage for the VM using DRBD
- ProxmoxVE HA i.e. guest starting on secondary node when primary node goes down
- openvz ubuntu server 10.04LTS guest running tomcat6 on sun-java6-jdk
- kvm ubuntu server 10.04LTS guest running tomcat6 on sun-java6-jdk

Installing ProxmoxVE 2.0rc1 on the hardware was fairly straight forward.
The installer took over the whole 1TB drive and partitioned it on its own.

running fdisk -l /dev/sda && df -Th revealed:
- /dev/sda1 is 495MB ext3 mounted on /boot
- dev/sda2 is 999.5GB LVM

running pvscan && vgscan revealed:
- /dev/sda2 was configured as an lvm2 physcial volume
- a volume group named pve was created by proxmox

running lvscan && mount revealed:
- a 4GB /dev/pve/swap volume was created for swap
- a 96GB /dev/pve/root ext3 volume mounted as the root file system
- a 899GB /dev/pve/data ext3 volume mounted on /var/lib/vz

Next I updated ProxmoxVE 2.0rc1 which is Debian based.
The host is behind a proxy server so to update I ran:

export http_proxy=http://10.0.0.1:3128

apt-get update && apt-get -y upgrade

I proceeded to install some useful tools:
apt-get install screen dstat

Next, I configured NTP to sync to a time server on the local network

Next I downloaded an openvz template for ubuntu 10.04LTS and proceeded to create and run a guest.

I ran across an annoying bug while accessing the guest using the console. Tab completion was not working which made it difficult to work on the web console. I found a work around and made a separate post for the web console tab fix

Stay tuned for my next post in this series: Setting up DRBD backed storage even with just one hard disk

I hope you found the post useful. You can subscribe via email or subscribe via a feed reader to get relevant updates from this blog. Have a nice day.

ProxmoxVE 2.0rc1 web console tab fix

I ran across an annoying bug while accessing the guest using the console. Tab completion was not working which made it difficult to work on the web console. I found a work around which modifies /usr/share/pve-manager/ext4/pvemanagerlib.js with:

cp /usr/share/pve-manager/ext4/pvemanagerlib.js /root/pvemanagerlib.js.bak

cat /usr/share/pve-manager/ext4/pvemanagerlib.js | \
sed "s/'Show Controls', value: 'No'/'Show Controls', value: 'Yes'/" \
> /usr/share/pve-manager/ext4/pvemanagerlib.js


I hope you found the post useful. You can subscribe via email or subscribe via a feed reader to get relevant updates from this blog. Have a nice day.

Mount a Proxmox KVM raw image in an LVM over DRBD

Hardware:
2 desktop computers
Intel i3-530
4GB DDR3
1TB Hard drive

Setup:
- 2 node cluster of ProxmoxVE 2.0-35/d07f49c3(rc1)
- LVM volume group named drbd_storage created on top of a drbd block device
- Installed Ubuntu Server 10.04LTS in a kvm container named ubuntu-kvm-template
- proxmox created a logical volume in drbd_storage named vm-108-disk-1

I want to mount the disk so that I can check something out. Apparently it is not that simple.

I eventually figured it out thru trial and error of suggestions across serveral sources.

losetup /dev/loop0 /dev/drbd_storage/vm-108-disk-1
fdisk -l /dev/loop0
apt-get install kpartx
kpartx -av /dev/loop0
pvscan
vgscan
lvscan
vgchange -ay
lvscan
mkdir -p /tmp/108
mount /dev/ubuntu-kvm-template/root /tmp/108
umount /mnt/108
lvchange -an /dev/ubuntu-kvm-template/*
kpartx -dv /dev/loop0
losetup -d /dev/loop0


I hope you found the post useful. You can subscribe via email or subscribe via a feed reader to get relevant updates from this blog. Have a nice day.