HowTo create KVM Virtual Machines on the cluster

This is a howto for creating KVM virtual machines on the NCIT-Cluster. You can run any virtual machine that has a qcow2 disk and you can use how many processors and memory you wish without disturbing the other users.Because we're using KVM you will have VNC access to the virtual machine and an IP in 10.42.X.X/16.

Objectives

You have a lab virtual machine and you want to run several instances of that virtual machine, and after your you're done, you want to revert to the main virtual machine. You can achieve this by using a master vm instance. You setup this lab into this master instance, shutdown the virtual machine and run as many copies you need by using KVMs copy-on-write options. KVM will use your harddisc as a readonly file, and write all diffs to another file. If you need to update the lab machine, shutdown all instances, delete the instance files, and just update the master machine and you're done.

 

Virtual Infrastructure Architecture

 

Basically you use the SunGridEngine batch system to reserve resources (cpu and memory). All virtual machines are hosted on LustreFS, so you have an infiniband connection on the opteron nodes and 4x1Gb maximal total throughput if running on the other nodes. Total cores available: 380 (for now).

[alexandru.herisanu@fep-53-1 ~]$ qstat -g c
all.q                             0.52      0      0    180    380    200      0
ibm-nehalem.q                     -NA-      0      0      0      0      0      0
ibm-opteron.q                     0.77    120      0     36    168      0     12
ibm-quad.q                        0.34     80      0    144    224      0      0


Setup

First of all you need to be in the kvm-user group. There are some templates available, but you can use any KVM machine. The templates are located into /d02/home/ncit-cluster/kvm-templates.

 

There are two available templates:

Copy one of those to your home folder ~/LustreFS. This wil copy it into your Lustre share (email us if you can not acces it).

It should look like this:

[alexandru.herisanu@fep-53-1 ~]$ id
uid=9552(alexandru.herisanu) gid=9000(profi) groups=514(app-gaussian03),515(vtune),518(kvm-user),9000(profi)
[alexandru.herisanu@fep-53-1 ~]$ cd LustreFS/virtual_machines/db2-image/
[alexandru.herisanu@fep-53-1 db2-image]$ ls
db2-hda.qcow2                           hda-inst_ABDLab.qcow2
db2-hdb.qcow2                           hdb-inst_ABDLab01.qcow2
db2-image.sh                            hdb-inst_ABDLab02.qcow2
Enterprise-R5-U5-Server-x86_64-dvd.iso  hdb-inst_ABDLab.qcow2
hda-inst_ABDLab01.qcow2                 out
hda-inst_ABDLab02.qcow2                 start_machines.sh

There are several important files:

Setup your start_machines.sh startup script.It looks like this:

#!/bin/bash

##
## Master Copy
#
#apprun.sh kvm --queue This email address is being protected from spambots. You need JavaScript enabled to view it. --vmname ABDLab --cpu 2 --memory 2048M --hda db2-hda.qcow2 --hdb db2-hdb.qcow2 --cdrom Enterprise-R5-U5-Server-x86_64-dvd.iso --status status.txt --mac 52:54:01:12:34:01 --vncport 0 --master

##
## Slave Copy
#
apprun.sh kvm --queue This email address is being protected from spambots. You need JavaScript enabled to view it. --vmname ABDLab01 --cpu 2 --memory 2048M --hda db2-hda.qcow2 --hdb db2-hdb.qcow2 --cdrom Enterprise-R5-U5-Server-x86_64-dvd.iso --status status.txt --mac 52:54:01:12:34:01 --vncport 1
apprun.sh kvm --queue This email address is being protected from spambots. You need JavaScript enabled to view it. --vmname ABDLab02 --cpu 2 --memory 2048M --hda db2-hda.qcow2 --hdb db2-hdb.qcow2 --cdrom Enterprise-R5-U5-Server-x86_64-dvd.iso --status status.txt --mac 52:54:01:12:34:02 --vncport 1

 

Let me explain (you only have to write this once):

apprun.sh kvm --queue This email address is being protected from spambots. You need JavaScript enabled to view it. --vmname ABDLab --cpu 2 --memory 2048M --hda db2-hda.qcow2 --hdb db2-hdb.qcow2 --cdrom Enterprise-R5-U5-Server-x86_64-dvd.iso --status status.txt --mac 52:54:01:12:34:01 --vncport 0 --master

How do you connect?

You can connect to fep using SSH, VNC, FreeNX or SSH + X11 forwarding. From then on, if you know the ip of the machine, you can access it from fep, if not, you can connect to the machine using kvm like this:

[alexandru.herisanu@fep-53-1 db2-image]$ cat start_machines.sh | grep -v ^#
apprun.sh kvm --queue This email address is being protected from spambots. You need JavaScript enabled to view it. --vmname ABDLab01 --cpu 2 --memory 2048M --hda db2-hda.qcow2 --hdb db2-hdb.qcow2 --cdrom Enterprise-R5-U5-Server-x86_64-dvd.iso --status status.txt --mac 52:54:01:12:34:01 --vncport 1
apprun.sh kvm --queue This email address is being protected from spambots. You need JavaScript enabled to view it. --vmname ABDLab02 --cpu 2 --memory 2048M --hda db2-hda.qcow2 --hdb db2-hdb.qcow2 --cdrom Enterprise-R5-U5-Server-x86_64-dvd.iso --status status.txt --mac 52:54:01:12:34:02 --vncport 1

This is how my machine startup looks like. I will run two instances on two separate hosts.

[alexandru.herisanu@fep-53-1 db2-image]$ ./start_machines.sh

Checking /export/home/ncit-cluster/prof/alexandru.herisanu/.ncit-cluster/ ...
Using /srv/ncit-cluster/apprun/kvm as environment ...
Running custom run script ...

JobId: 6732
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
   6732 0.00000 ABDLab01-K alexandru.he qw    02/24/2011 15:01:26                                    2

Qsub command:
qsub -q This email address is being protected from spambots. You need JavaScript enabled to view it. -pe openmpi 2 -N ABDLab01-KVM -o out -e out -cwd /tmp/tmp.DPDOqGl533

Running script:

module load compilers/gcc-4.1.2
module load mpi/openmpi-1.5.1_gcc-4.1.2


sudo /srv/ncit-cluster/scripts/run-vm.sh --vmname ABDLab01 --cpu 2 --memory 2048M --hda db2-hda.qcow2  --hdb db2-hdb.qcow2  --cdrom Enterprise-R5-U5-Server-x86_64-dvd.iso  --status status.txt --mac 52:54:01:12:34:01 --vncport 1 --user alexandru.herisanu


Checking /export/home/ncit-cluster/prof/alexandru.herisanu/.ncit-cluster/ ...
Using /srv/ncit-cluster/apprun/kvm as environment ...
Running custom run script ...
JobId: 6733
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
   6732 0.00000 ABDLab01-K alexandru.he qw    02/24/2011 15:01:26                                    2
   6733 0.00000 ABDLab02-K alexandru.he qw    02/24/2011 15:01:26                                    2

Qsub command:
qsub -q This email address is being protected from spambots. You need JavaScript enabled to view it. -pe openmpi 2 -N ABDLab02-KVM -o out -e out -cwd /tmp/tmp.rhCEktl561

Running script:

module load compilers/gcc-4.1.2
module load mpi/openmpi-1.5.1_gcc-4.1.2


sudo /srv/ncit-cluster/scripts/run-vm.sh --vmname ABDLab02 --cpu 2 --memory 2048M --hda db2-hda.qcow2  --hdb db2-hdb.qcow2  --cdrom Enterprise-R5-U5-Server-x86_64-dvd.iso  --status status.txt --mac 52:54:01:12:34:02 --vncport 1 --user alexandru.herisanu


[alexandru.herisanu@fep-53-1 db2-image]$

Great! They are up and running, and we can see we have those slots reserved. No batch jobs will harm us.

[alexandru.herisanu@fep-53-1 db2-image]$ qstat
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
   6732 0.51409 ABDLab01-K alexandru.he r     02/24/2011 15:01:34 This email address is being protected from spambots. You need JavaScript enabled to view it. .     2
   6733 0.51409 ABDLab02-K alexandru.he r     02/24/2011 15:01:36 This email address is being protected from spambots. You need JavaScript enabled to view it. .     2

You know where you put the vm, so connect using vncviewer and get your ip. Or connect using your ip directly if you've set it up beforehand into dhcp.

How do I shut down the VM?

You can shutdown the vm from the virtual machine. shutdown - h now

Or you can use qdel

[alexandru.herisanu@fep-53-1 db2-image]$ qstat
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
   6732 0.51409 ABDLab01-K alexandru.he r     02/24/2011 15:01:34 This email address is being protected from spambots. You need JavaScript enabled to view it. .     2
   6733 0.51409 ABDLab02-K alexandru.he r     02/24/2011 15:01:36 This email address is being protected from spambots. You need JavaScript enabled to view it. .     2
[alexandru.herisanu@fep-53-1 db2-image]$ qdel 6732
alexandru.herisanu has registered the job 6732 for deletion
[alexandru.herisanu@fep-53-1 db2-image]$
[alexandru.herisanu@fep-53-1 db2-image]$
[alexandru.herisanu@fep-53-1 db2-image]$ qstat
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID
-----------------------------------------------------------------------------------------------------------------
   6733 0.51409 ABDLab02-K alexandru.he r     02/24/2011 15:01:36 This email address is being protected from spambots. You need JavaScript enabled to view it. .     2
[alexandru.herisanu@fep-53-1 db2-image]$

Voilla!

Heri