From Granizada

Jump to: navigation, search



To build a Debian Etch server from scratch that is a minimal Xen 3 Dom0, then start populating it with DomU's.

This recipe is the simplest I know of and was designed as a teaching tool, however the result is good enough to run production services in many cases.

The server will of course be accessible over the network. In addition you may want to choose one of these:

  • serial console access, most likely what you'll do when putting a Xen server in a datacentre
  • X capability, able to display screens from DomUs on the console, useful for installing a physically local server

Minimal is a good goal because it reduces risk a lot, and is easily documented for repeatability. It also help in debugging if systems are less complicated. Xen can be made very stable, but it automatically veers to the unstable if you don't take care, mostly due to lack of polish.

This is also minimal in another sense, the setup isn't optimised for speed at all. Especially, you don't want to use loopback mounted files for DomUs if you care about speed (and occasionally, stability.) However I'm using this as a training setup and I find it much better than jumping straight into blktap, LVM, nbd and more.

I'm keeping abreast of the upgrade pitfalls over time as I find them.

This is something you can do if you have a dedicated host somewhere like Bytemark . I am using this sort of setup in several places (and some much more complicated ones, which are correspondingly more fragile.) I am also testing KVM and other solutions that are much better-designed than Xen.

Dan 06:07, 18 April 2007 (CST)

Etch System

Minimal etch install: Step through the installer as usual then, at the Debian Software Selection screen make sure all options are deselected (by default, Desktop Environment and Standard System are selected. You don't want them.) Make sure you have specified a Debian package repository. Make sure you enable serial support as follows or you will have problems configuring the server after the first boot, if you want to access it via a real or virtual serial console.

Installing Etch with Serial Support

Only read this if you want to be able to access your server over a serial connection (that is, not over a network. Handy for when kernel upgrades don't go as planned.) Serial access isn't the default in Etch.

After you've gone through the installation steps and before selecting Finish the Installation, exit to a shell prompt and make the following changes to make sure you have serial access after rebooting. Most of the following files will not be as full as you would expect in a normal Linux installation. That's because the installation hasn't quite finished, but you don't get a later opportunity to launch a shell before the installer exists and reboots, and if your machine is locked in a datacentre things might get difficult.

I have used the first serial port (/dev/ttyS0) in the following. You may need different port or speed settings.

In the global section of /boot/grub/menu.lst :

 serial --unit=0 --speed=115200
 terminal --timeout=15 serial console

In the kernels section, append com1=115200 and console=ttyS0,115200 as follows:

 kernel          /boot/xen-3.0.3-1-i386.gz com1=115200
 module          /boot/vmlinuz-2.6.18-3-xen-k7 root=/dev/md1 ro console=ttyS0

If you are following these instructions step-by-step, you won't have a xen kernel in menu.lst yet. Append these values to whatever your current kernel is, and you are less likely to be locked out when you do install xen kernels as per instructions further down the page.

The com1= parameter is hardware-dependent, you may not need it. The serial console feature is a default in Xen, it grabs the first serial port from the hardware. However in this case we have to tell it the speed or it is stubbornly silent, grub works but the kernel doesn't.

In /etc/inittab, add :

 S0:12345:respawn:/sbin/agetty 115200 ttyS0 vt100

The Debian installer will create a secure tty file when finishing the installation, so don't get worried if you notice it doesn't exist yet. Without an entry in /etc/securetty root will not be allowed to log on from the serial port.

And if you get it all wrong before rebooting into a new kernel? If you're doing a local install then you can just start again. If your server is in a datacentre and you have serial access, use it :-) Some datacentres, including Bytemark, provide PXE boot servers. PXE also knows about serial terminals so you have the next-best thing to control at the BIOS level. If the PXE menu lets you boot Linux then mount your local drive and edit away :-)

After First Boot

You should be able to login at the serial console (or over the network if you set this up in the install, but it is perhaps wiser not to.)

 apt-get update
 apt-get dist-upgrade 

The dist-upgrade is not strictly required right now, but you want to know if there is a fundamental package problem, not later when you may get confused with other problems. Now:

 apt-get install ssh openssh-server gpm less telnet nmap tcpdump file links

(If your server is in a datacentre you don't need gpm.)

This is the minimal set of generic tools for a useful text mode virtualisation host. Or any other minimal networked server for that matter. Always include links even if it isn't connected to the Internet, for reading local package documentation. On local servers (not hosted servers where you only have serial console access) gpm is highly useful for cut/paste between virtual consoles. Consider adding vim to the package list simply because it has internal cut/paste buffers, unlike plain vi. These save time and reduce errors. (Consider adding rcs for managing versions of text files, although some dislike it!)

Edit the file /etc/modules and append the single line if it doesn't appear already:


which will modprobe loop at every boot. (Already in etch by default) To make it do so with the correction options, create the new file /etc/modprobe.d/local.loop containing:

 # Set high for Xen
 options loop max_loop=255

now execute:

 rmmod loop
 modprobe loop

dmesg | tail should show a line that says loop: loaded (max 255 devices).

Now, if you're on a server hosted at a datacentre where they have paranoid port security switched on you'll need to enable proxy arp for eth0 for reasons explained in the comments below. This is in fact the wrong file (because it applies to all interfaces) but seems to be as legal a place as any in Debian.

 cat > /etc/network/if-up.d/tcpopts
 # turn on proxy arp on eth0 to answer DomU requests, otherwise Xen
 # bridging code sends out made-up MAC addresses, triggering port security on the 
 # datacentre switch and cutting us off.
 echo 1 > /proc/sys/net/ipv4/conf/eth0/proxy_arp


 chmod 755 tcpopts


Now install dom0 and tools. You will probably need to adjust the kernel version number in the following, valid in April 2007. and perhaps the hardware type. Use apt-cache search xen-linux-system to give you the current options. You don't want to install a kernel flavour with vserver in the name, that is yet another form of virtualisation to confuse matters!

apt-get install xen-linux-system-2.6.18-4-xen-686 \
                xen-utils-3.0.3-1 \
                xen-docs-3.0 \
                bridge-utils \ 
                xen-tools \

Reboot. See if xm list gives a sensible report about dom0, such as:

 Name                                      ID Mem(MiB) VCPUs State   Time(s)
 Domain-0                                   0     3404     2 r-----      7.6

If you are running a modern AMD machine with SVM (Secure Virtual Machine) support, xm dmesg | grep SVM should give something like:

 (XEN) AMD SVM Extension is enabled for cpu 0.
 (XEN) AMD SVM Extension is enabled for cpu 1.
 (XEN) AMD SVM Extension is enabled for cpu 2.
 (XEN) AMD SVM Extension is enabled for cpu 3.

Alternatively you can look for a match for grep svm /proc/cpuinfo.

On a modern Intel machine with VMX (Virtual Machine Extensions) support, you can look for a match for grep vmx /proc/cpuinfo

Fix any errors, eg machine freezes with Xen kernel panic. Ok, so that's trivialising it a little.

Uncomment the line in /etc/xen/xend-config.sxp :

(network-script network-bridge)

Need to reboot after this or the vif virtual network interfaces won't come up. Request: can someone following this recipe please see if all of the above can be combined into a single reboot (probably can.)

Preparing to Build DomUs

A DomU (user domain, terminology specific to Xen) consists of:

  • a Linux kernel with the Xen DomU patches, patches which are not part of Linux and unlikely to ever be so (as it seems in April 2007)
  • a Xen configuration file
  • a bootable image. This can be a disk partition, or a file containing a Linux filesystem, an LVM logical volume or other things including an iso-format image of a CD

You will already have the Xen DomU kernels from installing the Etch Xen packages.

If you don't already have Xen configurations and a bootable image, you can use xen-tools by Steve Kemp for making DomUs . xen-tools is not required for Xen to work, but if you're new to Xen you will find it an easy way to start. The rest of this section assumes you are using xen-tools.

Edit /etc/xen-tools/xen-tools.conf according to taste. There's support for Ubuntu, Centos etc.

 xyz:~# grep -v '^#' /etc/xen-tools/xen-tools.conf | grep -v '^$'
 dir = /home/xen
 debootstrap = 1
 size   = 4Gb      # Disk image size.
 memory = 128Mb    # Memory size
 swap   = 128Mb    # Swap size
 fs     = ext3     # use the EXT3 filesystem for the disk image.
 dist   = etch    # Default distribution to install.
 image  = sparse   # Specify sparse vs. full disk images.
 dhcp = 1
 kernel = /boot/vmlinuz-2.6.18-3-xen-k7
 initrd = /boot/initrd.img-2.6.18-3-xen-k7
 mirror =

Then start the daemon:

 /etc/init.d/xend start

and check for errors under /var/log/xen/ (might want to think about upping logging in general.)

Building DomUs

The most basic command is:

xen-create-image --hostname=firstxen --passwd

adding options such as --ip --gateway and --netmask only if you haven't specified it in xen-tools.conf , or to keep to your standards. For example:

 xen-create-image --hostname splendid --mac  00:16:3E:AA:AA:112  --ip

Xen DomUs and Generated MAC Addresses

If you're connected to a switch that has paranoid security settings such as run by Bytemark, then due to the issue noted in BytemarkDedicatedHost under Hosing the Switch Port, you'll need to either:

  • enable proxy_arp and make sure it stays enabled, and that eth0 isn't on any virtual bridges, or
  • specify a unique MAC per DomU

I prefer to create a specified MAC because it gives so much more control. You can:

  • distinguish among VMs when doing packet traces just by MAC
  • filter intelligently in ebtables
  • let's DHCP work the way it was meant to for VMs

To do this add a line such as:

 vif = [ 'type=ioemu, mac=00:16:3e:00:00:01, bridge=xenbr0' ]

where there is a numbering scheme in the last three bytes of the MAC that makes sense to you.

This is where you start be glad of a VM provisioning script to remove even more of the drudgery.

All In One Script

All of the foregoing can be done by pasting the following into root console on a minimal Etch install, noting that these are very specific versions that may well not apply by the time you read this:

apt-get update -y
apt-get dist-upgrade -y

# if you're going to use loopback images with xen-tools
mkdir /home/xen

# maybe add gpm if you have a local server
apt-get install ssh openssh-server less telnet nmap tcpdump file links -y

echo  "# Set high for Xen" > /etc/modprobe.d/local.loop
echo "options loop max_loop=255" >> /etc/modprobe.d/local.loop

modprobe loop

apt-get install xen-linux-system-2.6.18-4-xen-686 \
               xen-utils-3.0.3-1 \
               xen-docs-3.0 \
               bridge-utils \ 
               xen-tools \
echo "(network-script network-bridge)" >> /etc/xen/xend-config.sxp

Start DomU

 xm create firstxen.cfg

Don't forget any extension such as .cfg.

If you get errors about vif interfaces make sure you really did modprobe loop since changing the options, and that brctl show doesn't return errors (such as no brctl at all.) It might also be a problem with udevd although Etch is very well behaved with udev now. This is where a lot of new Xeners get stuck!

Started without errors? But can't get in because the network is broken?

 xm console firstxen

is the virtual console (or more like, the serial console.) Fix networking, make sure you can ping out then try to get in again from the host.

The Xen Dom0 kernel sometimes panics at this stage. Almost always for a fairly simple reason, just undocumented.

Kernel Crash Bug with Debian Xen Linux kernel 2.6.17-2

This shouldn't apply to you if you're installing Etch as of February 2007 or later because you will be using a later kernel.

There is a problem with the Xen patches applied to Debian Linux kernel 2.6.17-2 which can cause the Dom0 to reboot when you start a second DomU . Kernels 2.6.16 and 2.6.18+ are fine.

To confirm you have this bug comment out the line

 (network-script network-bridge) 

in /etc/xen/xend-config.sxp and also the


in the individual DomU configuration files. Now start any two DomUs. If they boot ok but 2.6.17 otherwise crashes then it looks like you have this bug which is not at all diagnosed, but does appear to be fixed.

One working fix is to move to 2.6.18+, although Etch is unlikely to ship with 2.6.18 enabled by default even now from the unstable (Sid) distribution 2.6.18 seems to be solid. And now in December 2006 2.6.19 is in wobbly pre-release so that's all going to be history soon anyway :-)

Kernel Crash Non-bug with Debian Xen Linux kernel 2.6.18-4

Debian bug 410480 contains complaints from people who upgraded from linux-image-2.6.18-3-xen-686 to linux-image-2.6.18-4-xen-686 . If you do this your previously working Xen dom0 will stop and by default it will flash a message on the screen "PAE mode mismatch error (xen=yes dom0=no)" and then reboot after a few seconds. This is perfectly normal :-)

The problem is that the new kernel has been enabled for pae and the Xen dom0 also needs to be enabled for pae. So in silently enabling pae on the kernel and not automatically updating the dom0 image Debian has caused a guaranteed crash. This was quite bad customer relations, but in fact it did protect from more subtle instability as illustrated in Debian bug 399113 . Etch is unstable, and it is only instances like this that show it really is unstable, since most of the time Etch works well.

The fix is to boot under the previous working kernel version and:

 apt-get install xen-hypervisor-3.0.3-1-i386-pae

which is then compatible with the kernel, and Xen should boot happily as before.

If you look at the flags listed in /proc/cpuinfo you should see "pae", assuming you have the right hardware (very many people do.) Some people will either have to recompile the kernel image without pae or buy new hardware. I'm not sure why the package xen-hypervisor-3.0.3-1-i386 (non-pae) would still exist, presumably it will be purged since if people were to be encouraged to used non-pae a non-pae kernel would be provided.

No K7 Flavour of Xen as of Debian kernel 2.6.18-4

Debian bug 405187 is a classic badly-answered question, but it is correct: if you have an AMD system and you don't want to compile your own kernels then you will need to use the 686 version for anything after kernel 2.6.18-3. In other words, instead of using linux-image-2.6.18-3-xen-k7 you will need to use linux-image-2.6.18-4-xen-686 regardless of your hardware type. There is no such thing as linux-image-2.6.18-4-xen-k7 nor will there ever be, according to bug 405187. Debian Etch is a testing distribution so lots of things may change, however there is usually good communication about why. Not this time.

The way to do this safely and en-masse in a Xen system that has lots of VMs is:

  • symlink /boot/vmlinuz-..-k7 to /boot/vmlinuz-..-i686 for both the kernel and initrd. These images are normally pointed at by your xen .cfg configuration files. Linux doesn't care what the filename is called that it boots from, the Linux kernel has other ways of knowing its own version. So all your VMs should automatically boot with a 686 kernel instead of a k7 one.
  • apt-get install linux-modules-2.6.18-4-xen-686 inside each VM. You need this if you want any modules to load, because you need modules that match your kernel. For example, if you use iptables to maintain a firewall, you'll normally need modules or iptables won't work at all.
  • if all that works you can remove the symlinks and the old packages, but if there are any problems in the meantime you can just break two symlinks and be back exactly where you were with -k7 kernel and initrd. There are extra modules on disk in each VM but that doesn't matter because they aren't loaded until there is a matching kernel present.

X on Base Etch Dom0

Not strictly virtualisation, but if this is on hardware out on-site somewhere you can make a nice minimal GUI server:

 apt-get install xserver-xorg fvwm-gnome xterm xfonts-base

which will give you well enough configured that you can run most modern apps. Certainly Xen DomUs and VMware server are fine. There is no display manager such as xdm or gdm here so you will always have to log in text mode and then start the local X server if and when you want it.

If you need it:

 apt-get install firefox xfonts-terminus ttf-freefont

Which proves the X is reasonably featured (some windows don't have borders but really does it matter? :-)

Creative Commons License
Creative Commons Attribution iconCreative Commons Share Alike icon
This content is licensed under the Creative Commons
Attribution ShareAlike License v. 2.5:
GNU head GFDL: Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled "GNU Free Documentation License". ( uses but does not currently recommend the GDFL and here's the explanation why. )
Personal tools