I recently purchased a dedicated host from Bytemark. A dedicated host is a rented server, where you have full use of the hardware and have exclusive rights to access the hardware. Many people who rent dedicated hosts take the default install and carry on from there. They will have a very different experience from someone who wants to be as independent as possible of the hosting company, as I do. That means I am testing the limits of the service that Bytemark provides, limits most business customers will never see. Matthew Bloch of ̆Bytemark has been very forthcoming in describing Bytemark's current situation and where they are headed.
This article covers:
- What you get, and how the dedicated host management features work
- How to get a basic Etch setup (Sarge won't see the drives easily and besides, Etch is nearly released ho ho.)
After this you are ready to build a minimal Xen Dom0 on Etch, at which point you should have a functional Xen server at Bytemark.
Bytemark provide a range of solid range of facilities, a bit on the unpolished side. Staff are knowlegeable and friendly. This article is exclusively about the technical customer experience, not the business side (which feels exactly like a little company with a strictly technical background. Fine by me.)
What you Get from Bytemark
- A root password for an installed server of your preferred distro. Since this only happens once per server setup I've only been able to test with Etch. You also get an IP address, and sshd is listening.
- A private key for the remote console, called dshell. The private key is in /root on the delivered machine. Make sure you copy it somewhere for safekeeping!
- The supplied installation is done manually and there is no documentation available on how it was done. There is no way to re-initalise to the delivered state, or instructions for doing an identical install: in short, the customer has no idea what has been delivered. So a re-installation is required in order to run a known service.
- There is no documentation. However poking around with dmesg, fdisk, mdadm and friends should show you what the hardware is and how its set up. For example, as of December 2006 all the Bytemark servers used SATA drives which don't work well with all fdisks, but this is a well-known problem (now well-solved in 2007.)
- Write down the network config. They're using CIDR (of course) so depending on how much network space you've bought your netmask won't be easily guessed and it isn't in any other documentation you get from Bytemark. The server is also set up to DHCP which is another way of getting the network information if you forget it.
Bytemark are very responsive to datacentre problems, such as network outages. They do not try to hide information, and judging from the hit count on the Bytemark infrastructure support forum when there's problems customers are also on the ball when something goes wrong. Neither do Bytemark seem to redact forum history after the event like some do, which is all reassuring.
Bytemark's Manchester facility seems to be somewhat less reliable than its London one (anecdotal evidence from talking to customers, and looking at the forums. Bytemark have also committed to finding a third bytes supplier in Manchester so hopefully that is a thing of the past.
How Bytemark Hosted Servers Boot
PXE boot in the BIOS is enabled. It does a DHCP request, then loads the TFTP image specified from a Bytemark server somewhere. The image is PXELINUX from HPA, like SYSLINUX for network boot. That means it isn't very intelligent but is fast to boot and reliable.
Bytemark have made a simple PXELINUX menu that points off to a number of bootable images. The selected image will be booted over the network. The default is to boot from local disc after a delay of about 10 seconds. If you powercycle the server don't forget to keep watching with serial or your dead machine will try to boot and you'll have to powercycle again.
PXELINUX 3.11 2005-09-02 Copyright (C) 1994-2005 H. Peter Anvin . . . Welcome to Bytemark's boot server! Please type: local to boot off your hard drives (default) netboot-rescue to run our Knoppix rescue system with a default set of drivers (help! my kernel upgrade went wrong!) install-fc3 to run the Fedora Core 3 installer install-centos4 to run the Centos 4.2 installer install-debian to run the Debian installer install-debian-etch to run the Debian install (testing) install-freebsd to run the FreeBSD installer / rescue install-ubuntu-breezy to run the Ubuntu 5.1 installer memtest86+ to run memtest86+ (check for bad RAM) boot: Booting from local disk...
You can't see the above boot screen unless you are connected to the serial console. Connecting to the serial console is one of several commands you get access to by the Bytemark dhshell facility.
- Use a command similar to the following to access the shell: ssh -i keyfile email@example.com
- dhshell might be a Linux machine running sshd and all accounts linked to a limited, static binary.
- There are currently just four things dshell lets you do: powercycle, serial, watchdog and quit. There is no facility for reset (as opposed to powercycle, unless that is what the reboot watchdog command means) nor external temperature or other things. But it covers the most important points, like everything else Bytemark does.
Using the Serial Console
You use the serial connection from the dhshell prompt.
- type serial. This opens a connection to the Cisco serial terminal server.
- you will be presented with a login: prompt. This is because /etc/inittab has been configured in the Bytemark image so that getty is listening on the first serial port. login with the same root password as ssh.
- The Cisco times out and closes connections after 15 minutes or so of inactivity. Make ^L the first command you type when connecting to avoid unexpected surprises, followed by CR if nothing happens (getty waits for CR before saying anything back.)
- the serial connection runs at 115200bps
- Although multiple people can ssh to dshell (an essential feature, see Watchdog section below for how to use it), only one can telnet to the Cisco at a time and the error the second person gets is the same as if the terminal server isn't working. Bytemark hadn't thought of this (having multiple admins in multiple locations, for example) and don't feel it is a priority bug ("Nobody has ever said it was a problem before" ... :-)
The watchdog provides a good set of features (thanks Bytemark!), think of it as a cut-down commandline Nagios.
The usefulness is somewhat limited because we have no information about how many fallible network hops there are between the watchdog server (which might be the same as the dhshell server, or might not) nor about the likely reliability of the watchdog service itself (eg are there multiple servers that do failover? What IP address do we watch to be alerted when the watchdog server goes down and therefore our Bytemark servers are at increased risk?)
You can ssh to the dhshell from your own server. This is useful because you have a machine close to dhshell in network terms which can be used (in conjunction with an external server) to alert you when, worse than the watchdog server, the dhshell server goes down and therefore your server is at increased risk because there is no recovery mechanism in place. Similarly it would be good if there was a way of watching the Cisco terminal server, because without that again the server is at increased risk.
There isn't an advertised limit on the number of addresses that can be watched. You might consider watching other parts of the Bytemark infrastructure and other people's servers hosted with Bytemark for the potentially useful information you can get about uptimes and outages.
The following is entirely from the online help at Bytemark's dhshell server:
<xyz.dh> help watchdog Syntax: watchdog add <test name> HTTPFetch <ip> <port> <vhost> <path> <status> <oktext> action <sms|email|reboot> ip : IP address to check port : Port number to try vhost : Virtual host to query server for path : Path to query server for status : Expected status code (usually 200) oktext : Text to check for in response Syntax: watchdog add <test name> TCPConnect <ip> <port> action <sms|email|reboot> ip : IP address to check port : Port number to try <xyz.dh> help watchdogchecks Available watchdog checks: Syntax: watchdog add <test name> Ping <ip> action <sms|email|reboot> ip : IP address to check Syntax: watchdog add <test name> TCPBanner <ip> <port> <okbanner> action <sms|email|reboot> ip : IP address to check port : Port number to try okbanner : Banner string to check for Syntax: watchdog add <test name> HTTPFetch <ip> <port> <vhost> <path> <status> <oktext> action <sms|email|reboot> ip : IP address to check port : Port number to try vhost : Virtual host to query server for path : Path to query server for status : Expected status code (usually 200) oktext : Text to check for in response Syntax: watchdog add <test name> TCPConnect <ip> <port> action <sms|email|reboot> ip : IP address to check port : Port number to try <xyz.dh> help watchdogexamples Examples of watchdog checks: <mymachine> watchdog add "Machine up" Ping 126.96.36.199 action reboot This will reboot your machine if it stops responding to pings
Security problems in dhshell
The following three problems have been raised with Bytemark for some time before being written about here:
- last login information is shared among all dhshell users (a form of advertising to existing customers??)
- The Debian kernel reported by dshell here has at least one critical local root security issue (Debian #378324). On one hand this is not good, on the other at least we know about the bad situation thanks to the motd saying so. On the third hand, an apt-get install would fix this.
- dhshell is closed source, and since it appears to be a captive shell (not a low-risk undertaking) that isn't ideal. Which makes the known local root issue worse. Bytemark are intending to write a new and better dhshell and to opensource it, however this is some time off (discussion with Bytemark as of January 2007.)
Evidence of the first two points above, with a special guest appearance by a bastion host at Sun EU:
$ ssh -i mydhshellaccess.key firstname.lastname@example.org Linux dhshell.man 2.6.8-3-686 #1 Thu Sep 7 03:38:22 UTC 2006 i686 GNU/Linux The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright. Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. dhshell.man is configured via CFEngine Last login: Thu Mar 8 14:20:59 2007 from beryllium.eu.sun.com | Dedicated host shell for asno.dh | Type 'help' for help on commands <asno.dh>
Hosing the Bytemark Switch Port
More on this in MinimalEtchXen, but be aware that the port security on the Bytemark switch is set to trigger on bogus MAC addresses. And by default Xen creates random MAC addresses of the form 00:16:3E:xx:xx:xx unless a MAC is specified in the DomU config. So your first xm create could cut you off at the knees. You can get in via serial port provided you've enabled this access, however you cannot regain network access without asking Bytemark to re-enable the switch port. If you don't have serial access enabled on the server then give up, because you can't boot PXELinux since it relies on the switch port. You don't have a choice but to contact Bytemark. There's other ways to trigger this but the Xen one will happen by default.
CFEngine is mentioned in the banner of the original Etch image installed on the server and in dhshell. That's good, and it will be nice to see CFEngine's configuration facilities used to give Bytemark customers appropriately customised default images, for example with serial support configured in and optionally ssh but without what I recall to be a quite large assortment of other software. Bytemark say that something of the sort may happen at some point.
| || This content is licensed under the Creative Commons|
Attribution ShareAlike License v. 2.5:
|GFDL: Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled "GNU Free Documentation License". (shearer.org uses but does not currently recommend the GDFL and here's the explanation why. )|