Steve Glendinning

I just applied a hotfix to my XenSource virtual servers, which caused all the virtual machines to be suspended & resumed (while the xen host rebooted).  This caused a problem on the VMs I’ve upgraded to Debian Lenny: These messages were repeatedly logged:

clocksource/0: Time went backwards: ret=18f26176a5 delta=-8292949932632971 shadow=18e9e0b4b9 offset=880c4ce

This message wasn’t “harmless” – it caused TCP connections to break, so ssh was practically unusable.

The immediate solution: using the XenCenter console, forcefully reboot the affected VM. That’s “force shutdown”, followed by “start”. I tried the simple “reboot” option, but it wouldn’t shut down (probably because it was confused over the time). VMs are now back up and there’s no sign of time going backwards.

It seems this was triggered by suspending & resuming machines running the Debian Lenny kernel. Other VMs which are still running the 2.6.18 based XenSource kernels survived suspend/resume ok.

The actual cause is documented here, here, here & here. I think I’ll be following that advice and decoupling my VM clocks from the host (using NTP to keep them all in sync the old-fashioned way).

Debian Lenny has been released, so I’ve started upgrading some of my Xen virtual servers.  For Etch, Xensource shipped their own modified PV kernel package (2.6.18 based).  With Lenny the same kernel can be used, or the standard Debian xen kernel (2.6.26 based) can be used instead.

Upgrading the userland is identical to a physical server – instructions are contained within the Debian Lenny release notes.

To use the Debian packaged 2.6.26 kernel, several minor changes have to be made as per this form post:

1. Install the linux-image-2.6-xen-686 package (which depends on the latest xen kernel)
2. Re-order the kernels in /boot/grub/menu.lst so the Debian 2.6.26 kernel is the first (and therefore the default)
3. Add “console=hvc0″ to the end of the 2.6.26 kernel line in /boot/grub/menu.lst, so it reads “kernel /boot/vmlinuz-2.6.26-1-xen-686 root=/dev/xvda1 ro console=hvc0“
4. edit /etc/inittab, find the line that looks like “1:2345:respawn:/sbin/getty 38400 tty1″ and add another line: “0:2345:respawn:/sbin/getty 38400 hvc0″

Then reboot the system, it should boot the new kernel.

Xensource doesn’t display the grub menu, it just seems to boot straight into the default option (even though I have “timeout 10″ specified in the config).  There must be a secret keypress or something to make it show the menu, but I don’t know what it is because the last step just worked for me!

Edit: Theoretically you shouldn’t need to use the -xen kernel variant, as the -686-bigmem kernel also supports running paravirtualised in domU.  With Xensource 5.0 this boots fine, but Xensource’s xen-tools refuse to start so XenCenter reports the status as “unknown”.  With the -xen kernel xen-tools start fine.

Vyatta 3.0.3 (and its community equivalent VC4) boots fine on Virtual PC, which is really handy for firing up a “test” router.  Unfortunately, the latest subscription release 3.1.3 (and probably VC5) fails shortly after the bootloader prompt with this error:

Vyatta Virtual PC boot error

The solution is to pass the linux kernel boot argument noreplace-paravirt.  This is done on the live CD by typing “live noreplace-paravirt” and pressing enter (instead of simply pressing enter or waiting for the CD to automatically boot).

When you’ve installed to the virtual hard disk (using the install-system command), you’ll find it displays the same error.  To boot the first time you’ll need to press Escape to access the grub menu, then press “e” to edit the default entry.  Add the word noreplace-paravirt to the end of the line (after “console=tty0″) and press ctrl-x to boot.

When you get the login prompt, you can log in as root and edit the grub config file (nano /boot/grub/grub.cfg) to add the option to the command line permanently.