Time went backwards
I just applied a hotfix to my XenSource virtual servers, which caused all the virtual machines to be suspended & resumed (while the xen host rebooted). This caused a problem on the VMs I’ve upgraded to Debian Lenny: These messages were repeatedly logged:
clocksource/0: Time went backwards: ret=18f26176a5 delta=-8292949932632971 shadow=18e9e0b4b9 offset=880c4ce
This message wasn’t “harmless” – it caused TCP connections to break, so ssh was practically unusable.
The immediate solution: using the XenCenter console, forcefully reboot the affected VM. That’s “force shutdown”, followed by “start”. I tried the simple “reboot” option, but it wouldn’t shut down (probably because it was confused over the time). VMs are now back up and there’s no sign of time going backwards.
It seems this was triggered by suspending & resuming machines running the Debian Lenny kernel. Other VMs which are still running the 2.6.18 based XenSource kernels survived suspend/resume ok.
The actual cause is documented here, here, here & here. I think I’ll be following that advice and decoupling my VM clocks from the host (using NTP to keep them all in sync the old-fashioned way).
Tags: Debian, Lenny, Virtualisation, Xen
September 10th, 2009 at 10:48 am
Thanks for this – I’ve noticed this at a couple of my VPSs. Decoupling the domU timesource worked perfectly here, so I’ll stick with that.
I didn’t notice this error when I was on the Etch kernel.
February 8th, 2010 at 8:10 pm
Thanks for your well researched report on this bug. This bug had been bugging me for weeks.