At approximately 10:15 this morning, pve-1’s hardware watchdog detected a kernel issue and caused the hypervisor to reboot. The kernel issue related to an automated security patch through the KernelCare package which applies security updates in real time, supposedly mitigating any downtime.
A QEMU VM handler thread was actively executing inside apic_timer_fn (the APIC timer interrupt handler) when kpatch tried to atomically redirect it. The consistency check failed because it couldn’t safely swap the function with a live thread inside it.
As a result of the unscheduled and unexpected reboot, a number of key services operated by ourselves and our upstream supplier (FidoNet) were briefly impacted. Normal services were restored by 10:46am.
We are discussing with Fido ways in which they can better manage automated updates, including limiting them to non-critical service hours. A busy hypervisor like pve-1 is always going to have problems with diverting apic timer functions and whilst the kernel patches are necessary from a security standpoint, there can be a tradeoff with timings to ensure there are no failures during business hours.
Fido are in fact currently scheduling server upgrades and reboots in order to apply updates and patches, and these will most likely happen overnight if urgent or at the weekend if less critical. We will post more information once we have it.
Apologies to anyone impacted by todays brief outage.