VM Failover and High Availability
A VM's disk lives on the server that runs it, encrypted inside the confidential VM. If that server dies, the VM can be restarted on another server in the same region from its latest backup. This is called a failover. It brings the VM back, but it is not high availability: the VM is down for several minutes and loses what it wrote since that backup.
How failover works
- The VM is restored from its newest backup of the current boot, on another server in the same region.
- Data written after that backup is lost. The most you can lose (the recovery point objective, or RPO) is the backup interval you chose: 24 hours, 6 hours, 1 hour or 15 minutes. If backups were failing or out of date, the newest backup is older and you lose more.
- No backups, no failover. A VM without a restorable backup can't be failed over. Turn on scheduled backups on any VM you want to be able to recover.
- It takes a few minutes. A failover typically takes about 5 to 10 minutes for a 40 GiB disk.
A reboot starts a new backup chain. Until its first full backup completes, the VM has no backup of its current boot, so it can't be failed over.
What is kept
- The same VM, with the same ID.
- The same size and the same region.
- Its public IPv4 address, if it has one.
- Its address on your private network (NetBird).
The VM's encryption keys are released to the new server only after it passes attestation, as at every boot. The old instance is permanently fenced: even if its server comes back, it can never run alongside the new one.
Manual and automatic failover
There are two modes:
- Manual, the default. When the server running your VM is confirmed dead, Hippius support can fail your VM over, for example after you open a ticket.
- Automatic, opt-in, per VM. Once the server has been proven dead for 15 minutes, the VM is restarted on another server in the same region from its latest backup. The server must first stop answering before it can be proven dead, so expect up to about half an hour between the outage and the start of the failover, then a few minutes for the restart.
Where automatic failover is offered
Automatic failover needs a pair of compatible servers in the region, so that one can restart the VMs of the other. Today, it is offered in France only:
| Region | Automatic failover |
|---|---|
| France | Offered. |
| Netherlands | Not offered yet: no pair of servers in this region can fail over yet. |
| Australia | Not offered: backups there aren't stored in the region. |
Where it isn't offered, support can still fail a VM over manually if it has backups.
Turn on automatic failover at creation
When you create a VM, turn on Automatic backups, then the Automatic failover switch under them. The switch is shown only once you pick a region where automatic failover is offered. In another region, the form says why instead, for example Automatic failover isn't offered in Netherlands yet: no pair of servers in this region can fail over yet. If you leave the region on automatic placement, it asks you to pick a region where it is offered.
The summary then lists Automatic failover: On. A VM launched with automatic failover is placed on a server that has a failover partner. Until the VM is running, the Failover section of its Backups panel says Automatic failover was asked for at launch: it is turned on with the backups once the VM is running. If it can't be turned on then, the section says why, for example Automatic failover could not be enabled: … followed by Backups are on; failover stays manual.
If the region can no longer offer automatic failover when you launch, the launch is refused with Automatic failover can't be turned on for this launch: launch without it, or pick another region.
Turn on automatic failover on an existing VM
On the VM's page, in the Backups panel, the Failover section sits between Scheduled backups and Restore points. Turn on Automatically restart on another server if this server goes down. The switch is hidden when automatic failover isn't available at all.
Automatic failover needs at least one complete backup, a region where it is offered, and a server that has a failover partner. When it can't be turned on, the toggle is disabled and the console says why:
| Message | What to do |
|---|---|
| Turn on backups to use automatic failover. | Turn on scheduled backups. |
| Available once the first backup is complete. | Wait for the first full backup to finish. |
| Not offered in this VM's region yet. | Automatic failover isn't available in this region. Support can still fail the VM over manually. |
| Not available on this VM's server: no compatible server in its region to restart it on. | The VM's server has no failover partner. Support can still fail the VM over manually. To get automatic failover, launch a new VM with it turned on: it is placed on a server that has a partner. |
| Not available yet. | Automatic failover isn't available yet. |
Automatic failover is active only after the first full backup. Until then, and whenever one of these becomes true later, for example after a reboot, the toggle stays on but shows a warning such as On, but not effective until the next backup completes. The VM can't be failed over automatically until the cause is gone.
Turning off backups also turns off automatic failover. Turning backups back on doesn't turn it back on.
During and after a failover
While a failover runs, a banner at the top of the VM's page says Restarting on another server, from which backup, and whether it was started automatically or by Hippius support. If no server in the region has room for the VM, the banner says Waiting for a server with capacity: the failover waits and retries until one does. The VM's status shows each step, for example Restarting on another server · Preparing. You can't restore the VM from a backup or resize it until the failover is done.
If the original server comes back before the VM has been handed over to the new server, the failover is cancelled: nothing changes, the VM keeps running where it was, and no email is sent. The Failover history shows it as Not needed, server came back. Once the handover has started, the failover goes on even if the original server comes back.
Once it is done, the banner says Restarted on another server, with the backup used, for 7 days. The Failover history table in the Backups panel lists the VM's 10 most recent failovers: when it started, its type (Automatic or By support), its status, the backup used and how long it took.
Whichever the mode, you get an email once your VM has been failed over, saying when, why, and which backup was used. A cancelled failover sends no email.
What failover costs
A failover doesn't change what you pay for the VM. It is billed once, continuously: the old server stops at the cutover, and the new one takes over. Backup storage is billed as usual; see What backups cost.
High availability
High availability is your responsibility. A single VM runs on one server, and a failover means minutes of downtime plus data lost back to the last backup. To keep a service up when a server dies, run it on several VMs, with your own load balancing and data replication between them, as on any cloud provider.
Placement groups
A placement group keeps your VMs on different servers. Two VMs of the same account in the same placement group are never placed on the same server.
- Set it when you create a VM: in the Finalize details step, open Advanced options and fill in Placement group (optional). Through the API, it is the optional
placement_groupfield. A name is 1 to 64 lowercase letters, digits and hyphens, for exampleweb-tier. Any other name is refused. - It is fixed at creation. You can't add a VM to a group, or move it to another, later.
- The form suggests the groups you already use, with how many VMs each has. When placement groups aren't available, the form has no Advanced options, and a launch that sets one is refused with Placement groups can't be used right now: launch without a placement group, or try again later.
- If no server can take the VM apart from the other VMs of its group, the launch fails with
placement-anti-affinity-unsatisfiable. Launch it in another group, or without one.
A placement group only guarantees different servers. Two servers can be in the same datacenter, and a datacenter outage can take both down.