What I Learned Running a Proxmox Cluster on Dedicated Servers for a Year
The networking mistakes, quorum failures, and storage choices that cost me a weekend each.
Three nodes. 48 cores, 384 GB RAM, 12 TB of NVMe. 41 running VMs and containers. 99.94 percent availability across twelve months, and one 47-minute outage that was entirely my fault.
Total cost: $487 per month.
The equivalent capacity on a major cloud, at on-demand pricing, prices out at roughly $3,900 per month.
Three lessons stand above the rest.
Corosync is a latency problem before it is anything else, and it decides your entire architecture.
Ceph on three nodes is possible but it is not what Ceph is for.
And high availability is not a backup, which sounds obvious right up until you delete the wrong volume on a replicated dataset and watch the deletion replicate.
Why bare metal instead of cloud VMs
Running Proxmox inside a cloud VM means nested virtualization, and nested virtualization is where performance goes to die. Most hyperscaler instance types do not expose hardware virtualization extensions at all. The ones that do, or the bare metal instance families, cost more than a dedicated server does.
When I benchmarked a nested setup during planning, guest CPU throughput came in 22 percent below the same workload on bare metal, and disk latency was worse than that.
The cost math decided it before the benchmarks did.
My three nodes deliver 48 dedicated cores at $487 per month, which works out to $10.15 per core per month.
Comparable dedicated vCPU capacity on a hyperscaler, before storage and egress, runs several times that.
A host has to meet three requirements before a Proxmox cluster is viable on it.
Hardware virtualization must be exposed, meaning VT-x or AMD-V visible to the operating system.
There must be a private network between nodes in the same facility. And disk latency must be predictable, which rules out shared storage tiers where a neighbor's workload becomes your problem.
1Gbits met all three, and the private networking piece is the one that turned out to matter most.
The cluster architecture
All three nodes sit in the same data center. That is not a compromise, it is the design. I explain why in the next section.
Networking runs on two interfaces per node.
The public 1 Gbps port carries VM traffic and management.
A private 10 Gbps VLAN carries Corosync, storage replication, and live migration.
Nothing about cluster coordination touches the public internet.
What I would change with hindsight is that I would have separated Corosync onto its own physical interface from day one rather than sharing the private VLAN with storage traffic.
Corosync is small, chatty, and extremely sensitive to delay. Storage replication is large, bursty, and will happily saturate a link. Putting them on the same wire is the mistake that produced my worst outage.
Corosync is the whole game, and it is a latency problem
Corosync is the messaging layer that keeps cluster nodes agreeing on who is alive. Everything else in Proxmox clustering depends on it.
The Proxmox documentation is direct about the requirement: the cluster network should have latency below 5 ms, meaning LAN-like conditions.
Higher latency causes token timeouts, and the default token timeout in current Proxmox versions is 1,000 ms with four retransmits before a node is declared lost.
Once latency climbs into the tens of milliseconds, or once jitter becomes significant, nodes start declaring each other dead while both are perfectly healthy.
My measured inter-node latency on the private VLAN was 0.4 ms median with 0.08 ms jitter. That is the condition Corosync is designed for.
This is why a cluster spanning two data centers does not work. Not "works worse," does not work.
At 20 ms between sites you will see intermittent token loss, nodes fencing themselves, and HA services restarting for no visible reason.
People try it constantly and the symptom is always the same: a cluster that appears fine for days and then fragments during a routine traffic spike.
If you need geographic redundancy, run two independent clusters and replicate between them. Do not stretch one.
Three nodes is the minimum for a real cluster because quorum requires a majority, and three nodes tolerate the loss of one.
If you have an even number of nodes, you need a QDevice, which is a lightweight external voter that breaks ties.
Mine runs on a $6 per month VPS in a different facility. A QDevice only casts a vote, so its latency tolerance is far looser than a full cluster member's.
Corosync survived every network event of the year except one, and that one taught me what fencing actually does.
The split-brain that took me down
Month seven, on a Saturday afternoon. A backup job kicked off and saturated the private 10 Gbps VLAN, which at the time carried both storage replication and Corosync traffic.
The symptom was Corosync token latency spiked past the timeout threshold. pve-02 and pve-03 lost contact with pve-01.
Both surviving nodes retained quorum with two votes out of three. pve-01, now in the minority, did exactly what it was configured to do. It self-fenced through the watchdog and hard-rebooted after the 60-second watchdog timer expired.
Blast radius: 14 VMs on pve-01 went down. HA restarted 9 of them on the surviving nodes within about 3 minutes.
The remaining 5 were pinned to local storage on pve-01 and stayed down until it came back. Total time to full service: 47 minutes, most of it spent on me diagnosing rather than the cluster recovering.
Root cause in one sentence: storage traffic starved the Corosync ring because they shared a link, and Corosync interpreted congestion as node failure.
Three changes fixed it permanently.
First, Corosync moved to its own dedicated interface with a second ring configured as a fallback on the public network, since Corosync supports multiple rings and will fail over between them.
Second, I set bandwidth limits on backup and replication jobs so no single job can saturate the storage link.
Third, I stopped pinning VMs to local storage without a replication target, because a VM that cannot migrate is a VM that will be down as long as its node is.
The storage question is what made those local-only VMs exist in the first place, and I got that decision wrong twice.
Ceph or ZFS replication, and how I chose wrong first
I started with Ceph because high availability with zero data loss is the correct goal. Ceph delivers that. It also wants more nodes than I had.
Three nodes is Ceph's documented minimum with replica size 3 and min_size 2, and it works, but you are running at the floor of the design envelope.
Lose one node and you have no redundancy left until it returns. Rebuilds pull heavily on the storage network.
My measured random write performance on Ceph came in at 41,000 IOPS against 118,000 for the same NVMe hardware under ZFS, which is the price of network round trips on every write acknowledgement.
Ceph also consumed 12 to 18 percent of each node's CPU just existing.
I moved to ZFS replication in month nine. The trade-off is explicit and you should understand it before copying me.
ZFS replication in Proxmox is asynchronous, with a minimum interval of one minute.
That means your recovery point objective is at best one minute of lost writes on an unplanned failover. For my workloads, which are web applications and internal tooling, one minute of loss on a rare node failure is acceptable. For a payment ledger it would not be.
What I run today is a ZFS replication at 1-minute intervals for HA-managed VMs, local ZFS with no replication for genuinely disposable workloads such as CI runners, and Proxmox Backup Server for everything else.
If I add three more nodes, I go back to Ceph, because at six nodes it becomes the tool it was designed to be.
That choice only works because the private network between nodes behaves like a LAN, which on rented hardware is not automatic.
Networking between nodes on rented hardware
You have two ways to get a private network between dedicated servers. The provider gives you a VLAN, or you build an overlay yourself.
1Gbits supports private VLANs between nodes in the same location, which is what Corosync needs and what I use. Measured overhead: none worth reporting, since it is layer 2 switching.
The overlay alternatives cost you. WireGuard between nodes added 0.3 ms of latency and capped throughput at roughly 6.2 Gbps on my hardware, with encryption consuming meaningful CPU on every transfer.
VXLAN without encryption added less delay but no security, which matters if the underlying network is not private. Both are workable for storage replication. Neither is what I would choose for Corosync if a real VLAN is available.
Here is the bridge configuration on each node, in /etc/network/interfaces:
auto lo iface lo inet loopback
auto eno1 iface eno1 inet static address 203.0.113.11/24 gateway 203.0.113.1
auto eno2 iface eno2 inet manual
auto vmbr0 iface vmbr0 inet manual bridge-ports eno1 bridge-stp off bridge-fd 0
auto vmbr1 iface vmbr1 inet static address 10.10.10.11/24 bridge-ports eno2 bridge-stp off bridge-fd 0 mtu 9000
vmbr0 carries VM traffic on the public interface. vmbr1 is the private 10.10.10.0/24 cluster network with jumbo frames enabled, which measurably helps replication throughput. Increment the last octet per node. Corosync binds to the 10.10.10.0/24 ring, and after the split-brain I added a second ring on the public subnet as a fallback.
Backups, because HA is not a backup
High availability protects against a node dying. It does nothing about you deleting the wrong thing, because deletion replicates faster than you can regret it.
Proxmox Backup Server runs on a separate machine outside the cluster, in a different facility. That placement is not optional. A backup server inside the cluster it protects is not a backup server.
Measured results across twelve months: 41 VMs, 8.4 TB of raw data, reduced to 1.9 TB stored after deduplication and compression. That is a 4.4 to 1 ratio, driven mostly by the fact that thirty of those guests run the same Debian base image. Nightly incrementals complete in 11 minutes.
Restore, which is the only number that counts: a 40 GB VM restores in 6 minutes 20 seconds. I test one restore per month, on the first Monday, into an isolated VLAN where it cannot talk to production. I started this practice after discovering in month three that a job I believed was running had been silently failing for eleven days. Backup jobs that report success are not the same as backups that restore.
Twelve-month cost and reliability
The labor line is the one people leave out, so I put it in. Six hours a month of my time at $75 an hour adds $5,400 a year.
The saving is still 77 percent, but it is 77 percent and not 88 percent, and anyone presenting this internally should use the honest figure.
The build order I would use again
Provision three nodes in one facility. Not two facilities, and not two nodes with plans to add a third.
Set up the private VLAN and verify inter-node latency stays under 1 ms before installing anything else. If it does not, stop and fix that first.
Put Corosync on a dedicated interface, and configure a second ring as fallback. This is the day-one item I did in month seven.
Choose storage deliberately. Three nodes means ZFS replication unless you have a hard zero-RPO requirement and are prepared to feed Ceph a proper storage network.
Stand up Proxmox Backup Server outside the cluster before you migrate a single production workload.
Set bandwidth limits on every backup and replication job.
Test a restore. Then schedule the next test.
The three things I wish I had done on day one: dedicated Corosync interface, rate-limited replication jobs, and a monthly restore drill on the calendar. Each one existed only after an incident taught me it should.
FAQ
Can you run a Proxmox cluster across two data centers?
Not reliably. Proxmox documentation calls for cluster network latency under 5 ms, and Corosync interprets delay and jitter as node failure. A stretched cluster will appear stable and then fragment under load. Run two independent clusters with replication between them instead.
What is the maximum latency for Corosync?
Proxmox specifies LAN-like conditions with latency below 5 ms for the cluster network. The default token timeout is 1,000 ms with four retransmits before a node is declared lost. Consistency matters as much as the average, since a jitter spike can trigger fencing on an otherwise healthy node.
Do you need Ceph for Proxmox high availability?
No. ZFS replication supports HA with an asynchronous recovery point objective of at least one minute. Ceph provides zero data loss but wants more nodes and a dedicated 10 Gbps storage network. On three nodes, ZFS replication is usually the better trade.
Can you run Proxmox on a rented dedicated server?
Yes, provided the host exposes hardware virtualization extensions and offers private networking between nodes in the same facility. Both are worth confirming with the provider before ordering, since cluster networking is not something you can add later without a rebuild.
How many nodes does a Proxmox cluster need?
Three for a functioning cluster with automatic failover, because quorum requires a majority and three nodes tolerate one failure. Two nodes work only with a QDevice acting as an external tiebreaker.
