Setting up High Availability on pfSense is not that difficult. It consists of configuring the state and settings synchronization (pfsync), creating virtual IPs (CARP) and changing the settings for DHCP, DNS and NTP so that you clients use the virtual IP. But things get tricky if you have only 1 IPv4 WAN address and it is assigned via DHCP by your ISP. If CARP is not an option for the WAN interface, how can we make sure we get the same public IP address on both firewalls?
Inspired by a post by dsmith10 I managed to get it working on pfSense 2.6 with these features:
- fast fail-over in 2 seconds
- fast fail-back in 8 seconds
A “nice to have” feature that I have not yet succeeded in configuring is still on my wish list:
- When the standby firewall transitions to the BACKUP state, automatically add a temporary default route that points to the LAN interface of the MASTER firewall. This way the standby firewall can access the internet.
- When the standby firewall transitions to the MASTER state, remove the temporary default route that points to the LAN interface of the MASTER firewall.
These are the steps I took to make HA work with a WAN interface that only gets 1 IPv4 address via DHCP from my ISP:
- Spoof the MAC-address of the WAN interface of the primary firewall on the WAN interface on the standby firewall.
- On the primary firewall, go to Status > Interfaces and copy the MAC Address of the WAN Interface. Also take note of the interface device name. On my primary firewall this was igb0.
- On the standby firewall, go to Interfaces > WAN and go to the field MAC Address.
- Paste the MAC address of the of the primary firewall in the field.
- hit Save on the bottom of the page.
- Automatically disable the WAN interface on the primary firewall when it is in the BACKUP state: SSH to the primary firewall, login, choose 8 ) Shell and perform these commands:
cd /usr/local/etc/devd
vi carp.conf
- press
i
to go to insert mode in vi and paste this text, make sure you use the interface device name that you noted earlier (igb0 on my firewall).
notify 200 {
match "system" "CARP";
match "type" "MASTER";
action "ifconfig igb0 up; /usr/local/sbin/pfSctl -c 'interface carpmaster $subsystem'";
};
notify 200 {
match "system" "CARP";
match "type" "BACKUP";
action "ifconfig igb0 down; /usr/local/sbin/pfSctl -c 'interface carpbackup $subsystem'";
}; - press
Esc
followed by:wq<ENTER>
to save and exit - close the SSH session
- Automatically enable the WAN interface on the standby firewall when it is in the MASTER state: SSH to the standby firewall, login, choose 8 ) Shell and perform these commands:
cd /usr/local/etc/devd
vi carp.conf
- press
i
to go to insert mode in vi and paste this text, make sure you use the interface device name that you noted earlier (igb0 on my firewall).
notify 200 {
match "system" "CARP";
match "type" "MASTER";
action "ifconfig vnet1 up; /usr/local/sbin/pfSctl -c 'interface carpmaster $subsystem'";
};
notify 200 {
match "system" "CARP";
match "type" "BACKUP";
action "ifconfig vnet1 down; /usr/local/sbin/pfSctl -c 'interface carpbackup $subsystem'";
}; - press
Esc
followed by:wq<ENTER>
to save and exit - close the SSH session
Ideally both firewalls should have the same hardware but I built this setup with a physical firewall as primary and a virtual firewall as secondary.
I think I found a solution to the unfinished part. Put a VIP on the sync network (requires a minimum of a /29 for the sync network,) create a gateway on the Sync VIP IP. Create a gateway group with the Sync VIP Gateway on the lowest tier relevant to your configuration. Permit traffic sourced from the Sync network to reach the internet, create an outbound NAT rule sourced from the Sync network to the internet (requires “hybrid” or “manual” NAT.)
better is to create a gateway group, one with the WAN on tier 1 and LAN (VIP) on tier 2, then use the gateway group as default. when wan is unavailable on the secondary, it will use the VIP as secondary gateway.
when failover happens the same will apply for the primary, making it seamless.
Just wanted to clarify: By permit traffic I mean via firewall rules on the Sync interface and the outbound NAT rule is on the WAN interface for traffic sourced on the Sync subnet, destine for the internet.
one caveat:
for the inotify command runs for each VIP you have, I have my network microsegmented into about 25 vlan’s, each with their own VIP. so the command runs 25 times xD, which could lead to some wierd behaviour