OPNsense and PPPoE with High Availability: Tricky, but Doable

Posted on 2024-04-18

Why

Recently, my fiber-to-the-home (FTTH) connection changed from a slighly easier DHCP based setup to an additional required authentication step using PPPoE to connect to the Internet. The advantage is that, for IPv4, I no longer have to use double-NAT, as my own OPNsense firewall can directly get the (dynamically assigned through PPPoE, but statically allocated at the ISP end) IPv4 WAN address instead of the previous ISP router in between that then handed out transfer net non-routable IPv4 addresses via DHCP.

However, the disadvantage is that now only one of the two firewalls in my OPNsense cluster can get an actual WAN address at the same time (the one statically allocated by my ISP). That is, only one of the two can successfully connect its PPPoE WAN interface at any given time.

How

Without real dynamic routing, e.g. with BGP towards the ISP (which none of them supports for non-business accounts), the best solution is to run the firewalls in an active-passive cluster configuration with only the active/master one being in charge of routing/NAT and the single WAN IPv4 address. OPNsense supports this configuration easily with CARP. As I already had such a CARP setup for all my local network interfaces and previously for the WAN transfer net towards the ISP router, the only missing link was to connect PPPoE dialup to CARP state. With OPNsense version 24.1, the necessary scripts (particularly /usr/local/etc/rc.syshook.d/carp/20-ppp) have already been integrated upstream, but the setup is slightly tricky.

For additional context, my new ISP (Infotech) uses the BBOÖ FTTH network that terminates with a Huawei EG8010Hv6-10 ONT, offering a single 1Gbps Ethernet interface towards the customer. The admin account is locked, but a separate user called “Epuser” (with, as far as I can tell, unique password for each device) can be used to query some state info. The default local (transfer net) IP is 192.168.18.1/24, which I didn’t change.

To connect both OPNsense instances to the ONT, I created a separate VLAN on my Mikrotik CRS326-24G-2S+ core switch to bridge the APU4d4 hardware instance, the VM instance (running on Proxmox), and the ONT.

After some experimentation and trial-and-error, this setup works for me:

Interfaces -> Assignments: Assign the actual Ethernet interface connecting to the ONT (e.g., igb0) to a new interface name different to the wan interface. I called it WAN_Port, and internally on my system the assigned port name is opt12 (but that assigned name is not important). Note the physical Ethernet interface name, though.
- [optional] Interfaces -> Devices -> VLAN: If your upstream fiber provider requires a VLAN tag on that port (e.g., VLAN ID 7 for German Telekom, as I have been told), create a new VLAN device (e.g., called WAN_VLAN) with the required tag on the parent device assigned above. In all steps below, use this VLAN interface instead of the actual Ethernet device. That is, where it says WAN_Port below, replace with WAN_VLAN (respectively the names you used) for both the static and CARP IP addresses and use the extended device name (e.g., igb0_vlan7) for the PPP link interface.
- [alternative] If your provider needs a VLAN tag but you want to avoid the complexity of exposing this in OPNsense, a managed switch could also add the tag transparently between the Ethernet interface exposed to OPNsense and the Ethernet interface of the ONT. Both solutions should work.
Interfaces -> WAN_Port: Assign a Static IPv4 address. I use 192.168.18.2/24 for the master, and 192.168.18.3/24 for the backup firewall.
Interfaces -> Virtual IPs -> Settings: Create a CARP IP address on that newly assigned interface (WAN_Port) with the same VHID group and password for both firewalls. I use 192.168.18.10/24 as virtual IP. Note that the specific interface name and IP address are unimportant and not used as part of the actual PPPoE setup. For the HA integration scripts to work, the Ethernet interface simply needs to have a CARP virtual IP address working and switching over between the firewalls.
System -> High Availability -> Settings: Turn on Disconnect dialup interfaces.
Interfaces -> WAN:
- Switch IPv4 Configuration Type to PPPoE
- Switch IPv6 Configuration Type to DHCPv6 for BBOÖ/Infotech
- Under section PPPoE configuration, set Username and Password appropriately (for OPNsense version <=24)
- Under section DHCPv6 client configuration, select Use IPv4 connectivity and set Prefix delegation size appropriately (60 in my case)
- Save those settings, then click Click here for additional PPP-specific configuration options. under Point-to-Point configuration
Under the Interfaces -> Point-to-Point -> Devices screen this brings up, set Link interface(s) to the actual Ethernet interface connecting to the ONT - in my case this is igb0. For OPNsense version >=25, set Username and Password on this screen.

Notes

Note 1: With OPNsense version 24.1.5_3, at the time of this writing, the script /usr/local/etc/rc.syshook.d/carp/20-ppp only stops the PPPoE connection when the associated CARP interfaces goes into “BACKUP” mode, but not when manually setting Temporarily Disable CARP under Interfaces -> Virtual IPS -> Status. This is bad for debugging/testing. I have therefore patched both of my firewalls to also disable PPPoE when CARP is in “INIT” mode (which is the case when temporarily disabling CARP):

--- /tmp/20-ppp.orig    2024-04-18 12:53:43.767650142 +0200
+++ /tmp/20-ppp.fixed   2024-04-18 12:55:51.638549481 +0200
@@ -36,7 +36,7 @@
 
 $a_hasync = &config_read_array('hasync');
 if (!empty($a_hasync['disconnectppps'])) {
-    if ($type != 'MASTER' && $type != 'BACKUP') {
+    if ($type != 'MASTER' && $type != 'BACKUP' && $type != 'INIT') {
        log_msg("Carp '$type' event unknown from source '{$subsystem}'");
        exit(1);
     } elseif (!strstr($subsystem, '@')) {
@@ -51,7 +51,7 @@
             foreach($config['interfaces'] as $ifkey => $interface) {
                 if ($ppp['if'] == $interface['if']) {
                     log_msg("{$iface} is connected to ppp interface {$ifkey} set new status {$type}");
-                    if ($type == 'BACKUP') {
+                    if ($type == 'BACKUP' || $type == 'INIT') {
                         interface_suspend($ifkey);
                     } else {
                         interface_ppps_configure($ifkey);

Update: This patch is no longer necessary for OPNsense version >= 25.7.8, as it has been merged into upstream.

Note 2: The overall solution was initially unstable because PPPoE connection setup after an active/passive switch-over failed with timeouts (no replies from the OLT headend or PPPoE server). This seemed to be because of a MAC address filter implemented on the OLT side, allowing only 2 MAC addresses seen behind the ONT (a configuration applied by Energie-AG as part of the BBOÖ fiber network), and not because of the CARP-based switching on OPNsense. One workaround that seems to work is to block other MAC source addresses than the real ones of the (physical and virtual) OPNsense firewall from going out of the switch interface towards the ONT.

~~On Mikrotik RouterOS, one option could be adding two rules under Switch -> Rule:~~

~~Match~~:
- ~~Switch: the (single) hardware switch in my CRS326-24G-2S+, i.e. switch1~~
- ~~Ports: the (single) Ethernet port to which the ONT is directly connected, e.g. ether10~~
- ~~Src MAC Address: different for the 2 rules:~~ * B8:69:F4:00:00:00 with mask FF:FF:FF:00:00:00 for blocking Mikrotik RouterOS announcements * 00:00:5E:00:01:01 with mask FF:FF:FF:00:00:00 for blocking CARP packets
~~Action:~~
- ~~Set New Dst. Ports checked, but no ports listed under New Dst. Ports, which basically blackholes packets matching the above filter on the switch level~~

Unfortunately, I found out much later that these rules don’t actually work, because on CRS3xx series switches, the switch rules policer only applies to ingress but not to egress traffic on a port. So these rules won’t have any effect, as I found out through mirroring the ONT port to another and still seeing the CARP traffic and assosicated MAC addresses. The “correct” version is therefore to use bridge filters under Bridge -> Filters:

Create a new rule to “Block CARP MAC addresses from appearing to FTTH ONT so as not to trigger MAC filters” (my comment)
- General:
  - Chain: set to forward
  - Out interface: the (single) Ethernet port to which the ONT is directly connected, e.g. ether10
  - Src MAC Address: 00:00:5E:00:01:01 with Src MAC Mask FF:FF:FF:FF:00:00
- Action:
  - Action: drop
Create another rule to “Block Mikrotik MAC addresses from appearing to FTTH ONT so as not to trigger MAC filters” (my comment)
- General:
  - Chain: set to output
  - Out interface: the (single) Ethernet port to which the ONT is directly connected, e.g. ether10
  - Src MAC Address: B8:69:F4:00:00:00 with Src MAC Mask FF:FF:FF:00:00:00
- Action:
  - Action: drop

Important note: For the bridge filters to work on a CRS3xx series switch, so-called bridge hardware (HW) offloading needs to be disabled. Otherwise, those bridge filters will simply be skipped (the switch HW doesn’t support filtering, only packets routed through the switch CPU are filtered). If you look at the flow diagrams (which are really well done for RouterOS), it becomes clear that the Hardware Offload flag needs to be disabled for the incoming interface for the filters to be applied. Therefore, disable this flag for the etherX interface connected to the two firewalls (it doesn’t hurt to disable it for the port connected to the ONT either, considering the typical packet rates compared to the LAN, but according to the flow diagram shouldn’t matter for this purpose (but for some reason that I haven’t figured out yet might still disable switch HW forwarding if HW offload is disabled for the outgoing port - take with a grain of salt)).

Note 2 addendum: There are some additional items to disable on a Mikrotik RouterOS device for the Ethernet port pointing towards the FTTH ONT to really stop it from sending other packets with different source MAC addresses:

Disable STP by setting the respective switch port to edge=yes.
Disable neighbor discovery through various protocols by:
- Put the OPNsense cluster WAN ports as well as the ONT port into the WAN list under /interface/list (Interfaces -> Interface List in the UI)
- Disable the various discovery mechanisms, and enable normal IP discovery and mac-winbox only for the (other) interfaces that remain in the LAN interface list:

# This is for discovery
/ip neighbor discovery-settings/set discover-interface-list=LAN \
	mode=tx-and-rx protocol=cdp,lldp,mndp
# This is for mac-ping
/tool mac-server ping set enabled=no
# This is for mac-telnet
/tool mac-server set allowed-interface-list=none
# This is for mac-winbox
/tool mac-server mac-winbox set allowed-interface-list=LAN

Conclusions

Active/passive OPNsense firewall clusters are possible with PPPoE upstream WAN interfaces, but getting them to run is tricky and the official OPNsense documentation doesn’t mention this option at all. Finding the right combination of options took reading a lot of forum posts and eventually adding log messages to the CARP ppp script to find out how to set the Link interface(s) parameter to make it work.