In order to separate the VM IP space from the upstream WAN connection a firewall was setup to provide NAT, DHCP and DNS to the VMs and containers. PfSense was used as the packaged firewall software is widely used by the homelab community, as such much community information exists for many levels of configuration.
For the pfSense machine a VM with 1 CPU core and 1GB of RAM was created, as Proxmox doesn't have an option for FreeBSD the OS type is set to other. Once the initial creation wizard was completed a second interface was created in the hardware tab and assigned to the second network bridge. From now on VMs will be attached to the second network bridge and not the first.
Once the hardware is configured the VM can be booted into the ISO and there will be an option to do an automatic install where the full hard drive is provisioned to pfsense and the bootloader is setup. Once the installer reboots a few options can be set-up from the command line. Once pfSense completes booting you will be required to set which interface is the LAN and WAN connections, this is easiest done by comparing the MAC addresses to those in Proxmox. Once pfSense reaches the option screen select option 2 to configure the interface IPs.
Select option 2 to configure the LAN IP, enter the IP for the LAN interface this cannot be in the same subnet range as the WAN IP address that will likely be assigned by DHCP. Next select the subnet size, 24 is a standard size or 23 if you intend to create many VMs and containers.
Press enter for the upstream gateway and IPv6 address questions. Enable DHCP for the interface and select start and end values, these must be within the subnet set earlier and you should leave some spare for staticly assigned addresses. I left 1-9 for network infrastructure such as switches and APs and 201-254 for staticly assigned addresses.
Once the DHCP range is assigned select no to downgrading from HTTPS to HTTP then wait for the configuration to reload. An address will then be given to reach the web interface, this can be reached by connecting an external machine to the network device associated with the LAN interface or moving a graphical VM's network interface to the LAN bridge.
Once at the web UI login with the default user 'admin' and 'pfsense', this will give you the setup wizard. The majority of the settings can be left at default initially set the hostname and domain name fields. Then set the DNS servers to use and if you wish to use the ISP provided DNS servers select the Override DNS option. Next set the NTP server, the default offered is fine.
The majority of the WAN options page can be left because these were set in the shell. If the WAN interface is part of an overall local network it may be preferable to be able to access the outer LAN from the inner LAN. This can be done in the WAN settings by Unchecking the RFC1918 Networks and Bogon Networks options.
The LAN options can be left as set and then a new admin password can be set.
As the firewall is running on virtualised hardware it cannot make use of typical hardware acceleration as such hardware checksum offloading needs to be disabled to use software checksums. This is done in the System -> Advanced -> Networking Page, make sure all the offloading disable options at the bottom of the page are checked.
To configure DNS go to the Services -> DNS Resolver page. To enable all DHCP clients to have their hostname registered as DNS records check the Register DHCP leases and Register Static DHCP mappings options. To manually add DNS records select add in the Host Overrides section and from there add forward and reverse DNS records as well as aliases.
To configure port forwarding go to the Firewall -> NAT page and select add. Select the protocol and leave the destination at WAN address. Select the protocol or port range for the external interface, then select the internal address and starting port number or protocol. Port forwarding rules are processed in order so with overlapping rules the first rule will be used.
In order to make proxmox only available from within the LAN network go to the System -> Network tab on the node and remove the IP address, subnet mask and gateway from the WAN bridge and then set the gateway on the LAN bridge to the pfSense IP address. On next reboot the node will only be accessible from the LAN subnet.
To use the pfsense terminal with xterm.js the serial terminal must be enabled. This is done from the web UI under System -> Advanced -> Admin Access -> Serial Communicactions. Check the enable serial console box and set the speed to 115200 and save. Next shut the VM down and go to the Proxmox node's shell. Then run the following command with 100 replaced with your VMID:
qm set 100 -serial0 socket
Start the VM and select xterm.js from the dropdown next to the console button.