For the last couple of years I’ve used PFSense as my home router. It’s been great - it’s easy to manage with the web interface, and really easy to add features like an OpenVPN server, bandwidth monitoring, etc.

But, I like to manage all of my servers and zones at home using Chef, and my router has always been left out as it required being configured manually through the web interface. So now, I’ve replaced PFSense with 2 SmartOS zones: 1 for NAT and the other for DHCP - both managed by Chef, and both monitored with Nagios. This also had the added effect of reducing two physical servers at home down to one, for a cheaper power bill.

There are 3 steps required to configure a SmartOS server as a home router replacement.

  1. Add the External Interface
  2. Create the NAT Zone
  3. Create the DHCP Zone

1. Add the External Interface

The first thing is to ensure that you have 2 (or more) NICs on the SmartOS server - one for the external network (The Internet) and one for the internal network that will be behind NAT. To list the interfaces run

root - datadyne sunos ~ # dladm show-phys -m
LINK         SLOT     ADDRESS            INUSE CLIENT
rge0         primary  f4:6d:4:X:XX:XX    yes  rge0
e1000g0      primary  0:4:23:XX:XX:XX    yes  e1000g0

Cross-referencing this information with output from ifconfig(1M) reveals that the e1000g0 interface is not currently in use, and can be used as the external interface.

root - datadyne sunos ~ # ifconfig e1000g0
ifconfig: status: SIOCGLIFFLAGS: e1000g0: no such interface

Add the following line to /usbkey/config to put this interface on a tag called external - modify the mac address to match the output from the dladm command above

external_nic=0:4:23:XX:XX:XX

At this point I rebooted the hypervisor for the change to take effect, but I believe running the following will do the same thing without rebooting the machine.

sysinfo -u

You can verify this worked by running

root - datadyne sunos ~ # nictagadm list
NAME           MACADDRESS         LINK           TYPE
external       00:04:23:XX:XX:XX  e1000g0        normal
admin          f4:6d:04:XX:XX:XX  rge0           normal

You should see an external NIC tag with the same interface name and mac address if everything went right.

Now that the NIC has been added and a tag is ready, a new zone must be created with VNICs on both the external and the internal networks (in my case, the external and admin networks)

2. Create the NAT Zone

To create the NAT zone you can use this JSON

{
  "brand": "joyent",
  "image_uuid": "c02a2044-c1bd-11e4-bd8c-dfc1db8b0182",
  "autoboot": true,
  "alias": "nat",
  "hostname": "nat.rapture.com",
  "dns_domain": "rapture.com",
  "resolvers": [
    "208.67.222.222",
    "208.67.222.220"
  ],
  "max_physical_memory": 512,
  "nics": [
    {
      "nic_tag": "admin",
      "ip": "10.0.1.1",
      "gateway": "10.0.1.1",
      "netmask": "255.255.255.0",
      "allow_ip_spoofing": true,
      "primary": true
    },
    {
      "nic_tag": "external",
      "ip": "dhcp",
      "allow_ip_spoofing": true
    }
  ]
}

And run

vmadm create -f nat.json

In the nics section of the config there are 2 VNICs defined, one on the internal (admin) network with a static IP set, and the other on the external network with DHCP set - as my ISP provides a dynamic IP. Both VNICs require allow_ip_spoofing to be enabled for NAT to work properly.

vmadm(1M) creates the VNICs in the order they are given, so they will be seen on the zone as

  • net0: the internal interface
  • net1: the external interface (The Internet)

Login to the zone

zlogin "$(vmadm list -o uuid -H alias=nat)"

Verify that the interfaces are setup correctly

dave - nat sunos ~ $ ifconfig net0
net0: flags=1100843<UP,BROADCAST,RUNNING,MULTICAST,ROUTER,IPv4> mtu 1500 index 2
        inet 10.0.1.1 netmask ffffff00 broadcast 10.0.1.255
dave - nat sunos ~ $ ifconfig net1
net1: flags=1104843<UP,BROADCAST,RUNNING,MULTICAST,DHCP,ROUTER,IPv4> mtu 1500 index 3
        inet 72.231.XXX.XXX netmask fffffe00 broadcast 72.231.XXX.XXX

net0 should have the static IP set from the JSON config, and net1 should be configured by DHCP from your ISP.

Create a config for ipnat(1M)

cat <<-EOF > /etc/ipf/ipnat.conf
# NAT
map net1 10.0.1.0/24 -> 0/32 proxy port ftp ftp/tcp
map net1 10.0.1.0/24 -> 0/32 portmap tcp/udp auto
map net1 10.0.1.0/24 -> 0/32

# Port Forwards
# rdr net1 0/0 port 80 -> 10.0.1.5 port 8080 tcp
EOF

The first 3 lines of the config will setup ipnat(1M) to NAT from the internal interface to the external one. The last line (commented out) will create a port forward from the external interface on port 80 over TCP, to the internal host 10.0.1.5 on port 8080 as an example.

To enable the service, run

routeadm -u -e ipv4-forwarding
svcadm enable ipfilter

At this point, any machine on your internal network with this zones IP set as its default gateway will be able to reach the internet.

3. Create the DHCP Zone

To create the DHCP zone you can use this JSON

{
  "brand": "joyent",
  "image_uuid": "c02a2044-c1bd-11e4-bd8c-dfc1db8b0182",
  "autoboot": true,
  "alias": "dhcp",
  "hostname": "dhcp.rapture.com",
  "dns_domain": "rapture.com",
  "resolvers": [
    "208.67.222.222",
    "208.67.222.220"
  ],
  "max_physical_memory": 512,
  "nics": [
    {
      "nic_tag": "admin",
      "ip": "10.0.1.4",
      "allow_dhcp_spoofing": true,
      "netmask": "255.255.255.0",
      "gateway": "10.0.1.1",
      "primary": true
    }
  ]
}

And run

vmadm create -f dhcp.json

In the nics section of the config there is 1 VNIC defined on the internal (admin) network with a static IP set. The setting allow_dhcp_spoofing must be enabled in order for the DHCP server to work as expected.

Login to the zone

zlogin "$(vmadm list -o uuid -H alias=dhcp)"

Install the isc-dhcpd package

pkgin up
pkgin in isc-dhcpd

Create a config for dhcpd, modifying it as necessary to fit your environment

cat <<-EOF > /opt/local/etc/dhcp/dhcpd.conf
default-lease-time 600;
max-lease-time 7200;

option subnet-mask 255.255.255.0;
option broadcast-address 10.0.1.255;
option routers 10.0.1.1;
option domain-name-servers 10.0.1.2, 10.0.1.3;
option domain-name "rapture.com";

subnet 10.0.1.0 netmask 255.255.255.0 {
    range 10.0.1.200 10.0.1.250;
}
EOF

Finally, start the service

svcadm enable isc-dhcpd

Conclusion

With this all setup, any host on your network that requests DHCP will be answered by the dhcp zone, which will tell the host to use the nat zone as the default gateway.

In other words, with just 2 low-powered zones, any existing SmartOS server can replace a dedicated home/NAT router on a network.

Notes / Tips

A lot of this blog post was based around this incredibly helpful SmartOS Wiki page

  • https://wiki.smartos.org/display/DOC/NAT+using+Etherstubs

You can view statistics for NAT by running

ipnat -s
ipnat -l

If you modify /etc/ipf/ipnat.conf and want the changes to take effect without restarting the service you can run

ipnat -FC -f /etc/ipf/ipnat.conf

On the dhcp zone, you can look at the current leases by running

cat /var/db/isc-dhcp/dhcpd.leases

Alternatively, check out the DHCPD Dashboard I wrote to create a web interface that shows all of the current DHCP leases on the dhcp zone

  • https://github.com/bahamas10/node-dhcpd-dashboard

Monitoring

I’ve created a Nagios style script to check the current number of NAT mappings on the nat zone to alert if the percentage is over a certain threshold

$ ./check_ipnat_mappings
ok: 347 / 30000 in use - 1% total|inuse=347;max=30000;perc=1

#!/usr/bin/env bash
#
# nagios check for ipnat mappings inuse
#
# Author: Dave Eddy <[email protected]>
# Date: May 19, 2015
# License: MIT

warning=80
critical=90
while getopts 'w:c:' option; do
        case "$option" in
                w) warning=$OPTARG;;
                c) critical=$OPTARG;;
        esac
done

ipnats=$(ipnat -s)
if (($? != 0)); then
        echo "unknown: failed to call ipnat(1M)"
        exit 3
fi

inuse=$(echo "$ipnats" | awk '/^inuse/ { print $2 }')
max=$(ipf -T ipf_nattable_max | awk '{ print $NF }')

perc=$((inuse * 100 / max))

if [[ -z $perc ]]; then
        echo 'unknown: error retrieving data'
        exit 3
elif ((perc >= critical)); then
        msg='critical'
        ret=2
elif ((perc >= warning)); then
        msg='warning'
        ret=1
else
        msg='ok'
        ret=0
fi

echo "$msg: $inuse / $max in use - $perc% total|inuse=$inuse max=$max perc=$perc%"
exit "$ret"