FreeNAS: Jumbo Frames and LAGG Interfaces
FreeNAS is awesome! It's got tons of features and it's polished. Some of the features that make is awesome are being able to set MTU size (jumbo frames) and use LACP.
To setup an LACP group, simply follow the Link Aggregation and Failover instructions and choose LACP as the Aggregation protocol
Jumbo Frames is simply setting an MTU higher than 1500. The MTU can be set easily via the Interfaces: menu.
With all of this ease, I ran into surprising difficulty when trying to marry the two concepts. No matter what I did, I could not get the MTU on my lagg interface to go above 1500. While digging through man pages and google searches, it looked as if setting the MTU was a simple ifconfig command:
# ifconfig lagg0 mtu 9000 ifconfig: ioctl (set mtu): Invalid argument
I was very disappointed when it didn't work. Many more google searches later, I found a little gem that shed some light. One can't actually use the lagg interface to set the MTU. The MTU needs to be set on the member interfaces before the lagg is created. I tested the theory from the command line and it worked. The only problem remaining was to get it to survive a reboot. FreeNAS doesn't have a GUI for setting NIC parameters on NICs that aren't enabled.
Through much help from bubulein and zoon01 in #freenas on irc.freenode.net, I was able to use rc.conf to set the MTUs of the NICs before the lagg is created.
The steps to set the MTU in rc.conf are:
Go to System->Advanced and then the rc.conf tab.
Click on the + to add another entry
Repeat for additional NICs in the lagg
In my case, I am using Intel PRO/1000 NICs, so the interface names are emX where X is the interface number. If using Broadcom NICs, you'll use bceX instead of emX. Simply look at the interfaces named in your LAGG configuration.
After setting the MTU for the member NICs, I set the MTU on lagg interface via the GUI. I haven't tested this to see if it's absolutely necessary, but it doesn't hurt. It also helps show that the lagg is using jumbo frames.
After a reboot, the lagg came up with the proper MTU of 9000. I am now a happy camper and excited about all the NASy goodness I now have.
One last note: using jumbo frames must be supported by your NIC. Additionally, different NICs have different max MTU sizes. The Intel PRO/1000 that I have has a max of 16128 where as the Broadcom NetXtreme II BCM5708 has a max of 9022.