FreeBSD* Driver for the 10 Gigabit Server Adapters

NOTES: This release includes the 10 gigabit ixgbe FreeBSD Base Driver for Intel® Network Connection.  All 82599 and 82598EB-based network connections require the ixgbe driver. All other network connections require the ixgb driver. The ixgb driver can be downloaded at:  http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/dev/ixgb/. First identify your adapter.  Then follow the appropriate steps for building, installing, and configuring the appropriate driver.

Identifying Your Adapter

To verify that your Intel adapter is supported, find the board ID number on the adapter. Look for a label that has a barcode and a number in the format of 123456-001 (six digits hyphen three digits).  For more information on how to identify your adapter, go to the Adapter & Driver ID Guide at:

http://support.intel.com/support/go/network/adapter/home.htm

For the latest Intel network drivers for FreeBSD, see:

http://support.intel.com/support/go/network/adapter/home.htm

Using Your ixgbe Driver

Building and Installation

Additional Configurations

Known Limitations

Support

To verify that your Intel adapter is supported, find the board ID number on the adapter. Look for a label that has a barcode and a number in the format of 123456-001 (six digits hyphen three digits).  For more information on how to identify your adapter, go to the Adapter & Driver ID Guide at:

http://support.intel.com/support/go/network/adapter/home.htm

For the latest Intel network drivers for FreeBSD, see:

http://downloadcenter.intel.com/scripts-df-external/Support_Intel.aspx

SFP+ Devices with Pluggable Optics

82599-BASED ADAPTERS

NOTE:

  • If your 82599-based Intel® Network Adapter came with Intel optics, or is an Intel® Ethernet Server Adapter X520-2, then it only supports Intel optics and/or the direct attach cables listed below.

Supplier            Type                                                     Part Numbers
SR Modules                 
Intel              DUAL RATE 1G/10G SFP+ SR (bailed) AFBR-703SDZ-IN2
Intel              DUAL RATE 1G/10G SFP+ SR (bailed) FTLX1471D3BCL
LR Modules                
Intel               DUAL RATE 1G/10G SFP+ LR (bailed) FTLX1471D3BCV-IT
Intel               DUAL RATE 1G/10G SFP+ LR (bailed) AFCT-701SDZ-IN2

The following is a list of 3rd party SFP+ modules and direct attach cables that have received some testing. Not all modules are applicable to all devices.

Supplier            Type                                                     Part Numbers
Finisar SFP+ SR bailed, 10g single rate FTLX8571D3BCL
Avago SFP+ SR bailed, 10g single rate AFBR-700SDZ
Finisar SFP+ LR bailed, 10g single rate FTLX1471D3BCL
     
Finisar DUAL RATE 1G/10G SFP+ SR (No Bail) FTLX8571D3QCV-IT
Avago DUAL RATE 1G/10G SFP+ SR (No Bail) AFBR-703SDZ-IN1
Finisar DUAL RATE 1G/10G SFP+ LR (No Bail) FTLX1471D3QCV-IT
Avago DUAL RATE 1G/10G SFP+ LR (No Bail) AFCT-701SDZ-IN1
     
Molex 1m - Twin-ax cable  74752-1101
Molex 3m - Twin-ax cable 74752-2301
Molex 5m - Twin-ax cable 74752-3501
Molex 10m - Twin-ax cable 74752-9004
Tyco 1m - Twin-ax cable 2032237-2
Tyco 3m - Twin-ax cable 2032237-4
Tyco 5m - Twin-ax cable 2032237-6
Tyco 10m - Twin-ax cable 1-2032237-1

82598-BASED ADAPTERS

NOTES:

  • Intel® Network Adapters that support removable optical modules only support their original module type (i.e., the Intel® 10 Gigabit SR Dual Port Express Module only supports SR optical modules). If you plug in a different type of module, the driver will not load.

  • Hot Swapping/hot plugging optical modules is not supported.

  • Only single speed, 10 gigabit modules are supported.

  • LAN on Motherboard (LOMs) may support DA, SR, or LR modules. Other module types are not supported. Please see your system documentation for details.

The following is a list of SFP+ modules and direct attach cables that have received some testing. Not all modules are applicable to all devices.

Supplier            Type                                                     Part Numbers
Finisar              SFP+ SR bailed, 10g single rate   FTLX8571D3BCL
Avago              SFP+ SR bailed, 10g single rate   AFBR-700SDZ
Finisar              SFP+ LR bailed, 10g single rate FTLX1471D3BCL
Molex               1m - Twin-ax cable 74752-1101
Molex               3m - Twin-ax cable 74752-2301
Molex               5m - Twin-ax cable 74752-3501
Molex               10m - Twin-ax cable  74752-9004
Tyco                 1m - Twin-ax cable   2032237-2
Tyco                 3m - Twin-ax cable    2032237-4
Tyco                 5m - Twin-ax cable 2032237-6
Tyco                 10m - Twin-ax cable   1-2032237-1

THIRD PARTY OPTIC MODULES AND CABLES REFERRED TO ABOVE ARE LISTED ONLY FOR THE PURPOSE OF HIGHLIGHTING THIRD PARTY SPECIFICATIONS AND POTENTIAL COMPATIBILITY, AND ARE NOT RECOMMENDATIONS OR ENDORSEMENT OR SPONSORSHIP OF ANY THIRD PARTY’S PRODUCT BY INTEL. INTEL IS NOT ENDORSING OR PROMOTING PRODUCTS MADE BY ANY THIRD PARTY AND THE THIRD PARTY REFERENCE IS PROVIDED ONLY TO SHARE INFORMATION REGARDING CERTAIN OPTIC MODULES AND CABLES WITH THE ABOVE SPECIFICATIONS. THERE MAY BE OTHER MANUFACTURERS OR SUPPLIERS, PRODUCING OR SUPPLYING OPTIC MODULES AND CABLES WITH SIMILAR OR MATCHING DESCRIPTIONS. CUSTOMERS MUST USE THEIR OWN DISCRETION AND DILIGENCE TO PURCHASE OPTIC MODULES AND CABLES FROM ANY THIRD PARTY OF THEIR CHOICE. CUSTOMERS ARE SOLELY RESPONSIBLE FOR ASSESSING THE SUITABILITY OF THE PRODUCT AND/OR DEVICES AND FOR THE SELECTION OF THE VENDOR FOR PURCHASING ANY PRODUCT. THE OPTIC MODULES AND CABLES REFERRED TO ABOVE ARE NOT WARRANTED OR SUPPORTED BY INTEL. INTEL ASSUMES NO LIABILITY WHATSOEVER, AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF SUCH THIRD PARTY PRODUCTS OR SELECTION OF VENDOR BY CUSTOMERS.

Building and Installation

NOTE: You must have kernel sources installed in order to compile the driver module.

In the instructions below, x.x.x is the driver version as indicated in the name of the driver tar file.

ixgbe instructions

  1. Move the base driver tar file to the directory of your choice. For example, use /home/username/ixgbe or /usr/local/src/ixgbe.
  2. Untar/unzip the archive:

        tar xfz ixgbe-x.x.x.tar.gz

  3. To install man page:

        cd ixgbe-x.x.x
        gzip -c ixgbe.4 > /usr/share/man/man4/ixgbee.4.gz

  4. To load the driver onto a running system:

        cd ixgbe-x.x.x/src
        make load

  5. To assign an IP address to the interface, enter the following:

        ifconfig ix<interface_num> <IP_address>

  6. Verify that the interface works. Enter the following, where <IP_address> is the IP address for another machine on the same subnet as the interface that is being tested:

        ping <IP_address>

  7. If you want the driver to load automatically when the system is booted:

        cd ixgbe-x.x.x/src
        make
        make install

    Edit /boot/loader.conf, and add the following line:
        ixgbe_load="YES"

    OR

    compile the driver into the kernel (see item 8).

    Edit /etc/rc.conf, and create the appropriate ifconfig_ixgbe<interface_num> entry:

        ifconfig_ix<interface_num>

    Example usage:

        ifconfig_ix0="inet 192.168.10.1 netmask 255.255.255.0"

    NOTE: For assistance, see the ifconfig man page.

  8. If you want to compile the driver into the kernel, enter:

    FreeBSD 7 or later:

        cd ixgbe-x.x.x/src

        cp *.[ch] /usr/src/sys/dev/ixgbe

        cp Makefile.kernel /usr/src/sys/modules/ixgbe/Makefile

    Edit the kernel configuration file (i.e., GENERIC or MYKERNEL) in  /usr/src/sys/i386/conf (replace "i386" with the appropriate system architecture if necessary), and ensure the following line is present:

        device ixgbe

    Compile and install the kernel. The system must be reboot for the kernel updates to take affect. For additional information on compiling the kernel, consult the FreeBSD operating system documentation.

Configuration and Tuning

The driver supports Transmit/Receive Checksum Offload and Jumbo Frames on all 10GbE adapters.

Jumbo Frames

To enable Jumbo Frames, use the ifconfig utility to increase the MTU beyond 1500 bytes.

NOTES: The Jumbo Frames setting on the switch must be set to at least 22 bytes larger than that of the MTU. 

There are known performance issues with this driver when running UDP traffic with Jumbo Frames.

The Jumbo Frames MTU range for Intel Adapters is 1500 to 16114. The default MTU range is 1500. To modify the setting, enter the following:

ifconfig ix<interface_num> <hostname or IP address> mtu 9000

To confirm an interface's MTU value, use the ifconfig command. To confirm the MTU used between two specific devices, use:

route get <destination_IP_address>

VLANs

To create a new VLAN interface:

ifconfig <vlan_name> create

To associate the VLAN interface with a physical interface and assign an IP address, netmask, and VLAN ID:

ifconfig <vlan_name> <ip_address> netmask <subnet_mask> vlan <vlan_id> vlandev <physical_interface>

Example:

ifconfig vlan10 10.0.0.1 netmask 255.255.255.0 vlan 10 vlandev ixgb0

In this example, all packets will be marked on egress with  802.1Q VLAN tags, specifying a VLAN ID of 10.

To remove a VLAN interface:

ifconfig <vlan_name> destroy

Checksum Offload

Checksum offloading supports both TCP and UDP packets and is supported for both transmit and receive.

Checksum offloading can be enabled or disabled using ifconfig. Both transmit and receive offloading will be either enabled or disabled together. You cannot enable/disable one without the other.

To enable checksum offloading:

ifconfig <interface_num> rxcsum

To disable checksum offloading:

ifconfig <interface_num> -rxcsum

To confirm the current setting:

ifconfig <interface_num>

TSO

TSO is enabled by default.

To disable:

         ifconfig <interface_num> -tso

To re-enable:

         ifconfig <interface_num> tso

LRO

Large Receive Offload is available in the driver; it is on by default.  It can be disabled by using:

ifconfig <interface_num> -lro

To enable:

ifconfig <interface_num> lro

Important System Configuration Changes for ixgbe Driver

When there is a choice, run on a 64bit OS rather than 32, it makes a significant difference in improvement.

The default scheduler SCHED_4BSD is not smart about SMP locality issues. Significant improvement can be achieved by switching to the ULE scheduler.

This is done by changing the entry in the config file from SCHED_4BSD to SCHED_ULE. Note that this is only advisable on FreeBSD 7, on 6.X there have
been stability problems with ULE.

The interface can generate high number of interrupts. To avoid running into the limit set by the kernel, adjust hw.intr_storm_threshold setting using sysctl:

sysctl hw.intr_storm_threshold=9000 (the default is 1000)

For this change to take effect on boot, edit /etc/sysctl.conf and add the line:

hw.intr_storm_threshold=9000

If you still see Interrupt Storm detected messages, increase the limit to a higher number.

Best throughput results are seen with a large MTU; use 9000 if possible.

The default number of descriptors is 256, increasing this to 1024 or even 2048 may improve performance.

Known Limitations

Under small packets UDP stress test with the 10GbE driver, the FreeBSD system will drop UDP packets, due to socket buffers being full. You may want to change the driver's Flow Control variables to the minimum value for controlling packet reception.

Attempting to configure larger MTUs with a large numbers of processors may generate the error message "ix0:could not setup receive structures"

When using the ixgbe driver with RSS autoconfigured based on the number of cores (the default setting) and that number is larger than 4, increase the memory resources allocated for the mbuf pool as follows:

Add to the sysctl.conf file for the system:

kern.ipc.nmbclusters=262144

kern.ipc.nmbjumbop=262144

Lower than expected performance on dual port 10GbE devices

Some PCI-E x8 slots are actually configured as x4 slots. These slots have insufficient bandwidth for full 10Gbe line rate with dual port 10GbE devices. The driver can detect this situation and will write the following message in the system log: “PCI-Express bandwidth available for this card is not sufficient for optimal performance. For optimal performance a x8 PCI-Express slot is required.”

If this error occurs, moving your adapter to a true x8 slot will resolve the issue.

Support

For general information and support, go to the support website

If an issue is identified, support is through email only at: freebsd@intel.com

Last modified on 9/09/09 1:04p Revision 70