Sun Microsystems Dual XFP, User Manual

Page 1: ...Sun Microsystems Inc www sun com Submit comments about this document at http www sun com hwdocs feedback Sun Dual 10GbE XFP PCI Express Card User s Guide Part No 820 0184 13 January 2009 Revision A...

Page 2: ...e un ou plus des brevets am ricains list s l adresse http www sun com patents et un ou les brevets suppl mentaires ou les applications de brevet en attente aux Etats Unis et dans les autres pays Sun S...

Page 3: ...is SPARC or x86 Platform 5 To Download the Driver on a Solaris Platform 5 To Remove the Driver from a Solaris Platform 8 Downloading and Installing the Driver on a Linux Platform 9 To Remove the Drive...

Page 4: ...Linux Systems 29 5 Configuring the nxge Device Driver Parameters 33 nxge Hardware and Software Overview 33 Setting nxge Driver Parameters on a Solaris Platform 34 Setting Parameters Using the ndd Uti...

Page 5: ...To Check Layer 3 Configuration 55 Enabling Jumbo Frames in a Linux Environment 55 To Enable Jumbo Frames in a Linux Environment 55 To Show the Driver Statistics in a Linux Environment 56 7 Configurin...

Page 6: ...rs 71 Performance Specifications 72 Physical Characteristics 73 Power Requirements 73 B Diagnostic Software 75 SunVTS Diagnostic Software 75 Updating SunVTS to Recognize the Adapter 76 To Update SunVT...

Page 7: ...Requirements for Sun SPARC and X64 Servers 2 TABLE A 1 XFP Connector Characteristics 72 TABLE A 2 Performance Specifications 72 TABLE A 3 Physical Characteristics 73 TABLE A 4 Low Profile Power Requir...

Page 8: ...viii Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009...

Page 9: ...Ethernet Fiber XFP Low Profile Adapter 2 FIGURE 8 1 Example of Servers Supporting Multiple VLANs With Tagging Adapters 68 FIGURE 8 2 Ethernet Tag Header Format 70 FIGURE A 1 Sun x8 Express Dual 10 Gi...

Page 10: ...x Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009...

Page 11: ...er as applicable EN 55022 1994 A1 1995 A2 1997 Class A EN 61000 3 2 2000 Pass EN 61000 3 3 1995 A1 2000 Pass EN 55024 1998 A1 2001 A2 2003 Required Limits IEC 61000 4 2 4 kV Direct 8kV Air IEC 61000 4...

Page 12: ...xii Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009...

Page 13: ...es For supported systems see the Solaris Hardware Compatibility Guide How This Document Is Organized Chapter 1 provides an overview of the Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile...

Page 14: ...ss Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter Appendix B provides an overview of the SunVTS diagnostic application and instructions for updating the SunVTS software to recognize the adapte...

Page 15: ...c123 The names of commands files and directories on screen computer output Edit your login file Use ls a to list all files You have mail AaBbCc123 What you type when contrasted with on screen computer...

Page 16: ...ilable on or through such sites or resources Sun will not be responsible or liable for any actual or alleged damage or loss caused by or in connection with the use of or reliance on any such content g...

Page 17: ...umentation and welcomes your comments and suggestions You can submit your comments by going to http www sun com hwdocs feedback Please include the title and part number of your document with your feed...

Page 18: ...xviii Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009...

Page 19: ...the following items Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter with a low profile bracket attached Standard height bracket Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP...

Page 20: ...1 Hardware and Software Requirements for Sun SPARC and X64 Servers Requirements Hardware or Software Hardware Sun Fire T1000 Sun Fire T2000 Sun Fire V445 Sun Fire V245 Sun Fire V215 Sun Fire U45 Sun...

Page 21: ...tworking I O virtualization supporting Solaris LDOM 1 0 VMware support planned for mid 07 Hardware based flow classification for extending parallelism and virtualization to networking Up to 16 Receive...

Page 22: ...4 Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009...

Page 23: ...e data link interfaces and link aggregations This chapter contains the following sections Downloading and Installing the Driver on a Solaris SPARC or x86 Platform on page 5 Downloading and Installing...

Page 24: ...ecture your system is running a For sun4v systems install the software packages by typing the following at the command line b For sun4u systems install the software packages by typing the following at...

Page 25: ...u want to install Press Return or type all to accept the default and install all packages Type the specific numbers separated by a space if you prefer not to install any optional packages usr sbin pkg...

Page 26: ...Processing system information 4 package pathnames are already properly installed Verifying package dependencies Verifying disk space requirements Checking for conflicts with packages already installe...

Page 27: ...etwork interfaces before adding the package by using the ifconfig a command 4 Use the rpm tool to install the driver on SuSe and RedHat Linux nxge 1 0 1 x86_64 rpm ifconfig a grep eth eth0 Link encap...

Page 28: ...loaded after the rpm installation enter the following 5 Verify the new network interface instances corresponding to the Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter The Sun x8...

Page 29: ...r From a Linux Platform To remove the driver packages from a Linux Platform use the rpm e command For 10G ethtool i eth4 driver nxge version 2 0 1 firmware version 2XGF PXE1 47 FCode 3 9 07 04 24 bus...

Page 30: ...12 Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009...

Page 31: ...mic Reconfiguration DR is not supported Installing the Adapter If you are installing the Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter into a machine running Solaris 10 you mus...

Page 32: ...card of the Sun Fire X4600 system Note When you install the adaper ensure that the optical transceiver has not been installed The optical transceiver must be installed into the adapter when it is in...

Page 33: ...dard height bracket for the low profile adapter 6 Open the retaining clip at the edge of the PCI E opening 7 Holding the adapter by the edges align the adapter edge connector with the PCI E slot Slide...

Page 34: ...Caution Do not use excessive force when installing the adapter into the PCI E slot You might damage the adapter s PCI connector If the adapter does not seat properly when you apply even pressure remov...

Page 35: ...r installing the optical tranceiver and shows the most complicated adapter installation Installing an Optical Transceiver The Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter requ...

Page 36: ...e If you are installing an optical transceiver in a SunFire T1000 ensure that the label and the edge connector is visible to you 2 Holding the optical transceiver by the edges align the transceiver wi...

Page 37: ...the second optical transceiver 6 Detach the wrist strap Caution If you pull the locking handle down when the optical transceiver is installed remove the optical transceiver entirely and reinstall it...

Page 38: ...he banner appears press the Stop A key sequence to interrupt the boot process and display the OpenBoot ok prompt 2 List the network devices on your system Checking the properties output for each devic...

Page 39: ...03600000 00000000 00100000 local mac address 00 14 4f 6b ce 88 phy type xgf reg 00080000 00000000 00000000 00000000 00000000 03080010 00000000 00000000 00000000 01000000 03080018 00000000 00000000 00...

Page 40: ...Guide January 2009 5 Type the following when you finish looking at the properties values Rebooting the System After verifying the adapter installation use the boot r command to perform a reconfigurati...

Page 41: ...rk on a Diskless Client System on page 25 Installing the Solaris Operating System Over a 10 Gigabit Ethernet Network on page 27 Booting Over the 10 Gigabit Ethernet Network for Solaris x86 and Linux S...

Page 42: ...g ip address with the adapter s IP address Refer to the ifconfig 1M man page and the Solaris documentation for more information If you want a setup that remains the same after you reboot create an etc...

Page 43: ...diskless client system across a 10 Gigabit Ethernet network you must first install the 10 Gigabit Ethernet software packages into the root directory of the diskless client You can find the 10 Gigabit...

Page 44: ...the server Install the software packages to the client s root directory 4 Create a hostname nxgenumber file in the diskless client s root directory Create an export root client name etc hostname nxgen...

Page 45: ...Gigabit Ethernet software packages to the install server These software packages are on Sun 10 Gigabit Ethernet Driver CD Note Refer to the Solaris Advanced Installation Guide for more information ab...

Page 46: ...er s hard drive The package is a folder SUNWnxge v or SUNWnxge u which you can download from the following web site http www sun com download products xml id 44eb1efd 4 On the install server install t...

Page 47: ...system using the full device path of the 10 Gigabit Ethernet device for example 8 Proceed with the Solaris Operating System installation Refer to the Solaris Advanced Installation Guide for more infor...

Page 48: ...ware is not installed install the software from the download center See Chapter 2 for instructions on installing the required software packages 10 Confirm that the network host files have been configu...

Page 49: ...with the MAC addresses 3 Choose one of the adapter ports as the boot interface 4 Plug the Ethernet cable to the adapter port 5 Power on the system 6 Press the F2 key or the Control E keys to go to the...

Page 50: ...32 Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009 9 Press the F10 key to save the boot configuration changes and exit System should reboot after saving the boot configuration...

Page 51: ...E MOF Exiting Intel Boot Agent NVIDIA Boot Agent 217 0513 Copyright C 2001 2005 NVIDIA Corporation Copyright C 1997 2000 NVIDIA Corporation PXE E61 Media test failure check cable PXE MOF Exiting Intel...

Page 52: ...34 Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009...

Page 53: ...Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter provides two 10 Gigabit Full Duplex networking interfaces The device driver automatically sets the link speed to 10000 Mbit sec and confo...

Page 54: ...oot the system create a platform sun4u kernel drv nxge conf file and add parameter values to this file when you need to set a particular parameter for a device in the system Setting Parameters Using t...

Page 55: ...value which must be specified down to the named dev nxgedriver_instance and assigns the value to the parameter Where X is the driver instance for example dev nxge0 dev nxge1 2 To display the value of...

Page 56: ...ing parameters in a nxge conf file To access any man page type the man command plus the name of the man page For example to access man pages for prtconf 1M type ndd dev nxge1 name to get set read only...

Page 57: ...quotes is the driver name To identify a PCI E device unambiguously in the nxge conf file use the name parent name and the unit address for the device Refer to the pci 4 man page for more information a...

Page 58: ...1 adv_100fdx_cap Advertise 100mbps Full duplex capability default is 1 adv_100fdx_cap 1 adv_10fdx_cap Advertise 10mbps Full duplex capability default is 1 adv_10fdx_cap 1 adv_asmpause_cap Advertise A...

Page 59: ...fine how each IP class is configured Configuration options range from whether TCAM lookup ie is enabled to flow hash generation This parameters also control how the flow template is constructed and ho...

Page 60: ...nd set a parameter for a particular nxge interface Using jumbo support as an example Use the following command to find out the device paths for nxge more etc path_to_inst grep nxge For example if you...

Page 61: ...atform or an AMD CPU based platform To Improve Performance on an UltraSPARC CPU Based Sun Platform 1 Improve performance by adding the following etc system file Increasing the MSI improves the Rx perf...

Page 62: ...n an AMD CPU Based Sun Platform 1 Enable soft rings and change to a higher value than the default of 2 by adding the following to the etc system file Bulk throughput for Rx can be significantly improv...

Page 63: ...he Solaris TCP IP Stack To obtain higher throughput add the following to a startup script ndd set dev tcp tcp_conn_req_max_q 8192 ndd set dev tcp tcp_conn_req_max_q0 8192 ndd set dev tcp tcp_max_buf 4...

Page 64: ...NAME ethtool a DEVNAME ethtool A DEVNAME autoneg on off rx on off tx on off ethtool c DEVNAME ethtool C DEVNAME adaptive rx on off adaptive tx on off rx usecs N rx frames N rx usecs irq N rx frames ir...

Page 65: ...th N ethtool E DEVNAME magic N offset N value N ethtool k DEVNAME ethtool K DEVNAME rx on off tx on off sg on off tso on off ethtool r DEVNAME ethtool p DEVNAME d ethtool t DEVNAME online offline etht...

Page 66: ...to set rx usecs to smaller value For bulk traffic it is recommended to use larger values of rx usecs and control the rate with rx frames rx frames irq controls the maximum number of rx packets process...

Page 67: ...usecs 0 sample interval 0 pkt rate low 0 pkt rate high 0 rx usecs 20 rx frames 512 rx usecs irq 0 rx frames irq 512 tx usecs 0 tx frames 0 tx usecs irq 0 tx frames irq 0 rx usecs low 0 rx frame low 0...

Page 68: ...apter 2 To get a particular variable use the nxge_config if_name get param_name usr local bin nxge_config eth4 get The tunable parameters exported by this device are class_opt_ipv4_tcp Read Write clas...

Page 69: ...s the big gainer net ipv4 tcp_sack 0 sets min default max TCP read buffer default 4096 87380 174760 net ipv4 tcp_rmem 10000000 10000000 10000000 sets min pressure max TCP write buffer default 4096 163...

Page 70: ...52 Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009 2 Set up the sysctl utility sysctl p etc sysctl_nxge conf...

Page 71: ...ronment on page 57 Jumbo Frames Overview Configuring Jumbo Frames enables the Ethernet interfaces to send and receive packets larger than the standard 1500 bytes However the actual transfer size depen...

Page 72: ...example diplays the receive packet counts on all of the eight Receive DMA channels on interface 1 Using the kstat nxge 1 shows all the statistics that the driver supports for that interface kstat nxg...

Page 73: ...frames in both a SPARC and an x86 environment kstat nxge 38001 module nxge instance 38001 name nxge38001 class net brdcstrcv 0 brdcstxmt 0 collisions 0 crtime 3842 493000352 ierrors 0 ifspeed 1000000...

Page 74: ...2 Configuration View the maximum transmission unit MTU configuration of an nxge instance at any time with the kstat command The kstat mac_mtu variable represents the complete size of the Ethernet fram...

Page 75: ...Ensure that the nxge software is installed 2 Plumb the Sun x8 Express Dual 10 Gigabit Ethernet Fiber XFP Low Profile Adapter interface where xxx xxx xx xxx is the IP address of the interface dladm sho...

Page 76: ...ined by the driver ifconfig ethx mtu 9194 ifconfig eth4 eth9 Link encap Ethernet HWaddr 00 14 4F 6C 78 E9 inet addr 38 38 9 42 Bcast 38 38 9 255 Mask 255 255 255 0 inet6 addr fe80 214 4fff fe6c 78e9 6...

Page 77: ...cs use the grep command on the output of ethtool S ethtool S eth9 grep tx_pac tx_packets 8645339 tx_packets 3418334 tx_packets 26 tx_packets 0 tx_packets 22 tx_packets 401791 tx_packets 1466540 tx_pac...

Page 78: ...60 Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009...

Page 79: ...m a link aggregation group This link aggregation group appears to MAC clients as a regular link Link aggregation is defined by IEEE 802 2ad and it provides the following benefits Increased bandwidth L...

Page 80: ...ears in etc aggregation conf file and indicates that the default mode is off as shown in the following example ifconfig down unplumb nxge0 ifconfig down unplumb nxge1 dladm create aggr d nxge0 d nxge1...

Page 81: ...ether 0 3 ba d8 9d e8 ifconfig aggr33 192 168 1 1 24 broadcast up ifconfig aggr33 aggr33 flags 1000843 UP BROADCAST RUNNING MULTICAST IPv4 mtu 1500 index 8 inet 192 168 1 1 netmask ffffff00 broadcast...

Page 82: ...le by adding these lines where bond0 is the bonding device max_bonds is the number of bond interfaces to be created mode specifies the bonding policies miimon is the frequency in milliseconds that MII...

Page 83: ...ring Link Aggregation 65 3 Configure bond0 interface In this example bond0 is the master of the two interfaces eth4 and eth5 ip addr add 192 12 38 64 24 brd dev bond0 ip link set dev bond0 up ifenslav...

Page 84: ...66 Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009...

Page 85: ...essary Overview of VLANs VLANs enable you to split your physical LAN into logical subparts providing an essential tool for increasing the efficiency and flexibility of your network VLANs are commonly...

Page 86: ...bnet consisting of the Main Server Client 3 and Client 5 This represents an engineering group VLAN 2 includes the Main Server Clients 1 and 2 by means of a shared media segment and Client 5 This is a...

Page 87: ...d off Client 3 is a member of VLAN 1 and can communicate only with the Main Server and Client 5 Tagging is not enabled on Client 3 s switch port Client 4 is a member of VLAN 3 and can communicate only...

Page 88: ...ou configure a VLAN virtual device for an adapter all traffic sent or received by that adapter must be in VLAN tagged format To Configure Static VLANs 1 Create one hostname nxgenumber file for each VL...

Page 89: ...ure VLANs in a Linux Environment 1 Ensure that the nxge module is loaded ifconfig nxge123000 plumb up ifconfig nxge224000 plumb up ifconfig a lo0 flags 1000849 UP LOOPBACK RUNNING MULTICAST IPv4 mtu 8...

Page 90: ...le Adapter interface where xxx xxx xx xxx the IP address of the interface 3 Insert the VLAN module 4 Add the VLAN instance VID 5 Configure the nxge VLAN eth2 in this example where xxx xxx xx xxx the I...

Page 91: ...s the following sections Connectors on page 73 Performance Specifications on page 74 Physical Characteristics on page 75 Power Requirements on page 75 Connectors FIGURE A 1 shows the connector for the...

Page 92: ...ce 10GBase SR Depending on fiber quality up to 300 meters for high bandwidth multimode fiber 10GBase LR Depending on fiber quality 10km maximum Fiber type 10GBase SR multimode 10GBase LR single mode T...

Page 93: ...cs Dimension Measurement Length 167 65 mm 6 6 inches Width 68 90 mm 2 713 inches Height Component side 14 48 mm 0 57 inches maximum Solder side 2 67 mm 0 105 inches maximum TABLE A 4 Low Profile Power...

Page 94: ...76 Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009...

Page 95: ...and devices The SunVTS software operates primarily from a graphical user interface enabling test parameters to be set quickly and easily while a diagnostic test operation is being performed Note SunV...

Page 96: ...unVTS to Recognize the Adapter 1 Plug in a loopback cable 2 Ensure that the SunVTS software and the nxge driver are installed on your system 3 Add the following lines to the opt SUNWvts lib conf netlb...

Page 97: ...evice be configured offline before running the test Use the ifconfig 1M command to bring the Ethernet device down before running netlbtest To Use the netlbtest 1 Ensure that the SunVTS software and th...

Page 98: ...80 Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009...

Page 99: ...onfig command 24 installing the adapter without DR 13 installing the Solaris operating environment over the network 27 N ndd utility 37 P parameter values how to modify and display 37 S setting nxge d...

Page 100: ...82 Sun Dual 10GbE XFP PCI Express Card User s Guide January 2009...