Broadcom NetXtreme-E, User Manual

Page 1: ...NetXtreme UG100 August 23 2018 NetXtreme C NetXtreme E User Guide...

Page 2: ...For more information please visit www broadcom com Broadcom reserves the right to make changes without further notice to any products or data herein to improve reliability function or design Informati...

Page 3: ...1 4 5 3 Header Payload Split 11 4 6 UDP Fragmentation Offload 11 4 7 Stateless Transport Tunnel Offload 12 4 8 Multiqueue Support for OS 12 4 8 1 NDIS VMQ 12 4 8 2 VMware NetQueue 12 4 8 3 KVM Xen Mul...

Page 4: ...ux 22 9 System Level Configuration 23 9 1 UEFI HII Menu 23 9 1 1 Main Configuration Page 23 9 1 2 Firmware Image Properties 23 9 1 3 Device Level Configuration 23 9 1 4 NIC Configuration 23 9 1 5 iSCS...

Page 5: ...Configuration and Use Case Examples 35 12 SR IOV Configuration and Use Case Examples 36 12 1 Linux Use Case Example 36 12 2 Windows Use Case Example 37 12 3 VMware SRIOV Use Case Example 38 13 NPAR C...

Page 6: ...Guest OS 48 15 DCBX Data Center Bridging 50 15 1 QoS Profile Default QoS Queue Profile 50 15 2 DCBX Mode Enable IEEE only 51 15 3 DCBX Willing Bit 51 16 DPDK Configuration and Use Case Examples 54 16...

Page 7: ...CNS14336 Class B Table 2 Safety Approvals Country Certification Type Standard Compliance International CB Scheme ICES 003 Digital Device UL 1977 connector safety UL 796 PCB wiring safety UL 94 flammab...

Page 8: ...ons not expressly approved by the manufacture responsible for compliance could void the user s authority to operate the equipment 2 Functional Description The Broadcom NetXtreme C BCM573XX and NetXtre...

Page 9: ...and receive side TCP Checksum Offload Support for transmit and receive side UDP Checksum Offload Support for transmit and receive side NDIS TCP Large Send Offload Support for LSOV1 and LSOV2 NDIS Rece...

Page 10: ...IEEE 802 1q VLAN Supported Interrupt Moderation Supported MAC VLAN filters Supported Table 7 Virtualization Features Feature Details Linux KVM Multiqueue Supported VMware NetQueue Supported NDIS Virtu...

Page 11: ...e TPA context is identified by the 4 tuple Source IP destination IP source TCP port and destination TCP port TPA improves system performance by reducing interrupts for network traffic and lessening CP...

Page 12: ...allows the scaling of virtual machines across multiple processors See Driver Advanced Properties for information on VMQ 4 8 2 VMware NetQueue The VMware NetQueue is a feature that is similar to Micros...

Page 13: ...ver Converge Ethernet RoCE Remote Direct Memory Access RDMA over Converge Ethernet RoCE is a complete hardware offload feature in the BCM5741X that allows RDMA functionality over an Ethernet network R...

Page 14: ...pported RoCE on PFs Up to 4 PFs RoCE on VFs Valid for VFs attached to RoCE enabled PFs Host OS Linux Windows ESXi no vRDMA support Guest OS Linux and Windows DCB Up to two COS per port with non shared...

Page 15: ...dcom NetXtreme E Ethernet adapter verify that the system meets the requirements listed for the operating system 5 2 1 Hardware Requirements Refer to the following list of hardware requirements One ope...

Page 16: ...em case and disconnect any personal antistatic devices 5 4 Connecting the Network Cables Broadcom Ethernet switches are productized with SFP SFP28 QSFP28 ports that support up to 100 Gb s These 100 Gb...

Page 17: ...ware ESXi 6 0 U3 or above Table 11 Linux Ethtool Commands Command Description ethtool s ethX speed 25000 autoneg off Set the speed If the link is up on one port the driver does not allow the other por...

Page 18: ...ethX ntuple on Enable Accelerated RFS Ethtool t ethX Performs various diagnostic self tests echo 32768 proc sys net core rps_sock_flow_entries echo 2048 sys class net ethX queues rx X rps_flow_cnt Ena...

Page 19: ...mnicX esxcli network nic down n vmnicX Disable vmnicX esxcli network nic up n vmnic6 Enable vmnicX bnxtnetcli s n vmnic6 S 25000 Set the link speed Bnxtnetcli is needed for older ESX versions to suppo...

Page 20: ...can Auto negotiate speeds whereas 25 Gb s ports are set to forced speeds SR IOV Enable or Disable Default Enabled This parameter works in conjunction with HW configured SR IOV and BIOS configured SR I...

Page 21: ...Port 1 and Port 2 Reported link speeds are correct and might not match Speed and Duplex setting 0x001B Incompatible speed selection between Port 1 and Port 2 Link configuration became illegal 0x001C...

Page 22: ...ding driver and adding team members to the bond which would load balance the traffic Use the following steps to setup Linux bonding 1 Execute the following command modprobe bonding mode balance alb Th...

Page 23: ...Management CCM version numbers 9 1 3 Device Level Configuration Main configuration page The device level configuration allows the user to enable SR IOV mode number of virtual functions per physical f...

Page 24: ...ers that can be configured using this section are the same as the HII menu NIC configuration 9 2 3 iSCSI Boot Main Menu Parameters that can be configured using this section are the same as the HII men...

Page 25: ...1 AN 1G P1 AN 1G P1 AN 1G P1 AN 1G P2 no AN P2 no AN P2 no AN P2 AN 1G P2 AN 10G P2 AN 25G P2 AN 1 10G P2 AN 1 25G P2 AN 10 25G P2 AN 1 10 25G AN Enabled 10G P1 AN 10G P1 AN 25G Not supported P1 AN 10...

Page 26: ...10 25G Forced 1G 1G No link No link No link No link No link No link No link No link No link Forced 10G No link 10G No link No link No link No link No link No link No link No link Forced 25G No link No...

Page 27: ...speed with the link partner This option must match the AN protocol setting in the link partner port The Broadcom NetXtreme E NIC supports the following auto negotiation protocols IEEE 802 3by 25G 50G...

Page 28: ...esults in esxcli command failure NOTE VMware does not support 25G 50G speeds in ESX6 0 In this case use the second utility BNXTNETCLI to set 25G 50G speed For ESX6 0U2 the 25G 50G speed is supported e...

Page 29: ...arameters via DHCP This option is specific to IPv4 Controls whether the iSCSI boot host software acquires the IP address information using DHCP Enabled or use a static IP configuration Disabled IP Aut...

Page 30: ...Pv4 IP Autoconfiguration Disabled For IPv6 non offload iSCSI parameters via DHCP Disabled CHAP Authentication Disabled DHCP Vendor ID BRCM ISAN Link Up Delay Time 0 Use TCP Timestamp Enabled for some...

Page 31: ...tions in Configuring the DHCP Server to Support iSCSI Boot on page 33 For IPv4 with the exception of the initiator iSCSI name any settings on the Initiator Parameters 1st Target Parameters or 2nd Targ...

Page 32: ...P Version IPv6 For IPv6 non offload 2 Select ESC to return to the Main menu NOTE Information on the Initiator Parameters and 1st Target Parameters screens are ignored and do not need to be cleared 3 S...

Page 33: ...ient The format of the root path as defined in IETC RFC 4173 is iscsi servername protocol port LUN targetname The parameters are defined in Table 19 Table 19 DHCP Option 17 Parameter Definition Parame...

Page 34: ...iguration menu 10 3 2 DHCP iSCSI Boot Configuration for IPv6 The DHCPv6 server can provide a number of options including stateless or stateful IP configuration as well as information to the DHCPv6 cli...

Page 35: ...he VXLAN interface is brought up the host present in system 1 can communicate with host present in system 2 The VLXAN format is shown in Table 22 Table 23 provides VXLAN command and configuration exam...

Page 36: ...fig o boot grub2 grub cfg for legacy mode For UEFI mode edit etc default grub cfg and run grub2 mkconfig o etc grub2 efi cfg Refer to the following example Linuxefi vmlinuz 3 10 0 229 el7 x86_64 root...

Page 37: ...ization mode to SR IOV c Set the number of virtual functions per physical function d Set the number of MSI X vectors per the VF and Max number of physical function MSI X vectors If the VF is running o...

Page 38: ...nce the number of MSI X vectors per VM using CCM 2 Enable virtualization in the BIOS a During system boot enter the system BIOS Processor settings Virtualization Technologies and set it to Enabled b D...

Page 39: ...below For example to enable four VFs on PF 0 and 2 VFs on PF 2 esxcfg module s max_vfs 4 2 bnxtnet reboot required The required VFs of each supported PF are enabled in order during the PF bring up Ref...

Page 40: ...partitions per physical device In NPAR SRIOV mode only two VFs from each parent physical port can enable RDMA support or total of four VFs RDMA per physical device 13 3 Configuration NPAR can be conf...

Page 41: ...partitions that should be allocated from the selected physical port Each BCM5741X NIC can support a maximum of 16 partitions on an ARI capable server By default dual port adapters are configured for...

Page 42: ...ber of RSS queues as fewer cores participate in processing receive packets from that NIC Per processor CPU utilization should be monitored to ensure that there are no hot processors after this change...

Page 43: ...he HII menu in the BIOS before the RDMA option takes effect in the host or guest OS To enable RDMA in single function mode if Virtualization Mode is None or SR IOV During the system boot access the Sy...

Page 44: ...S11SP4 Refer to the OFED release notes at the following link and install OFED before compiling bnxt_re driver http downloads openfabrics org downloads OFED release_notes OFED_3 18 2_release_notes To c...

Page 45: ...using the rpm e kmod bnxt_en command RHEL 7 3 SLES 12 Sp2 has bnxt_en inbox driver older than v1 7 x This driver must be removed and the latest bnxt_en be added before applying the bnxt_re RoCE driver...

Page 46: ...er of RDMA connections by adding multiple IP addresses each with different a subnet for the same physical port under test Multiple network shares can be created and mapped to each link partner using t...

Page 47: ...using the following commands cd var log vmware esxcli software vib install no sig check v bnxtroce driver version vib 2 Reboot the machine 3 Verify that the drivers are correctly installed using the...

Page 48: ...VM To add a PVRDMA device to the VM 1 Select the VM and right click on Edit Settings 2 Add a new Network Adapter 3 Select the network as a Distributed Virtual Switch and Port Group 4 For the Adapter...

Page 49: ...ewall settings may need to be modified to allow RDMA traffic Ensure the proper firewall settings are in place 5 Add the usr lib in the etc ld so conf file and reload the ldconf by running ldconfig 6 L...

Page 50: ...llowing settings in the UEFI HII menu are required to enable DCBX support System Setup Device Settings NetXtreme E NIC Device Level Configuration 15 1 QoS Profile Default QoS Queue Profile Quality of...

Page 51: ...CB specification If the Willing bit on a device is true the device is willing to accept configurations from a remote device through DCBX If the Willing bit on a device is false the device rejects any...

Page 52: ...wControl priority 4 Priority Enabled 4 False Use the following to create QoS Policy PS C New NetQosPolicy Name SMB policy SMB PriorityValue8021Action 4 Name SMB policy Owner Group Policy Machine Netwo...

Page 53: ...46BD A0A 6 4EE0E587B95 NetworkProfile All Precedence 127 IPProtocol Both IPDstPrefix 10 240 1 0 24 PriorityValue 7 Name SMB policy Owner 382AFAD 1E73 46BD A0A 6 4EE0E587B95 NetworkProfile All Preceden...

Page 54: ...huge echo 2048 sys devices system node node0 hugepages hugepages 2048kB nr_hugepages echo 2048 sys devices system node node1 hugepages hugepages 2048kB nr_hugepages modprobe uio insmod build kmod igb_...

Page 55: ...num The default burst number is 1 when burst_num not presented stop Stops packet forwarding and display accumulated statistics testpmd stop quit Quits to prompt testpmd quit 16 5 Display Functions The...

Page 56: ...kout cables are available from QSFP to 4xSFP28 ports What are the compatible port speeds For BCM57404AXXXX BCM57414 dual port devices the port speed of each port must be compatible with the port speed...

Page 57: ...NetXtreme UG100 57 NetXtreme C NetXtreme E User Guide Revision History NetXtreme UG100 August 23 2018 Initial Release...

Page 58: ......