HP NC570C, User Manual

Page 1: ...HP Dual port 4x Fabric Adapter User Guide November 2004 Second Edition Part Number 377704 002 ...

Page 2: ...ts and services are set forth in the express warranty statements accompanying such products and services Nothing herein should be construed as constituting an additional warranty HP shall not be liable for technical or editorial errors or omissions contained herein HP Dual port 4x Fabric Adapter User Guide November 2004 Second Edition Part Number 377704 002 ...

Page 3: ...mation viii 1 About the Host Channel Adapter HCA 1 HP Dual port 4x Fabric Adapters 1 Supported Protocols 1 HCA Package Contents 2 About the HCA Drivers 2 IPoIB 2 Socket Direct Protocol SDP 2 uDAPL 2 SCSI RDMA SRP 2 MPI 2 Linux Kernels 3 About Boot Over InfiniBand Functionality 3 How Boot Over IB Works 3 Value of Boot over IB 3 2 Installing the Host Channel Adapter HCA 5 Requirements 5 Installation...

Page 4: ...rs 23 Assign Interfaces to HCAs 23 About Assigning Interface for Single HCAs 23 About Assigning Interfaces for Multiple HCAs 23 View all the Interfaces 24 Create Interface Partitions 25 About Dividing an Interface 25 Configuring a Subinterface 26 Verify IPoIB Connectivity 26 Deleting an Interface Partition 27 Run an IPoIB Performance Test 27 5 Configuring MPI Drivers 29 Configure MPI 29 Configure ...

Page 5: ... View the Topology 47 Configure the Fibre Channel Gateway 48 Verify Configurations from the Host 51 Verify SRP Functionality 52 Configure the SRP Target 54 8 Configuring uDAPL Drivers 59 About the uDAPL Configuration 59 Building uDAPL Applications 59 Run a uDAPL Performance Test 60 Run a uDAPL Throughput Test 60 Run a uDAPL Latency Test 61 9 Troubleshooting the HCA Installation 63 Interpret HCA LE...

Page 6: ...B 69 IPoIB Performance vs Ethernet Using netperf 70 Perform a Throughput Test 70 Perform a Latency Test 71 SDP Performance vs IPoIB Using netperf 71 About SDP 71 Configuring SDP 71 Perform a Throughput Test 72 Perform a Latency Test 72 ...

Page 7: ...provide reasonable protection against harmful interference in a residential installation This equipment generates uses and can radiate radio frequency energy and if not installed and used in accordance with the instructions may cause harmful interference to radio communications However there is no guarantee that interference will not occur in a particular installation If this equipment does cause ...

Page 8: ...Stop 510101 Houston Texas 77269 2000 1 281 514 3333 To identify this product refer to the part series or model number found on the product Modifications The FCC requires the user to be notified that any changes or modifications made to this device that are not expressly approved by Hewlett Packard Company may void the user s authority to operate the equipment Cables Connections to this device must...

Page 9: ...lett Packard for this product or product family This compliance is indicated by the following conformity marking placed on the product Japanese Notice Korean Notice BSMI Notice This marking is valid for non Telecom products and EU harmonized Telecom products e g Bluetooth xxxx This marking is valid for EU non harmonized Telecom products Notified body number used only if applicable refer to the pro...

Page 10: ...s To Prevent Electrostatic Damage There are several methods for grounding Use one or more of the following methods when handling or installing electrostatic sensitive parts Use a wrist strap connected by a ground cord to a grounded workstation or computer chassis Wrist straps are flexible straps with a minimum of 1 megohm 10 percent resistance in the ground cords To provide proper ground wear the ...

Page 11: ...da call 1 800 HP INVENT 1 800 474 6836 This service is available 24 hours a day 7 days a week For continuous quality improvement calls may be recorded or monitored Outside the United States and Canada refer to www hp com Table 2 1 Customer Contact Information ...

Page 12: ...x ...

Page 13: ...iated protocol drivers are designed to run in conjunction with an HP Dual port 4x Fabric Adapter The HP Dual port 4x Fabric Adapters feature a full suite of upper layer protocols and APIs Supported Protocols IPoIB Internet Protocol over InfiniBand Refer to IPoIB on page 2 or Configuring IPoIB Drivers on page 23 SDP Socket Direct Protocol Refer to Socket Direct Protocol SDP on page 2 or Configuring...

Page 14: ...ry uDAPL defines a set of APIs that exploits RDMA capabilities uDAPL is installed transparently with the driver library Your application must explicitly support uDAPL uDAPL is transparently installed and requires no further configuration However if your application supports uDAPL it may require additional configuration changes Please refer to your application documentation for more information SCS...

Page 15: ... image The HCA firmware communicates with the connected Server Switch to load the operating system OS from Fibre Channel storage that the Server Switch accesses through the Fibre Channel gateway Once the host loads the image from the target FC storage it boots the OS Value of Boot over IB The Boot over IB feature serves as a manageability tool to help you more easily and centrally administer your ...

Page 16: ...4 ...

Page 17: ... watt less power For maximum performance 133 MHz PCI X or PCI Express is required 100 MHz is the minimum that can be utilized but is not recommended Installation Overview The following steps are required when performing the HCA installation procedure Selecting the Host Connector on page 6 Selecting the Type of Installation on page 8 Installing a High Profile HCA in a PCI X Connector on page 8 Inst...

Page 18: ...bus as performance will most likely be affected However if performance is not a concern and the frequency of the PCI X bus is 100MHz it is permissible to have two devices for example an IB HCA and GE NIC on the same bus If the bus is 133MHz it is mandatory that you remove any other devices so that the InfiniBand HCA is the only device on that bus Consider Cooling Most HCAs have totally passive coo...

Page 19: ...erify with the motherboard documentation that the connector is actually 8x and is supported by the BIOS as 8x Some early version of the PCI Express motherboards had issues on one of the PCI Express connectors If you encounter problems when using the HCA in one of the PCI Express connectors it might help to move the HCA to a different PCI Express connector Consider Cooling Most HCAs have totally pa...

Page 20: ... HCA s in a 2U Host with PCI Express Connectors on page 12 Installing a High Profile HCA in a PCI X Connector The HCA comes preconfigured You do not have to set any jumpers or connectors To install the HCA 1 Note the Global Unique ID GUID numbers from the hardware You will need this number when performing configurations Optionally you can run vstat a utility that is available after host driver ins...

Page 21: ... host system 13 Install the host drivers as described in page 15 14 Connect the InfiniBand cables as described in Connecting the InfiniBand Cables on page 12 Installing a Low Profile HCA in a PCI X Connector The HCA comes preconfigured You do not have to set any jumpers or connectors To install the HCA 1 Note the Global Unique ID GUID numbers from the hardware You will need this number when perfor...

Page 22: ...t drivers as described in page 15 14 Connect the InfiniBand cables as described in Connecting the InfiniBand Cables on page 12 Installing Two HCAs in One Host with PCI X Connectors The HCA comes preconfigured You do not have to set any jumpers or connectors To install the HCAs 1 Note the Global Unique ID GUID numbers from the hardware You will need this number when performing configurations Option...

Page 23: ... 14 Connect the InfiniBand cables as described in Connecting the InfiniBand Cables on page 12 Installing HCA s in a 1U Host with PCI Express Connectors The HCA comes preconfigured You do not have to set any jumpers or connectors To install the HCA 1 Note the Global Unique ID GUID numbers from the hardware You will need this number when performing configurations Optionally you can run vstat a utili...

Page 24: ...the root user 3 Power down the host system 4 Disconnect the power cable Note This is an important step as serious damage could be caused by the standby power accidently being powered on during the HCA installation 5 Ground yourself appropriately to the host chassis Remove the host system cover to access the PCI Express slots 6 Insert the HCA into a PCI Express slot and make sure the InfiniBand por...

Page 25: ...InfiniBand cables from the host to the InfiniBand switch a To plug in an InfiniBand cable push the connector into the interface until you hear feel a click Figure 2 2 Fully Installed IB Cable with Pinch Connector ...

Page 26: ...a pinch connector pinch both sides of the back of the connector and pull the connector away from the port Figure 2 3 Removing a Pinch Connector c To remove a cable with a pull connector grasp the connector with one hand and push it toward the port then pull the latch away from the port with your other hand and gently wiggle the connector away from the port Figure 2 4 Removing a Pull Connector 1 pu...

Page 27: ... that you install all the drivers If you use tsinstall as described all drivers are installed After the installation you can move on to configuring the drivers of your choice Configuring IPoIB Drivers on page 23 Configuring SDP Drivers on page 35 Configuring MPI Drivers on page 29 Configuring SRP Drivers on page 41 Configuring uDAPL Drivers on page 59 Installing HCA Host Drivers To install HCA sof...

Page 28: ...packages If you uninstall then re install the HCA drivers you must reboot the host before accessing the InfiniBand switch Note the log data displayed The following is a sample output of the tsinstall script It lists the OS kernels discovered by the installation program and installed HCA drivers It also lists the OS kernels for which there are currently no available host drivers tsinstall root elro...

Page 29: ...e installation Example 7 Refer to the HP website http support hp com for driver updates installing 100 Upgrading HCA 0 to firmware v2 00 0000 build 0 New Node GUID 0005ad0000001720 New Port1 GUID 0005ad0000001721 New Port2 GUID 0005ad0000001722 Programming Tavor Microcode Flash Image Size 309760 Failsafe Erasing Writing Verifying Flash verify passed root elrond rpm qa grep topspin topspin ib rh9 1...

Page 30: ...nds and check again b Note the hw_ver i e hardware version and fw_ver i e firmware version fields Check with HP Customer Support to determine the appropriate hardware and firmware versions for your HCA root gandalf usr local topspin bin vstat 1 HCA found hca_id InfiniHost0 vendor_id 0x02C9 part_id 0x5A44 hw_ver 0xA1 fw_ver 0x200000000 num_phys_ports 2 port 1 port_state PORT_ACTIVE sm_lid 0x0001 po...

Page 31: ...rotocols and subnet management root gandalf usr local topspin bin vstat 1 HCA found hca_id InfiniHost0 vendor_id 0x02C9 part_id 0x5A44 hw_ver 0xA1 fw_ver 0x200000000 num_phys_ports 2 port 1 port_state PORT_ACTIVE sm_lid 0x0001 port_lid 0x02b9 port_lmc 0x00 max_mtu 2048 gid_tbl_len 32 GID 0 fe 80 00 00 00 00 00 00 00 05 ad 00 00 00 16 70 port 2 port_state PORT_DOWN sm_lid 0x0000 port_lid 0x02ba por...

Page 32: ...ts_ib_useraccess_cm 15520 0 autoclean unused ts_udapl 36904 0 autoclean unused ts_ip2pr 28156 0 autoclean ts_sdp ts_ib_useraccess_cm ts_udapl ts_ipoib 57260 1 autoclean ts_udapl ts_ip2pr lp 9220 0 autoclean parport 39072 0 autoclean lp autofs 13780 1 autoclean nfs 96880 3 autoclean lockd 60624 1 autoclean nfs sunrpc 91996 1 autoclean nfs lockd ts_ib_cm 58808 0 ts_srp_host ts_sdp ts_ib_useraccess_c...

Page 33: ... sbin tvflash h 0 share fw AA BB XX YY 0000 bin Where 0 the HCA number h 0 specifies the HCA 1 h 1 would specify the HCA 2 AA the card type which is Cougar in the following example BB the ASIC revision which is A0 or A1 XX and YY the revision of the firmware file Example The example above shows a firmware upgrade on HCA 1 which has a Cougar ASIC the revision A0 and firmware file revision 1 18 3 Re...

Page 34: ...22 ...

Page 35: ...ut Assigning Interfaces for Multiple HCAs When you are installing multiple HCAs in one server the driver will keep numbering the ports consecutively For example the ports on the second HCA would be interfaces ib2 and ib3 To assign ib interfaces 1 Use ifconfig to assign IP addresses to the ib0 and ib1 interfaces These addresses work like any other IP address on the system Syntax ib is the HCA netwo...

Page 36: ...d as well as interfaces that are available to be configured use the ifconfig a command Interfaces that are configured will display the assigned address Interfaces that are not configured will appear but will not have an address to display root test root usr local topspin sbin root test root ifconfig ib0 192 168 0 0 netmask 255 255 255 0 root test root ifconfig ib1 192 168 0 1 netmask 255 255 255 0...

Page 37: ...B9 FB BROADCAST MULTICAST MTU 1500 Metric 1 RX packets 0 errors 0 dropped 0 overruns 0 frame 0 TX packets 0 errors 0 dropped 0 overruns 0 carrier 0 collisions 0 txqueuelen 1000 RX bytes 0 0 0 b TX bytes 0 0 0 b Interrupt 55 Base address 0x3040 Memory e8220000 e8240000 ib0 Link encap Ethernet HWaddr D8 15 05 AE F3 5A inet addr 192 168 0 2 Bcast 192 168 0 255 Mask 255 255 255 0 UP BROADCAST RUNNING ...

Page 38: ...0b IP addr is the IP address to assign the network interface netmask is a mandatory keyword mask is the netmask for the IP address Example 4 Create partitions on the ports of the InfiniBand server switch if you have not already done so Refer to the HP 24 Port 4x Fabric Copper Switch User Guide for information regarding partitions on the IB switch Verify IPoIB Connectivity Ping between two InfiniBa...

Page 39: ...lete the subinterface and the partition value that has been created on the InfiniBand switch ipoibcfg del parent interface p_key value Example Run an IPoIB Performance Test Refer to IPoIB Performance vs Ethernet Using netperf on page 70 root test root usr local topspin sbin bash usr local topspin sbin is a directory ipoibcfg del ib0 80 0b ...

Page 40: ...28 ...

Page 41: ...TH Variable on page 31 Perform Bandwidth Test on page 32 Perform Latency Test on page 32 For more information about MPI refer to MPI on page 2 The following procedure describes steps that will simplify your use of MPI Configure MPI Before you can rsh MPI you must establish a SSH connection between two hosts so that you can run commands between the nodes without a log in or password ...

Page 42: ...pts you to re enter the password Example 5 Press the Return key again once again omit a password The CLI displays the fingerprint of the host Example login username Password password Last login Tue Aug 31 14 52 42 from 10 10 253 115 You have new mail root qa bc1 blade4 root qa bc1 blade4 ssh keygen t rsa Generating public private rsa key pair Enter file in which to save the key root ssh id_rsa Ent...

Page 43: ...mmands between a local and remote node without a log in or password see Configure SSH on page 30 2 Verify that you do not need to add the compiler to the PATH 3 Add if required the following paths to your environment PATH usr local topspin mpi mpich bin usr local topspin bin qa bc1 blade4 cd ssh qa bc1 blade4 ssh qa bc1 blade4 ssh cp id_rsa pub authorized_keys4 login username Password password Las...

Page 44: ...s to transfer the data integer the number of bytes to transfer integer to perform the bandwidth test Example 3 Enter yes at the prompt to connect to your remote host Example The output in the example 241 250722 represents available bandwidth in MB sec Perform Latency Test Before you perform the bandwidth test configure rsh or ssh on your hosts To perform the test perform the following steps 1 Log ...

Page 45: ... command the number of times to transfer the data integer the number of bytes to transfer integer to run the latency test Example The output in the example 6 684000 represents the latency in microseconds root qa bc1 blade2 root usr local topspin mpi mpich bin mpirun_ssh np 2 qa bc1 blade2 qa bc1 blade3 usr local topspin mpi mpich bin mpi_latency 10000 1 1 6 684000 ...

Page 46: ...34 ...

Page 47: ...e 38 Configure IPoIB Interfaces SDP uses the same IP addresses and interface names as IPoIB 1 Configure the IPoIB IP interfaces if you have not already done so page 23 Specify Connection Overrides 2 Use a text editor to open the libsdp conf file located in usr local topspin etc This file defines when to automatically use SDP instead of TCP You may edit this file to specify connection overrides Con...

Page 48: ... instead of AF_INET when calling the socket system call AF_INET_SDP is defined in usr local topspin include sdp_sock h Explicit application Converts socket streams to use SDP based on the application environment 1 Load the installed libsdp_sys so library in one of the following ways Edit the LD_PRELOAD environment variable Set this to the full path of the library you want to use and it will be pre...

Page 49: ...P addresses or applications that explicitly use SDP by editing the libsdp conf file a locate libsdp conf located in usr local topspin etc b Make the following modifications Match on Destination Port Syntax destination ip_addr prefix_length start_port end_port Example match destination 192 168 1 0 24 Match on Listening Port Syntax listen ip_addr prefix_length start_port end _port Example match list...

Page 50: ... of the benefit that a database environment can achieve Overview To accelerate application performance in database systems an additional library is loaded for all binaries a configuration script is set up to focus the scope of the SDP acceleration to the appropriate processes This needs to be done on both the application server and database server in the same way Sample Topology The following exam...

Page 51: ...enabled properly on the server you should see at least one connection in wait state on the server Configure the Database Server Set up a Preload Script 1 Set up a preload script in order to load the SDP library for all programs Add Configuration Lines to the SDP Initialization Script 2 Add the appropriate configuration lines to the SDP initialization script This sets the host to listen on port 152...

Page 52: ...s using SDP mismatches will not work Configure Other Listeners If you need to set up other connections to clients that are not InfiniBand connected not using SDP you could configure other listeners using port numbers not specified in the libsdp conf file Refer to Examine Configuration Files on page 39 Confine SDP Processes As an alternative to configuring additional listeners you could confine SDP...

Page 53: ...s Auto mount SRP devices by putting them in etc fstab SRP LUNS are automatically configured when the system boots no further configuration is required Note that any LUN changes of Fibre Channel storage requires a host reboot in order for the host to see the changes Verify Configurations from the Host Once you have configured your storage and the Fibre Channel Gateway verify the gateway and the sto...

Page 54: ... 00 Id 00 Lun 00 Vendor SEAGATE Model ST336706LC Rev 010A Type Direct Access ANSI SCSI revision 03 Host scsi0 Channel 00 Id 01 Lun 00 Vendor SEAGATE Model ST336706LC Rev 010A Type Direct Access ANSI SCSI revision 03 Host scsi2 Channel 00 Id 00 Lun 00 Vendor DGC Model RAID 1 Rev 0099 Type Direct Access ANSI SCSI revision 04 Host scsi2 Channel 00 Id 00 Lun 01 Vendor DGC Model RAID 5 Rev 0099 Type Di...

Page 55: ... 00 Lun 00 Vendor SEAGATE Model ST336607LC Rev 0006 Type Direct Access ANSI SCSI revision 03 Scanning for device 0 0 6 0 OLD Host scsi0 Channel 00 Id 06 Lun 00 Vendor SUPER Model GEM318 Rev 0 Type Processor ANSI SCSI revision 02 Scanning for device 0 0 6 9 dd if dev sde of dev null bs 1000k raw dev raw raw1 dev sde Testing dd if dev raw raw1 of dev null bs 512k dd if dev raw raw1 of dev null bs 51...

Page 56: ... fits your scenario the SRP driver will start and discover the LUNs However it will discover only LUN 0 Use the steps below to discover the LUNs in the correct order 2 Rescan the SRP targets from the host to discover all the LUNs Note If you have multiple targets with multiple LUNs the LUNs will be discovered in the wrong order Instead of all the LUNs on one device being discovered first LUN 0 LUN...

Page 57: ...cal Volume Manager LVM based storage subsystem configuration Viewing the Storage Configuration In this example the CX200 configuration is displayed through the EMC Navisphere Management Suite usr local topspin sbin rescan scsi bus sh Host adapter 0 aic79xx found Host adapter 1 aic79xx found Host adapter 2 srp found Scanning for device 0 0 0 0 OLD Host scsi0 Channel 00 Id 00 Lun 00 Vendor SEAGATE M...

Page 58: ...Navisphere Viewing the SRP Host The InfiniBand driver on the SRP host system should be installed by using the procedure provided in Installing HCA Host Drivers on page 15 2 Use vstat to view information on the SRP host From the information below you can conclude the following The Node GUID of the SRP host is 00 05 ad 00 00 01 29 80 The node GUID is located in the GID field The GUID is the last 8 b...

Page 59: ...f switches that will appear in the Topology usr local topspin bin vstat 1 HCA found hca_id InfiniHost0 vendor_id 0x02C9 part_id 0x5A44 hw_ver 0xA1 fw_ver 0x300000002 num_phys_ports 2 port 1 port_state PORT_ACTIVE sm_lid 0x0007 port_lid 0x000f port_lmc 0x00 max_mtu 2048 gid_tbl_len 32 GID 0 fe 80 00 00 00 00 00 00 00 05 ad 00 00 01 29 81 port 2 port_state PORT_DOWN sm_lid 0x0000 port_lid 0x0002 por...

Page 60: ...eway The Fibre Channel gateway used in this example is the one in slot 11 although this particular Topspin 360 has several gateways installed The example shown here is for reference purposes and to help make sense of the SRP host configuration procedure It can also be used in troubleshooting the SRP host configuration ...

Page 61: ...Double click the Fibre Channel gateway The Fibre Channel Port Properties window appears The figure below shows the fibre channel port properties The fibre channel port is directly linked to the CX200 storage with no intermediate fibre channel switches Figure 7 4 Element Manager Detailed Port View Slot 11 ...

Page 62: ...w the General tab Figure 7 5 Element Manager SRP Host Information 4 View information for the SRP targets a Click open the Targets folder from the left navigation bar This example shows the available LUNs that are configured for the SRP host swclus6 Note that only the LUN that is visible via the SP B of the CX200 the last one in the figure will be accessible This is the LUN that has the WWN ending ...

Page 63: ...nager SRP Targets View Verify Configurations from the Host Once you have configured your storage and the Fibre Channel Gateway verify the gateway and the storage configuration from the host Verify the SCSI Devices from the Host The following example shows verification of an EMC CX200 configuration from the SRP host ...

Page 64: ...i scsi Attached devices Host scsi0 Channel 00 Id 00 Lun 00 Vendor SEAGATE Model ST336706LC Rev 010A Type Direct Access ANSI SCSI revision 03 Host scsi0 Channel 00 Id 01 Lun 00 Vendor SEAGATE Model ST336706LC Rev 010A Type Direct Access ANSI SCSI revision 03 Host scsi2 Channel 00 Id 00 Lun 00 Vendor DGC Model RAID 1 Rev 0099 Type Direct Access ANSI SCSI revision 04 Host scsi2 Channel 00 Id 00 Lun 0...

Page 65: ...ervice Active Port GID 0 T10 SRP5006016810201173 fe 80 00 00 00 00 00 00 00 05 ad 00 00 01 29 81 IOC GUID 00 05 ad 00 00 01 1e d8 64 256 255 Number of Pending Connections 0 Number of Active Connections 1 Number of Connections 1 srp_host target_bindings 5006016810201173 0 etc init d ts_srp restart usr local topspin sbin rescan scsi bus sh dd if dev sde of dev null bs 1000k raw dev raw raw1 dev sde ...

Page 66: ...mple 3 Prepare the physical volume Example iostat pkill dd root swclus6 root dd if dev zero of dev sde bs 1k count 1 1 0 records in 1 0 records out root swclus6 root blockdev rereadpt dev sde root swclus6 root vgscan vgscan reading all physical volumes this may take a while vgscan etc lvm tab and etc lvmtab d successfully created vgscan WARNING This program does not do a VGDA backup of your volume...

Page 67: ...ee PE Size 46380 181 17 GB VG UUID qyp8s0 D8zb ES8L m6R4 iRcm nPwF FDA6ny root swclus6 root mkfs t ext3 dev cx200_vg_000 swbld_lv mke2fs 1 32 09 Nov 2002 Filesystem label OS type Linux Block size 4096 log 2 Fragment size 4096 log 2 23724032 inodes 47448064 blocks 2372403 blocks 5 00 reserved for the super user First data block 0 1448 block groups 32768 blocks per group 32768 fragments per group 16...

Page 68: ... 16833 33 0 00 16833 33 0 00 526293 33 09 28 06 PM 13445 00 0 00 13445 00 0 00 419940 00 09 28 07 PM 15966 67 0 00 15966 67 0 00 500977 78 09 28 07 PM tps rtps wtps bread s bwrtn s 09 28 08 PM 14817 65 0 00 14817 65 0 00 468141 18 09 28 09 PM 15629 41 0 00 15629 41 0 00 494494 12 09 28 10 PM 14682 35 0 00 14682 35 0 00 463247 06 09 28 11 PM 15305 88 0 00 15305 88 0 00 483400 00 09 28 11 PM tps rtp...

Page 69: ...nt swbld root swclus6 fsck f dev cx200_vg_000 swbld_lv fsck 1 32 09 Nov 2002 e2fsck 1 32 09 Nov 2002 Pass 1 Checking inodes blocks and sizes Pass 2 Checking directory structure Pass 3 Checking directory connectivity Pass 4 Checking reference counts Pass 5 Checking group summary information dev cx200_vg_000 swbld_lv 54 23724032 files 0 0 non contiguous 752717 47448064 blocks ...

Page 70: ... the LUN Access tab Figure 7 7 Element Manager Storage Manager View c Click onto one of the LUNs in the Accessible LUNs window d Click the Edit ITL Properties button Figure 7 8 Element Manager Storage Manager Edit ITL Properties 10 View the ITL properties Assign a description or set the Port Mask if necessary ...

Page 71: ...PL on page 2 for information about the protocol Building uDAPL Applications on page 59 Run a uDAPL Performance Test on page 60 Building uDAPL Applications 1 The User Direct Access Programming Library uDAPL protocol is transparently installed and requires no further configuration 2 Verify the application requirements a Your application must support uDAPL Please refer to your application documentati...

Page 72: ...u_client x server IP address RDMA size Example ib1 is the name of the device 10 3 2 12 is the IPoIB address of computer 1 262144 is the size in bytes of the RDMA WRITE 3 View the Throughput results Example root cdrom usr local topspin bin thru_server x ib0 262144 500 100 root gcdrom usr local topspin bin thru_server x ib1 10 3 2 12 262144 RDMA throughput server started on ib0 Created an EP with ep...

Page 73: ...r IP address RDMA size Example ib1 is the name of the device 10 3 2 12 is the IPoIB address of computer 1 server device 150000 is the numbers of RDMAs to perform for the test 1 is the size in bytes of the RDMA WRITE 1 is a flag specifying whether polling or event should be used 0 signifies polling and 1 signifies events 3 View the Latency results Example root cdrom usr local topspin bin lat_server...

Page 74: ...62 ...

Page 75: ...erpret HCA LEDs There are two types of LEDs on the HCA card The top yellow LED indicates a logical link has taken place The bottom green LED indicates a physical link has occurred Figure 9 1 The HCA LEDs Table 9 1 Interpreting the LEDs LED Indication Top LED Off indicates there is no logical link detected If this LED is Off but the bottom LED is On then a logical link error has occurred This indic...

Page 76: ... which HCA is first ib0 and ib1 Refer to Verify the HCA and Driver Installation on page 18 Check the InfiniBand Network Interfaces Check for InfiniBand network interfaces using the ifconfig a command You should see interfaces that begin with ib i e ib0 ib1 Use the ifconfig a command to display InfiniBand interfaces If there are no ib0 and ib1 interfaces you may create them automatically or manuall...

Page 77: ...perly connected 4 View another example of the HCA Self test In the example shown in Figure 9 4 both ports on the HCA appear to be disconnected or are not connected properly The following errors appear Port State of Port 0 on HCA 0 is Down Error Counters Failure root 1750 usr local topspin sbin hca_self_test Performing InfiniBand HCA Self Test Number of HCAs Detected 1 PCI Device Check PASS Host Dr...

Page 78: ...HCA Ports Active 0 Port State of Port 0 on HCA 0 DOWN Port State of Port 1 on HCA 0 DOWN Error Counter Check FAIL REASON found errors in the following counters Errors in proc topspin core ca1 port1 counters Symbol error counter 29 Kernel Syslog Check PASS root 1750 cat proc topspin core ca1 port1 counters Symbol error counter 29 Link error recovery counter 0 Link downed counter 1 Port receive erro...

Page 79: ... IPoIB Setup on page 69 IPoIB Performance vs Ethernet Using netperf on page 70 SDP Performance vs IPoIB Using netperf on page 71 Requirements Prerequisites This test plan requires basic knowledge of Linux administration networking protocols and network administration This test of basic functionality and performance of the system should be completed in 3 days or less Hardware and Applications A min...

Page 80: ...he test setup Network Topology The network diagram in Figure 10 1 illustrates the way two servers a switch and an Ethernet network should be connected for basic testing Figure 10 1 Sample Test Topology Host and Switch Setup For basic inter fabric testing of the switch no configuration is required on the switch itself therefore configuration of the switch s management interface can be left for late...

Page 81: ...168 0 2 If the system failed to configure the interface properly you may not have successfully installed the HCA drivers on the operating system If the drivers did not install it is likely due to a version mismatch between the driver suite and the installed kernel 4 To test connectivity attempt to ping Server 2 from Server 1 using the ping command Example If you do not receive a response from the ...

Page 82: ... in gigabits per second d H 192 168 10 21 specifies the host to contact for running the test Example 4 Read the test results The sample results show about wire speed over the Gigabit Ethernet link with around 20 CPU utilization on both ends Example 5 Run the test over the IPoIB interface which was previously setup The results in this example show about a 28 increase in throughput but that has come...

Page 83: ...able connection over the InfiniBand fabric and TCP socket connections can be made without the overhead of TCP RDMA Remote Direct Memory Access semantics are used in the protocol which essentially transmits data between the two host s buffers without CPU intervention Configuring SDP The decision to use this protocol rather than setting up a normal TCP socket is made at the kernel level Applications...

Page 84: ...s been significantly reduced Perform a Latency Test In addition to the Throughput test you can also test the effect of using SDP on the request response test Example In the example above there is approximately a 50 increase in transactions per second from the IPoIB case In addition there is a reduction in CPU utilization on both the transmit and receive end killall netserver netserver sdp netperf ...

Page 85: ...error counters 66 F FCC notices device modifications vi file system 55 firmware upgrade 20 G GID 18 46 grounding methods to prevent electrostatic damage viii grub 45 GUID 18 46 H hardware version 21 HCA initialization 20 HCA self test 65 HCA version 21 high profile installation 8 I IB cable connection 12 ifconfig a 24 64 InfiniBand LEDs 63 InfiniBand partitions 25 initial ram disk 45 initializatio...

Page 86: ...rmance test 60 RDMA thru_client x 60 regulatory compliance notices device modifications vi remove IB cables 14 required speed 5 requirements dual HCA install 7 rescan SRP targets 43 restart 43 RHEL 3 SRP 44 rpm qa 54 rpm qa grep lvm 54 RPMs 59 rsh 29 S sample topology database cluster 38 sbin 21 SCSI show devices from SRP host 52 verify HCA driver on drive 42 SCSI drive local 44 SDP about 2 config...

Page 87: ... 60 tsinstall 16 tvflash 21 U uDAPL about 2 59 application configuation 59 sample make files 59 ULP performance test 67 upgrading firmware 20 upper layer protocols performance test 67 V verify modules lsmod 20 vgscan 54 vstat 18 46 ...