H3C S6890 Series, Troubleshooting Manual

Page 1: ...is manual may be reproduced or transmitted in any form or by any means without prior written consent of New H3C Technologies Co Ltd Except for the trademarks of New H3C Technologies Co Ltd any trademarks that may be mentioned in this document are the property of their respective owners The information in this document is subject to change without notice ...

Page 2: ...13 Related commands 15 Troubleshooting system management 15 High CPU utilization 15 Symptom 15 Troubleshooting flowchart 15 Solution 15 High memory utilization 17 Symptom 17 Troubleshooting flowchart 18 Solution 18 Temperature alarms 20 Symptom 20 Troubleshooting flowchart 20 Solution 20 Related commands 21 Troubleshooting ports 21 10 Gigabit SFP fiber port fails to come up 21 Symptom 21 Troublesh...

Page 3: ...ror packets on a port 37 Symptom 37 Troubleshooting flowchart 38 Solution 38 Related commands 39 Troubleshooting IRF 39 IRF fabric setup failure 39 Symptom 39 Troubleshooting flowchart 40 Solution 40 IRF split 42 Symptom 42 Troubleshooting flowchart 42 Solution 43 BFD MAD failure 43 Symptom 43 Troubleshooting flowchart 44 Solution 44 LACP MAD failure 47 Symptom 47 Troubleshooting flowchart 47 Solu...

Page 4: ...tion cable swapping and reboot Output from the commands executed during the troubleshooting process To ensure safety wear an ESD wrist strap when you replace or maintain a hardware component Collecting log and operating information IMPORTANT By default the information center is enabled If the feature is disabled you must use the info center enable command to enable the feature for collecting log m...

Page 5: ...ember device Display the log file on the master device Sysname dir flash logfile Directory of flash logfile 0 rw 21863 Jul 11 2013 16 00 37 logfile log 1048576 KB total 38812 KB free Display the log file on each subordinate device Sysname dir slot2 flash logfile Directory of flash logfile 0 rw 21863 Jul 11 2013 16 00 37 logfile log 1048576 KB total 38812 KB free 3 Transfer the files to the desired...

Page 6: ...rating statistics 1 Disable pausing between screens of output if you want to display operating statistics on the screen Skip this step if you are saving statistics to a file Sysname screen length disable 2 Collect operating statistics for multiple feature modules Sysname display diagnostic information Save or display diagnostic information Y save N display Y N 3 At the prompt choose to save or dis...

Page 7: ...t you as quickly as possible The following is the contact information for H3C Support Telephone number 400 810 0504 E mail service h3c com Removing deployment errors Use the deployment checklist in Table 2 to eliminate issues that might be introduced at the deployment stage Select items that are suitable for your site Table 2 Deployment checklist Question Command or method Result Remarks Environme...

Page 8: ...gher than 10 execute the debugging ip packet command to view the packets delivered to the CPU for analysis Does the memory usage exceed 60 display memory OK Not OK Not related If memory usage exceeds 60 execute the display memory command to identify the module that is using the most memory Ports Is half duplex used in port negotiation display interface brief OK Not OK Not related If the duplex mod...

Page 9: ...ration Do the peer trunk ports use the same PVID display current configuration interface OK Not OK Not related Make sure the same PVID is configured on the trunk ports between two devices Are the peer ports assigned to the same VLANs display current configuration interface OK Not OK Not related Make sure the trunk ports between two devices are assigned to the same VLANs For example if you assign a...

Page 10: ...king with a Cisco PVST device display stp OK Not OK Not related As a best practice to avoid interoperability issues set up a Layer 3 connection to the Cisco device Do the topologies of MSTIs meet the design Are there as few overlapping paths as possible among MSTIs display current configuration interface OK Not OK Not related If the topologies deviate from the design reassign ports to VLANs and re...

Page 11: ...K Not related If the log contains ARP conflict records verify that the hosts in conflict are legitimate and remove the conflicts OSPF Is the router ID of the device unique on the network display ospf peer OK Not OK Not related Change the router ID if it is not unique on the network To restart route learning after you remove the router ID conflict you must execute the reset ospf process command Are...

Page 12: ...eeded packets are received check for network route errors CPU security Are there packet attacks on CPU debug rxtx softcar show OK Not OK Not related Execute the debug rxtx softcar show command in probe view to view packet rate limit information for cards The CPU is under attack if the number of packets of a type keeps increasing unusually Records in the local log buffer Does the local log buffer c...

Page 13: ...ailure To troubleshoot transceiver modules ports and temperature alarms see Troubleshooting ports and Troubleshooting system management Switch reboot failure Symptom The switch fails to reboot Troubleshooting flowchart Figure 1 Troubleshooting switch reboot failure Solution System software image correct Memory runs correctly Reload the system software image Resolved Contact the support Switch rebo...

Page 14: ...mory is running correctly but there are still errors reported during the BootWare loading process replace the switch 4 If the issue persists contact H3C Support Operating power module failure Symptom An operating power module fails Solution To resolve the issue 1 Identify the operating state of the power module Execute the display power command to view the operating state of the power module Sysna...

Page 15: ...installed power module failure Symptom A newly installed power module fails Solution To resolve the issue 1 Identify the operating state of the power module Execute the display power command to view the operating state of the power module Sysname display power Input Power 132W PowerID State InPower W Current A Voltage V OutPower W Type 1 Absent 2 Normal PSR300 A Execute the display alarm command t...

Page 16: ...e persists contact H3C Support Fan tray failure Symptom An operating fan tray or a newly installed fan tray fails Solution To resolve the issue 1 Identify the operating state of the fan tray Execute the display fan command to view the operating state of the fan tray Sysname display fan Fan tray 1 Status Normal Fan Type LSWM1FANSA Fan number 2 Fan mode Auto Airflow Direction Port to power Fan Speed...

Page 17: ...an tray 3 Verify that the fan tray is operating correctly a Identify whether the fan tray is faulty Execute the display environment command to view temperature information If the temperature continues to rise put your hand at the air outlet to feel if air is being expelled out of the air outlet If no air is being expelled out of the air outlet the fan tray is faulty Execute the display fan command...

Page 18: ...ion provides troubleshooting information for common system management issues High CPU utilization Symptom The sustained CPU utilization on a device is apparently higher than the CPU utilization on other devices Troubleshooting flowchart Figure 2 Troubleshooting high CPU utilization Solution To resolve the issue 1 Identify the job that has a high CPU utilization For example Sysname system view Sysn...

Page 19: ...s for the last 5 seconds 1 minute and 5 minutes Typically the average CPU usage of a job is less than 5 2 Display the job s stack In this example the job uses the ID of 284 Sysname probe follow job 284 slot 1 Attaching to process 284 OPTK Iteration 1 of 5 Kernel stack ffffffff804ad9f0 schedule 0x710 0x1050 ffffffff804ae5d8 schedule_timeout 0x98 0xe0 ffffffff803187d0 kepoll_wait 0x2d0 0x450 fffffff...

Page 20: ... ffffffffc72e1894 thread_boot 0x74 0x90 system ffffffff80266470 kthread 0x140 0x150 ffffffff8021d910 kernel_thread_helper 0x10 0x20 Iteration 5 of 5 Kernel stack ffffffff804ad9f0 schedule 0x710 0x1050 ffffffff804ae5d8 schedule_timeout 0x98 0xe0 ffffffff803187d0 kepoll_wait 0x2d0 0x450 ffffffffc71b29d4 DWARE_OPTMOD_TaskEntry 0xa4 0xd0 system ffffffffc72e1894 thread_boot 0x74 0x90 system ffffffff802...

Page 21: ...226 226 kmalloc 192 22 22 524232 0 1 128 22 22 kmalloc 524232 1288 1344 128 8 21 1 64 64 kmalloc 128 0 0 67108808 0 1 16384 0 0 kmalloc 67108808 630 651 4096 8 7 8 93 93 kmalloc 4096 68 70 131016 0 1 32 68 70 kmalloc 131016 1718 2048 8 8 64 1 31 32 kmalloc 8 1 1 16777160 0 1 4096 1 1 kmalloc 16777160 2 15 2048 0 15 8 1 1 sgpool 64 0 0 40 0 42 1 0 0 inotify_event_cache 325 330 16328 8 2 8 165 165 k...

Page 22: ...2630142 4750157 pid 1 542 cpus 0 2 1 init_dev 0x4dc 0x870 age 510128 pid 542 cpus 2 2 kobj_map_init 0x2c 0xd0 age 4758371 4758535 4758700 pid 0 1 cpus 0 15 2 usb_alloc_dev 0x38 0x200 age 4750540 4750605 4750671 pid 1 cpus 15 1 usb_create_hcd 0x34 0x120 age 4750540 pid 1 cpus 15 16 exception_notifier_init 0x298 0x4f8 age 4750380 4750380 4750381 pid 1 cpus 15 1 drv_port_module_varialbe_init 0x24 0x8...

Page 23: ...he device temperature is higher than the high temperature warning threshold use the display fan command to verify that the fans a operating correctly If the status of a fan is Fault replace the fan 3 Check whether the air filters are clean If they are not clean them Temperature alarm Device temperature too high Y Resolved N N Y Air filters clean Clean air filters N Y Y N Contact the support End Re...

Page 24: ... the temperature alarm thresholds display fan Displays the operating status of all fans on the switch display memory Displays memory usage statistics display process cpu Displays the CPU usage statistics for jobs This command is available in probe view display system internal kernel memory pool Displays kernel memory pool usage information This command is available in probe view follow job job id ...

Page 25: ...e brief command to examine whether the speed and duplex mode of the port match the speed and duplex mode of the transceiver module b If they do not match use the speed command and the duplex command to set the speed and duplex mode for the port 3 Verify that the local and peer ports are operating correctly Speed duplex match on transceiver module and port Resolved Configure matching speed duplex N...

Page 26: ...play transceiver interface command to verify that the local transceiver module has the same wavelength and transmission distance as the peer transceiver module d If the transceiver module or cable is not operating correctly replace it with a new H3C transceiver module or cable that matches the fiber port For more information about transceiver modules and cables see the installation guide for the s...

Page 27: ... and transmission distance as the peer transceiver module d If the transceiver module is not operating correctly replace it with a new H3C transceiver module that matches the fiber port For more information about transceiver modules see the installation guide for the switch 2 Verify that the fiber matches the transceiver module If they do not match replace the fiber with a new one that matches the...

Page 28: ... 25 This transceiver is NOT sold by H3C H3C therefore shall NOT guarantee the normal function of the device or assume the maintenance responsibility thereof Troubleshooting flowchart Figure 7 Troubleshooting non H3C transceiver module error message Solution To resolve the issue 1 Identify whether the transceiver module is an H3C transceiver module Execute the display transceiver interface command ...

Page 29: ...3C Support to verify that the transceiver module is an H3C transceiver module If it is not replace it with an H3C transceiver module 3 If the issue persists contact H3C Support Transceiver module does not support digital diagnosis Symptom The output from the display transceiver diagnosis interface command shows that the transceiver module does not support the digital diagnosis function Sysname dis...

Page 30: ...e is an H3C transceiver module If it is not replace it with an H3C transceiver module 2 Execute the display transceiver interface command to save the transceiver module information and contact H3C Support to verify that the transceiver module supports the digital diagnosis function 3 If the issue persists collect diagnostic information and contact H3C Support To collect diagnostic information exec...

Page 31: ...tatistics and outgoing packet statistics of the port c Determine the type of error frames that are accumulating 2 If the port is a fiber port verify that the optical power of the transceiver module is operating correctly Replace it with a transceiver module with correct optical power Optical power of transceiver module correct Determine the type of error frames accumulating Yes Configure matching ...

Page 32: ...port c If they do not match use the speed command and the duplex command to set the speed and duplex mode for the port 4 Verify that the link medium connected to the port is operating correctly Plug the link medium into a new port that is operating correctly If error frames still exist replace the link medium 5 Determine whether the port has received a large amount of flow control frames a Use the...

Page 33: ...figurations are correct The port configurations include the duplex mode speed port type and VLAN configurations of the ports at both ends of the link If configuration errors exist modify the port configurations If the port still fails to receive packets use the shutdown command and then the undo shutdown command to re enable the port b If the port is configured with the spanning tree feature use t...

Page 34: ... Plug the link medium into a new port that is operating correctly If the new port cannot receive packets replace the link medium 5 If the issue persists collect diagnostic information and contact H3C Support To collect diagnostic information execute the display diagnostic information command to display or save running status data for multiple feature modules Sysname display diagnostic information ...

Page 35: ...e feature configurations to resolve the issue As a best practice configure the port as an edge port or disable the spanning tree feature on the port if it is directly connected to a terminal c If the port is in an aggregation group use the display link aggregation summary command to verify that the status of the port is Selected If the status of the port is Unselected the port cannot send or recei...

Page 36: ...H3C Support b If another port does not have this problem replace the transceiver module If the problem is resolved after replacement contact H3C Support 4 Execute commands in the following table to collect information and contact H3C Support Command Description display transceiver diagnosis interface Displays the current values of the digital diagnosis parameters on transceiver modules display tra...

Page 37: ...are configuration is correct a Read the Tx and Rx optical powers of the port multiple times If a problem exists troubleshoot the link b Execute the following command multiple times If a problem occurs contact H3C Support debug port optical module chassis 1 slot 0 Interface Exist Tx_fault Rx_los Tx_disable Software configuration correct No Yes Yes A port fails to come up Collect information and con...

Page 38: ...e Displays the key parameters of transceiver modules display transceiver manuinfo interface Displays electronic label information for transceiver modules display transceiver information interface Displays transmission information for transceiver modules bcm slot 6 chip 0 phy control xl1 dump Collects the port register information This operation will not cause flapping of ports on the same chip To ...

Page 39: ...the S6890 switch series usage guidelines to identify whether port flapping is caused by known problems If port flapping is caused by known problems upgrade the software or install patches according to the problem description and identify whether the port flapping problem has been resolved 3 Identify whether the problem is caused by the MAC faults of the port MAC faults include the following types ...

Page 40: ... CallLine 1314 Time 03 23 2016 14 03 55 610 linkinfo 0xfd down pma pcs xgs fault localfault 9 CallLine 1314 Time 03 23 2016 05 05 14 565 linkinfo 0xbd down pma pcs xgs fault rmtfault 4 Execute commands in the following table to collect information and contact H3C Support Command Description display transceiver diagnosis interface Displays the current values of the digital diagnosis parameters on t...

Page 41: ...by known problems If CRC error packets are caused by known problems upgrade the software or install patches according to the problem description and identify whether the problem of CRC error packets has been resolved b Troubleshoot the problems of port failure to come up port flapping and CRC error packets on a port together The three symptoms are sometimes related and it is hard to locate the roo...

Page 42: ...and Description display diagnostic information Displays or saves running status data for multiple feature modules display interface Displays Ethernet interface information display interface brief Displays brief interface information display link aggregation summary Displays the summary information for all aggregation groups display logbuffer Displays the state of the log buffer and the log informa...

Page 43: ...ic Yes No Same device model Replace with a device that is the same model as existing IRF member devices Assign a unique member ID to the device Number of member devices fewer than the upper limit Member ID unique No Resolved Physical IRF connections correct Modify IRF port bindings or reconnect physical interfaces Resolved IRF links up Remove link failure conditions to bring up the IRF links Resol...

Page 44: ...t bind a minimum of one physical interface to it The physical interfaces assigned to an IRF port automatically form an aggregate IRF link An IRF port goes down when all its IRF physical interfaces are down a Execute the display irf topology command and then check the Link field b If the Link field for an IRF port displays DOWN execute the display irf link command c Check the Status field for each ...

Page 45: ... Internet protocol processing disabled IP Packet Frame Type PKTFMT_ETHNT_2 Hardware Address 0023 8912 3d07 IPv6 Packet Frame Type PKTFMT_ETHNT_2 Hardware Address 0023 8912 3d07 b If the device has the same bridge MAC address as the IRF fabric remove the bridge MAC conflict 8 If the issue persists execute the display diagnostic information command and collect the device diagnostic information and t...

Page 46: ...te the display version command to identify the uptime of IRF member devices b Compare the uptime of IRF member devices to determine whether a member device had rebooted before the IRF split c If the IRF split is caused by a device reboot perform one of the following tasks If the split is caused by a device reboot use the methods described in Switch reboot failure to resolve the issue If the reboot...

Page 47: ...r BFD MAD are up a Execute the display interface command b Check the Current state field in the command output If the field displays Administratively DOWN for a port execute the undo shutdown command to bring up the port If the field displays DOWN for a port check the physical link for a link failure Correct the settings No Yes End BFD MAD failure Resolved No Yes Contact the support Correct the se...

Page 48: ... links to the BFD MAD VLAN On the IRF fabric create a VLAN interface for the BFD MAD VLAN Make sure the IRF fabrics on the network use different BFD MAD VLANs Make sure the BFD MAD VLAN contains only ports on the BFD MAD links Exclude a port from the BFD MAD VLAN if that port is not on a BFD MAD link If you have assigned that port to all VLANs by using the port trunk permit vlan all command use th...

Page 49: ...Ethernet ports for BFD MAD Connect a management Ethernet port on each IRF member device to the common Ethernet ports on the intermediate device BFD MAD VLAN On the intermediate device create a VLAN for BFD MAD and assign the ports used for BFD MAD to the VLAN On the IRF fabric you do not need to assign the management Ethernet ports to the VLAN Make sure the IRF fabrics on the network use different...

Page 50: ...es No Replace the device or use BFD MAD for detection Intermediate device supports extended LACPDUs for MAD Resolved Each member has a link to the intermediate device Correct the connections Resolved Dynamic aggregation mode used Enable dynamic aggregation mode for the aggregate interface Resolved Aggregate interface and its member ports all up Bring up the aggregate interface and all its member p...

Page 51: ... two IRF fabrics are unique CAUTION The IRF member devices send extended LACPDUs with TLVs that convey the domain ID and the active ID the member ID of the master of the IRF fabric To avoid split detection failure make sure the IRF fabric has a unique domain ID a Execute the display irf command to identify the domain ID of each IRF fabric b If the IRF fabrics use the same domain ID execute the irf...

Page 52: ...ted ACL rules or insufficient resources Symptom The system fails to apply a packet filter or an ACL based QoS policy to the hardware It displays an error message that an unsupported rule exists in the ACL or hardware resources are insufficient The following are sample error messages For unsupported rules Error Slot 2 Fail to apply or refresh packet filter policy 3180 rule 5 on interface Vlan inter...

Page 53: ...xecute the display qos acl resource command to display the resource usage If the number of ACLs to be applied is higher than the value of the Remaining field resources are insufficient If the ACLs are to be applied globally check the resource usage on all cards Sysname display qos acl resource Interfaces XGE1 0 1 to XGE1 0 24 HGE1 0 25 to HGE1 0 30 XGE1 0 31 to XGE1 0 54 MGE0 0 0 to MGE0 0 1 slot ...

Page 54: ...display diagnostic information Y save N display Y N Y ACL application failure without an error message Symptom The system applies a packet filter or an ACL based QoS policy to the hardware However the ACL does not take effect Troubleshooting flowchart Figure 21 Troubleshooting ACL application failure Solution To resolve the issue 1 Check the ACLs used by QoS policies and packet filters for overlap...

Page 55: ...eny filter permit filter deny The behavior configured first 3 Revise ACLs packet filters or QoS policies to remove the behavior conflict 4 If the issue persists collect diagnostic information by using the following command and contact H3C Support Sysname display diagnostic information Save or display diagnostic information Y save N display Y N Y Related commands This section lists the commands tha...