Oracle Solaris Cluster 3.3, System Manual

Page 1: ...Oracle Solaris Cluster 3 3With StorEdge A1000 Array Netra st A1000 Array or StorEdge A3500 System Manual For Solaris OS SPARC Platform Edition Part No E22718 May 2011 Revision A ...

Page 2: ...espect to third party content products and services Oracle Corporation and its affiliates will not be responsible for any loss costs or damages incurred due to your access to or use of third party content products or services Ce logiciel et la documentation qui l accompagne sont protégés par les lois sur la propriété intellectuelle Ils sont concédés sous licence et soumis à des restrictions d util...

Page 3: ... 29 Maintaining Storage Arrays 31 FRUs That Do Not Require Oracle Solaris Cluster Maintenance Procedures 32 How to Remove a Storage Array 34 How to Replace a Failed Controller or Restore an Offline Controller 37 How to Upgrade Controller Module Firmware 39 How to Add a Disk Drive 40 How to Replace a Disk Drive 41 How to Remove a Disk Drive 43 How to Upgrade Disk Drive Firmware 44 How to Replace a ...

Page 4: ...orEdge A3500 Storage System 51 Adding a Sun StorEdge A3500 Storage System 51 Index 53 Contents Oracle Solaris Cluster 3 3With StorEdge A1000 Array Netra st A1000 Array or StorEdge A3500 System Manual May 2011 Revision A 4 ...

Page 5: ...iguration and for system administrators who are responsible for maintaining the system This document is intended for experienced system administrators with extensive knowledge of Oracle software and hardware Do not use this document as a planning or a pre sales guide You should have already determined your system requirements and purchased the appropriate equipment and software before reading this...

Page 6: ... have these books available for your reference The following books support the Oracle Solaris Cluster 3 3 release You can also access the documentation for the Sun Cluster 3 1 and 3 2 releases All Sun Cluster and Oracle Solaris Cluster documentation is available at http www oracle com technetwork indexes documentation index html The related books are organized by the following products and subject...

Page 7: ...500 Storage System Title Part Number OpenBoot 2 x Command Reference Manual 806 2906 10 Sun StorEdge A3500 A3500FC Hardware Configuration Guide 805 4981 TABLE P 6 Oracle Solaris Cluster and Sun Cluster Documentation Documentation Oracle Solaris Cluster 3 3 Sun Cluster 3 2 Sun Cluster 3 1 Using UNIX Commands This document contains information about commands that are used to install configure or upgr...

Page 8: ...and Function prtconf v Displays the size of the system memory and reports information about peripheral devices psrinfo v Displays information about processors showrev p Reports which patches are installed prtdiag v Displays system diagnostic information usr cluster bin clnode show rev Displays Oracle Solaris Cluster release and package version information for each node Also have available the cont...

Page 9: ...een computer output Edit your login file Use ls a to list all files machine_name you have mail AaBbCc123 What you type contrasted with onscreen computer output machine_name su Password aabbcc123 Placeholder replace with a real name or value The command to remove a file is rm filename AaBbCc123 Book titles new terms and terms to be emphasized Read Chapter 6 in the User s Guide A cache is a copy tha...

Page 10: ...nd Bourne shell Bash shell Korn shell and Bourne shell for superuser C shell machine_name C shell for superuser machine_name Preface Oracle Solaris Cluster 3 3With StorEdge A1000 Array Netra st A1000 Array or StorEdge A3500 System Manual May 2011 Revision A 10 ...

Page 11: ...y Oracle s Sun StorEdge A3500 system This chapter contains the following sections Restrictions and Requirements on page 11 Installing Storage Arrays on page 12 Configuring Storage Arrays on page 22 Maintaining Storage Arrays on page 31 Restrictions and Requirements This section includes only restrictions and support information that have a direct impact on the procedures in this chapter For genera...

Page 12: ...I host adapter to connect to any other shared storage you need to update your nvramrc script and set the scsi initiator id by following this procedure Before performing this procedure ensure that you have met the following prerequisites This procedure relies on the following prerequisites and assumptions You have read the entire procedure You can access necessary patches drivers software packages ...

Page 13: ...ice see the service manual that shipped with your storage device b If necessary power on a node but do not allow it to boot If necessary halt the system to continue with OBP Monitor tasks c Set the scsi initiator id for one node to 6 1 ok setenv scsi initiator id 6 scsi initiator id 6 d Find the paths to the host adapters that connect to the local disk drives 0 ok show disks Use this information t...

Page 14: ...ontents of the nvramrc script are incorrect use the nvedit command to make corrections 0 ok printenv nvramrc nvramrc probe all cd pci 1f 4000 scsi 2 7 scsi initiator id integer property device end cd pci 1f 4000 scsi 3 7 scsi initiator id integer property device end install console banner 1 ok h Instruct the OpenBoot PROM OBP Monitor to use the nvramrc script as shown in the following example 0 ok...

Page 15: ...nvironment reboot both nodes reboot Install the RAID Manager software For the procedure about how to install the RAID Manager software see the Sun StorEdge RAID Manager User s Guide For the required version of the RAID Manager software that Oracle Solaris Cluster software supports see Restrictions and Requirements on page 11 Install patches for the controller modules and RAID Manager software For ...

Page 16: ...aid bin hot_add To continue with Oracle Solaris Cluster software and data services installation tasks see your Oracle Solaris Cluster software installation documentation and the Oracle Solaris Cluster data services developer s documentation For a list of Oracle Solaris Cluster documentation see Related Documentation on page 6 How to Add a Storage Array to an Existing Cluster Use this procedure to ...

Page 17: ...strictions and Requirements on page 11 For the procedure about how to install RAID Manager software see the Sun StorEdge RAID Manager Installation and Support Guide For the most current list of software firmware and patches that your storage array or storage system requires refer to the appropriate EarlyNotifier that is outlined in Related Documentation on page 6 This document is available online ...

Page 18: ...d for local devices back to 7 Note A slot in the storage array might not be in use However do not set the scsi initiator id to a SCSI address for that disk slot This precaution minimizes future complications if you install additional disk drives a If necessary power on the storage devices Note If necessary halt the nodes so that you can perform OpenBoot PROM OBP Monitor tasks at the ok prompt For ...

Page 19: ...of the nvramrc script You can continue to edit this copy without risk After you complete your edits save the changes If you are not sure about the changes discard them To store the changes type the following command 0 ok nvstore 1 ok To discard the changes type the following command 0 ok nvquit 1 ok g Verify the contents of the nvramrc script that you created as shown in the following example If t...

Page 20: ... Sun StorEdge A3500 Storage System on page 51 If you did not power off Node B to install a host adapter skip to Step 17 If you powered off Node B to install a host adapter power on Node B and the storage array or storage system Do not enable the node to boot If necessary halt the system to continue with OpenBoot PROM OBP Monitor tasks Verify that Node B recognizes the new host adapters and disk dr...

Page 21: ...er s Guide Verify the controller module firmware revision If necessary install the most recent firmware revision For the revision number and boot level of the controller module firmware see the Sun StorEdge RAID Manager Release Notes For the procedure about how to upgrade the controller firmware see How to Upgrade Controller Module Firmware on page 39 One node at a time boot each node into cluster...

Page 22: ...for nodename dev rdsk cXtYdZsN does not match physical device s id for ddecimalnumber device may have been replaced To fix device IDs that report this error run the cldevice repair command for each affected device TABLE 1 2 Task Map Configuring Disk Drives Task Information Create a logical unit number LUN How to Create a LUN on page 22 Remove a LUN How to Delete a LUN on page 25 Reset the LUN conf...

Page 23: ...enance procedures without volume management disruptions If you use hardware RAID level 0 some maintenance procedures require additional volume management administration because the availability of the LUNs is impacted This procedure provides the long forms of the Oracle Solaris Cluster commands Most commands also have short forms Except for the forms of the command names the commands are identical...

Page 24: ...n How to Correct Mismatched Device ID Numbers on page 29 before you proceed to Step 8 of this procedure A1000 Only If you want a volume manager to manage the new LUN incorporate the new LUN into a diskset or disk group For more information see your Solaris Volume Manager or Veritas Volume Manager documentation Note The StorEdge A3500 system does not support using LUNs as quorum devices Verifying t...

Page 25: ...phys schost 1 format Searching for disks done AVAILABLE DISK SELECTIONS 0 c0t5d0 SYMBIOS StorEdgeA3500FCr 0301 cyl3 alt2 hd64 sec64 pseudo rdnexus 0 rdriver 5 0 1 c0t5d1 SYMBIOS StorEdgeA3500FCr 0301 cyl2025 alt2 hd64 sec64 pseudo rdnexus 0 rdriver 5 1 A1000 Only If the LUN that you are removing is not a quorum device proceed to Step 4 Note Your storage array or storage system might not support LU...

Page 26: ...e following RAID Manager commands rm dev osa dev dsk cNtXdY rm dev osa dev rdsk cNtXdY StorEdge A3500 Only Determine the alternate paths to the LUNs you are deleting The RAID Manager software creates two paths to the LUN in the dev osa dev rdsk directory Substitute the cNtXdY number from the other controller module in the storage array to determine the alternate path For example lad c0t5d0 1T93600...

Page 27: ...aris Cluster commands Most commands also have short forms Except for the forms of the command names the commands are identical To perform this procedure become superuser or assume a role that provides solaris cluster read and solaris cluster modify RBAC authorization From one node that is connected to the storage array or storage system determine the paths to the LUNs you are resetting format For ...

Page 28: ...reset the LUN configuration For the procedure about how to reset the LUN configuration see the Sun StorEdge RAID Manager User s Guide StorEdgeA3500Only Setthecontrollermoduleback toactive active mode For more information about controller modes see the Sun StorEdge RAID Manager Installation and Support Guide and the Sun StorEdge RAID Manager User s Guide Use the format command to label the new LUN ...

Page 29: ...r boot instruction If an error message like the following appears ignore it Continue with the next step device id for dev rdsk c0t5d0 does not match physical disk s id After the node reboots and joins the cluster repeat Step 9 through Step 16 on the other node This node is attached to the storage array or storage system The device ID number for the original LUN 0 is removed A new device ID is assi...

Page 30: ... the LUNs that have different device ID numbers The RAID Manager software creates two paths to the LUN in the dev osa dev rdsk directory Substitute the cNtXdY number from the other controller module in the storage array to determine the alternate path For example lad c0t5d0 1T93600714 LUNS 0 1 c1t4d0 1T93500595 LUNS 2 Therefore the alternate paths are as follows dev osa dev dsk c1t4d1 dev osa dev ...

Page 31: ...hen you perform a check of the device ID configuration by running the cldevice check command the following error message appears on your console if the device ID changed unexpectedly device id for nodename dev rdsk cXtYdZsN does not match physical device s id for ddecimalnumber device may have been replaced To fix device IDs that report this error run the cldevice repair command for each affected ...

Page 32: ...torage devices Netra st A1000 array Sun StorEdge A1000 array Sun StorEdge A3500 system Sun StorEdge A1000 Array and Netra st A1000 Array FRUs The following is a list of administrative tasks that require no cluster specific procedures See the Sun StorEdge A1000 and D1000 Installation Operations and Service Manual and the Netra st A1000 D1000 Installation and Maintenance Manual for these procedures ...

Page 33: ...r or battery harness Replacing a controller module assembly Replacing a controller module power cord Replacing a controller module card cage The following is a list of administrative tasks that require no cluster specific procedures See the Sun StorEdge A3500 A3500FC Controller Module Guide the Sun StorEdge RAID Manager User s Guide the Sun StorEdge RAID Manager Release Notes the Sun StorEdge FC 1...

Page 34: ...e removing are quorum devices proceed to Step 4 Note Your storage array or storage system might not support LUNs as quorum devices To determine if this restriction applies to your storage array or storage system see Restrictions and Requirements on page 11 To determine whether any LUNs in the storage array are quorum devices use the following command clquorum show If one of the LUNs in the storage...

Page 35: ...v dsk cNtXdY rm dev osa dev rdsk cNtXdY On all nodes remove references to the storage array cldevice clear StorEdge A3500 Only Use the lad command to determine the alternate paths to the LUN you are deleting The RAID Manager software creates two paths to the LUN in the dev osa dev rdsk directory Substitute the cNtXdY number from the other controller module in the storage array to determine the alt...

Page 36: ... Perform a reconfiguration boot to create the new Solaris device files and links d Repeat Step a through Step c for Node B that was attached to the storage array or storage system Restore resource groups to their primary nodes Use the following command for each resource group you want to return to the original node clresourcegroup switch n nodename resourcegroup1 resourcegroup2 nodename For failov...

Page 37: ...entical To perform this procedure become superuser or assume a role that provides solaris cluster read and solaris cluster modify RBAC authorization StorEdge A1000 Only If none of the LUNs in the storage array is a quorum device proceed to Step 3 Note Your storage array or storage system might not support LUNs as quorum devices To determine if this restriction applies to your storage array or stor...

Page 38: ...ontroller Module Guide and the Sun StorEdge RAID Manager Installation and Support Guide for additional considerations On one node use the RAID Manager GUI s Recovery application to restore the controller online Note You must use the RAID Manager GUI s Recovery application to bring the controller online Do not use the Redundant Disk Array Controller Utility rdacutil because this utility ignores the...

Page 39: ...ding the NVSRAM file you can use the online method Upgrade the firmware by using the online method as described in the Sun StorEdge RAID Manager User s Guide No special steps are required for a cluster environment If you are upgrading the NVSRAM file you must use an offline method Use one of the following procedures If the data on your controller module is mirrored on another controller module use...

Page 40: ...e offline method as described in the Sun StorEdge RAID Manager User s Guide d Boot both nodes into cluster mode For more information about how to boot nodes see your Oracle Solaris Cluster system administration documentation For a list of Oracle Solaris Cluster documentation see Related Documentation on page 6 This step completes the firmware upgrade How to Add a Disk Drive Adding a disk drive ena...

Page 41: ...30 seconds Run Health Check to ensure that the new disk drive is not defective For instructions about how to run Recovery Guru and Health Check see the Sun StorEdge RAID Manager User s Guide Fail the new drive then revive the drive to update DacStore on the drive For the procedure about how to fail and revive drives see the Sun StorEdge RAID Manager User s Guide Repeat Step 1 through Step 5 for ea...

Page 42: ... that the new disk drive is not defective For the procedure about how to run Recovery Guru and Health Check see the Sun StorEdge RAID Manager User s Guide Does the failed drive belong to a drive group If the drive does not belong to a device group proceed to Step 5 If the drive is part of a device group reconstruction is started automatically If reconstruction does not start automatically for any ...

Page 43: ... nodes are not configured with dynamic reconfiguration functionality If your nodes are configured for dynamic reconfiguration see the Oracle Solaris Cluster system administration documentation and skip steps that instruct you to shut down the node For a list of Oracle Solaris Cluster documentation see Related Documentation on page 6 This procedure provides the long forms of the Oracle Solaris Clus...

Page 44: ...page 6 Caution After you remove the disk drive install a dummy drive to maintain proper cooling How to Upgrade Disk Drive Firmware Caution You must be a Oracle service provider to perform disk drive firmware updates If you need to upgrade drive firmware contact your Oracle service provider How to Replace a Host Adapter Note Several steps in this procedure require you to halt I O activity To halt I...

Page 45: ...de A clnode evacuate fromnode Without powering off the node shut down Node A For the procedure about how to shut down and power off a node see your Oracle Solaris Cluster system administration documentation For a list of Oracle Solaris Cluster documentation see Related Documentation on page 6 From Node B halt I O activity to SCSI bus A For instructions see the Sun StorEdge RAID Manager User s Guid...

Page 46: ...he SCSI cable to connect the controller module to Node A Restart I O activity on SCSI bus A For instructions see the Sun StorEdge RAID Manager User s Guide Did you install a differential SCSI terminator to SCSI bus B in Step 10 If no skip to Step 21 If yes halt I O activity on SCSI bus B then continue with Step 19 Remove the differential SCSI terminator from SCSI bus B Afterward reinstall the SCSI...

Page 47: ...ands devicegroup is one or more device groups that are returned to the node If you moved resource groups off their original node in Step 2 restore the resource groups that you identified in Step 1 to their original node Perform the following step for each resource group you want to return to the original node clresourcegroup switch n nodename resourcegroup1 resourcegroup2 nodename For failover res...

Page 48: ...48 ...

Page 49: ...laris Cluster environment To cable the storage arrays connect the differential SCSI cable between the node and the storage array Ensure that the entire SCSI bus length in each SCSI chain is less than 25 m This measurement includes the cables to both nodes as well as the bus length that is internal to each storage array node and host adapter Figure A 1 and Figure A 2 illustrate a storage array that...

Page 50: ...ray s ID dial is set to a unique number FIGURE A 1 Installing a Storage Array With One Pair Node A Node B FIGURE A 2 Storage Array With Two Pairs and Daisy Chaining Node A Node B Sun StorEdge A1000 and Netra st A1000 Cabling Diagrams Oracle Solaris Cluster 3 3With StorEdge A1000 Array Netra st A1000 Array or StorEdge A3500 System Manual May 2011 Revision A 50 ...

Page 51: ...Sun StorEdge A3500 Storage System Connect the differential SCSI cable between the node and the controller module as shown in Figure A 4 Ensure that the entire SCSI bus length to each storage array is less than 25 m This measurement includes the cables to both nodes as well as the bus length that is internal to each storage array node and host adapter FIGURE A 3 Installing a StorEdge A3500 Storage ...

Page 52: ... SCSI terminators Controller A Controller B FIGURE A 5 Cabling a StorEdge A3500 Storage System An Example Part 2 Node A Node B Sun StorEdge A3500 Cabling Diagrams Oracle Solaris Cluster 3 3With StorEdge A1000 Array Netra st A1000 Array or StorEdge A3500 System Manual May 2011 Revision A 52 ...

Page 53: ...rage arrays 22 31 configuring Continued storage systems 22 31 controller module fan canisters replacing 33 controller modules replacing assemblies 33 replacing card cages 33 replacing power cords 33 upgrading firmware 39 40 controllers replacing and restoring 37 39 cooling canisters replacing 32 correcting DID numbers 29 31 creating hot spares 22 LUNs 22 24 D deleting See also removing hot spares ...

Page 54: ...deleting 25 27 rebalancing 22 LUNs Continued resetting configuration 27 29 M maintaining storage arrays 31 47 storage systems 31 47 N Netra st A1000 procedures 11 47 O Oracle Parallel Server 5 P power cables replacing storage array cables 32 power cords replacing 33 replacing cabinet cords 33 replacing storage array cords 33 power supplies replacing 32 power supply housing replacing 33 R RAID hard...

Page 55: ...3 storage array power cords 33 switches 33 requirements 11 resetting LUN configurations 27 29 restoring controllers 37 39 restrictions 11 S SFPs replacing 33 Small Form Factor Pluggable See SFPs storage array procedures 11 47 storage arrays adding 16 21 configuring 22 31 installing 12 16 maintaining 31 47 removing 34 37 storage systems adding 16 21 cabling 49 52 configuring 22 31 installing 12 16 ...

Page 56: ...56 ...