Red Hat CLUSTER MANAGER - INSTALLATION AND, Administration Manual

Page 1: ...Red Hat Cluster Manager The Red Hat Cluster Manager Installation and Administration Guide ...

Page 2: ...rary PowerTools Linux Undercover RHmember RHmember More Rough Cuts Rawhide and all Red Hat based trademarks and logos are trademarks or registered trademarks of Red Hat Inc in the United States and other countries Linux is a registered trademark of Linus Torvalds Motif and UNIX are registered trademarks of The Open Group Itanium is a registered trademark of Intel Corporation Netscape is a register...

Page 3: ...t installation paradigm in order to simplify the end user s experience Addition of support for high availability NFS services Addition of support for high availability Samba services Addition of support for using watchdog timers as a data integrity provision Addition of service monitoring which will automatically restart a failed application Rewrite of the service manager to facilitate additional ...

Page 4: ...3 2 4 Steps for Setting Up and Connecting the Cluster Hardware 39 Chapter 3 Cluster Software Installation and Configuration 55 3 1 Steps for Installing and Initializing the Cluster Software 55 3 2 Checking the Cluster Configuration 62 3 3 Configuring syslog Event Logging 65 3 4 Using the cluadmin Utility 67 Chapter 4 Service Configuration and Administration 73 4 1 Configuring a Service 73 4 2 Disp...

Page 5: ...g the Cluster Software 132 8 3 Removing a Cluster Member 132 8 4 Modifying the Cluster Configuration 133 8 5 Backing Up and Restoring the Cluster Database 133 8 6 Modifying Cluster Event Logging 134 8 7 Updating the Cluster Software 135 8 8 Reloading the Cluster Database 136 8 9 Changing the Cluster Name 136 8 10 Reinitializing the Cluster 136 8 11 Disabling the Cluster Software 137 8 12 Diagnosin...

Page 6: ...2 A 5 SCSI Identification Numbers 162 A 6 Host Bus Adapter Features and Configuration Requirements 163 A 7 Tuning the Failover Interval 168 Appendix B Supplementary Software Information 171 B 1 Cluster Communication Mechanisms 171 B 2 Cluster Daemons 172 B 3 Failover and Recovery Scenarios 173 B 4 Cluster Database Fields 177 B 5 Using Red Hat Cluster Manager with Piranha 179 vi ...

Page 7: ...s into the cluster environment The foun dation of a cluster is an advanced host membership algorithm This algorithm ensures that the cluster maintains complete data integrity at all times by using the following methods of inter node commu nication Quorum partitions on shared disk storage to hold system status Ethernet and serial connections between the cluster systems for heartbeat channels To mak...

Page 8: ... cluster system This service failover capability ensures that no data is lost and there is little disruption to users When the failed system recovers the cluster can re balance the services across the two systems In addition a cluster administrator can cleanly stop the services running on a cluster system and then restart them on the other system This service relocation capability enables the admi...

Page 9: ...roperties for the service including the service name application start and stop script disk partitions mount points and the cluster system on which an administrator prefers to run the service After the administrator adds a service the cluster enters the information into the cluster database on shared storage where it can be accessed by both cluster systems The cluster provides an easy to use frame...

Page 10: ...th of the remote power switch if any and issues heartbeat pings over network and serial channels to mon itor the health of the other cluster system In addition each cluster system periodically writes a timestamp and cluster state information to two quorum partitions located on shared disk storage System state information includes whether the system is an active cluster member Service state informa...

Page 11: ...dware or software failure occurs the cluster will take the appropriate action to maintain ap plication availability and data integrity For example if a cluster system completely fails the other cluster system will restart its services Services already running on this system are not disrupted When the failed system reboots and is able to write to the quorum partitions it can rejoin the cluster and ...

Page 12: ...ling that it will restart on the other cluster member Status Monitoring Agent A cluster status monitoring agent is used to gather vital cluster and application state information This information is then accessible both locally on the cluster member as well as remotely A graphical user interface can then display status information from multiple clusters in a manner which does not degrade system per...

Page 13: ... meet the performance availability and data integrity needs of applications and users Cluster hardware ranges from low cost minimum configurations that include only the com ponents required for cluster operation to high end configurations that include redundant heartbeat channels hardware RAID and power switches Regardless of configuration the use of high quality hardware in a cluster is recommend...

Page 14: ... system In addition if a quorum daemon fails on a cluster system the system is no longer able to monitor the quorum partitions If you are not using power switches in the cluster this error condition may result in services being run on more than one cluster system which can cause data corruption See Section 2 4 2 Configuring Power Switches for more information about the benefits of using power swit...

Page 15: ...ces Ethernet connection for a heartbeat channel and client network access Shared disk storage for the cluster quorum partitions and service data See Section 2 1 5 Example of a Minimum Cluster Configuration for an example of this type of hard ware configuration The minimum hardware configuration is the most cost effective cluster configuration however it in cludes multiple points of failure For exa...

Page 16: ...e configuration Cluster hardware configurations can also include other optional hardware components that are com mon in a computing environment For example a cluster can include a network switch or network hub which enables the connection of the cluster systems to a network A cluster may also include a console switch which facilitates the management of multiple systems and eliminates the need for ...

Page 17: ...conclude that another has failed In this manner a healthy system can assume that prior to taking over services for a failed cluster member that it has safely removed itself from the cluster by rebooting and therefore is no risk to data integrity The underlying watchdog support is included in the core Linux kernel Red Hat Cluster Manager utilizes these watchdog features via its standard APIs and co...

Page 18: ...re as there are two in a cluster Requires purchase of power controller hardware and cables consumes serial ports Network attached power switches A single network attached power controller is required per cluster Affords strong data integrity guarantees Requires purchase of power controller hardware The power controller itself can be come a single point of failure although they are typically very r...

Page 19: ...w redhat com docs Table 2 3 Cluster System Hardware Table Hardware Quantity Description Required Cluster system Two Red Hat Cluster Manager supports IA 32 hardware platforms Each cluster system must provide enough PCI slots network slots and serial ports for the cluster hardware configuration Because disk devices must have the same name on each cluster system it is recommended that the systems hav...

Page 20: ...ch is available from http www wti com rps 10 htm Refer to Section A 1 1 Setting up RPS 10 Power Switches Latent support is provided for the following serial attached power switch This switch has not yet been fully tested APC Serial On Off Switch partAP9211 http www apc com Strongly recommended for data integrity under all failure conditions Null modem cable Two Null modem cables connect a serial p...

Page 21: ...undant power supplies Refer to Section A 1 2 Setting up WTI NPS Power Switches Baytech RPC 3 and RPC 5 http www baytech net Latent support is provided for the APC Master Switch AP9211 or AP9212 www apc com Strongly recommended for data integrity under all failure conditions Watchdog Timer Two Watchdog timers cause a failed cluster member to remove itself from a cluster prior to a healthy member ta...

Page 22: ...arent to the operating system The following are recommended SCSI RAID arrays that provide simultaneous access to all the logical units on the host ports this is not a comprehensive list rather its limited to those RAID boxes which have been tested Winchester Systems FlashDisk RAID Disk Array which is available from http www winsys com Dot Hill s SANnet Storage Systems which is available from http ...

Page 23: ...logic QLA1080 and QLA12160 Tekram Ultra2 DC 390U2W LSI Logic SYM22915 A recommended Fibre Channel host bus adapter is the Qlogic QLA2200 See Section A 6 Host Bus Adapter Features and Configuration Requirements for device features and configuration information Host bus adapter based RAID cards are only supported if they correctly support multi host operation At the time of publication there were no...

Page 24: ...bre Channel cable connects a host bus adapter to a storage enclosure port a Fibre Channel hub or a Fibre Channel switch If a hub or switch is used additional cables are needed to connect the hub or switch to the storage adapter ports Only for Fibre Channel con figurations Table 2 6 Network Hardware Table Hardware Quantity Description Required Network interface One for each network connection Each ...

Page 25: ...o for each channel Each Ethernet heartbeat channel requires a network interface installed in both cluster systems No Network crossover cable One for each channel A network crossover cable connects a network interface on one cluster system to a network interface on the other cluster system creating an Ethernet heartbeat channel Only for a redundant Ethernet heartbeat channel ...

Page 26: ... since configuration of serial heartbeat channels is optional it is not required to invest in additional hardware specifically for this purpose Should future support be provided for more than 2 cluster members serial heartbeat channel support may be deprecated No Null modem cable One for each channel A null modem cable connects a serial port on one cluster system to a corresponding serial port on ...

Page 27: ...ple of a Minimum Cluster Configuration The hardware components described in Table 2 11 Minimum Cluster Hardware Configuration Com ponents can be used to set up a minimum cluster configuration This configuration does not guarantee data integrity under all failure conditions because it does not include power switches Note that this is a sample configuration it is possible to set up a minimum configu...

Page 28: ...ingle Point Of Failure Configuration Components Hardware Quantity Two servers Each cluster system includes the following hardware Two network interfaces for Point to point Ethernet heartbeat channel Client network access and Ethernet heartbeat connection Three serial ports for Point to point serial heartbeat channel Remote power switch connection Connection to the terminal server One Tekram Ultra2...

Page 29: ...ing a point to point serial heartbeat channel FlashDisk RAID Disk Array with dual controllers Dual RAID controllers protect against disk and controller failure The RAID controllers provide simultaneous access to all the logical units on the host ports Two HD68 SCSI cables HD68 cables connect each host bus adapter to a RAID enclosure in port creating two single initiator SCSI buses Two terminators ...

Page 30: ... 2 Set up the optional console switch and connect it to each cluster system See Section 2 2 2 Setting Up a Console Switch for more information about performing this task If a console switch is not used then connect each system to a console terminal 3 Set up the optional network switch or hub and use conventional network cables to connect it to the cluster systems and the terminal server if applica...

Page 31: ...ts Remote power switch connection optional but strongly recommended One SCSI or Fibre Channel adapter to shared disk storage One for each bus adapter Network connection for client access and Ethernet heartbeat One for each net work con nection Point to point Ethernet heartbeat channel optional One for each channel Point to point serial heartbeat channel optional One for each channel Terminal serve...

Page 32: ...external cable connections for a typical cluster configuration 2 2 2 Setting Up a Console Switch Although a console switch is not required for cluster operation it can be used to facilitate cluster system management and eliminate the need for separate monitors mouses and keyboards for each cluster system There are several types of console switches For example a terminal server enables connection t...

Page 33: ...es Follow these steps 1 Install the Red Hat Linux distribution on both cluster systems If customizing the kernel be sure to follow the kernel requirements and guidelines described in Section 2 3 1 Kernel Requirements 2 Reboot the cluster systems 3 When using a terminal server configure Linux to send console messages to the console port 4 Edit the etc hosts file on each cluster system and include t...

Page 34: ...ed before the cluster software For example edit the startup scripts so that the driver is started before the cluster script It is also possible to statically build the SCSI driver into the kernel instead of including it as a loadable module by modifying the etc modules conf file In addition when installing the Linux distribution it is strongly recommended to do the following Gather the IP addresse...

Page 35: ...t creating ramdisks using mkinitrd 2 3 2 Editing the etc hosts File The etc hosts file contains the IP address to hostname translation table The etc hosts file on each cluster system must contain entries for the following IP addresses and associated host names for both cluster systems IP addresses and associated host names for the point to point Ethernet heartbeat connections these can be private ...

Page 36: ...em ifconfig eth0 Link encap Ethernet HWaddr 00 00 BC 11 76 93 inet addr 192 186 1 81 Bcast 192 186 1 245 Mask 255 255 255 0 UP BROADCAST RUNNING MULTICAST MTU 1500 Metric 1 RX packets 65508254 errors 225 dropped 0 overruns 2 frame 0 TX packets 40364135 errors 0 dropped 0 overruns 0 carrier 0 collisions 0 txqueuelen 100 Interrupt 19 Base address 0xfce0 eth1 Link encap Ethernet HWaddr 00 00 BC 11 76...

Page 37: ...lom Y PCI 1 0xd0002000 0xd0005fff IRQ9 4 channels starting from port 0 The following example of the dmesg command output shows that two external SCSI buses and nine disks were detected on the system note that lines with forward slashes will be printed as one line on most screens May 22 14 02 10 storage3 kernel scsi0 Adaptec AHA274x 284x 294x EISA VLB PCI Fast SCSI 5 1 28 3 2 4 May 22 14 02 10 stor...

Page 38: ...forming reset May 22 14 02 11 storage3 kernel SCSI bus is being reset for host 1 channel 0 May 22 14 02 11 storage3 kernel scsi detected 9 SCSI disks total The following example of the dmesg command output shows that a quad Ethernet card was detected on the system May 22 14 02 11 storage3 kernel 3c59x c v0 99H 11 17 98 Donald Becker May 22 14 02 11 storage3 kernel tulip c v0 91g ppc 7 16 99 becker...

Page 39: ... Hardware Configuration for more information about cluster configurations To set up the cluster hardware follow these steps 1 Shut down the cluster systems and disconnect them from their power source 2 Set up the point to point Ethernet and serial heartbeat channels if applicable See Section 2 4 1 Configuring Heartbeat Channels for more information about performing this task 3 When using power swi...

Page 40: ...l has been configured and can recognize the full set of shared disks Use the dmesg command to display console startup messages See Section 2 3 4 Displaying Console Startup Messages for more information about using this command 7 Verify that the cluster systems can communicate over each point to point Ethernet heartbeat con nection by using the ping command to send packets over each network interfa...

Page 41: ...or network connection depending on switch type When failover occurs a cluster system can use this connection to power cycle the other cluster system before restarting its services Power switches protect against data corruption if an unresponsive or hanging system becomes re sponsive after its services have failed over and issues I O to a disk that is also receiving I O from the other cluster syste...

Page 42: ...abling Failure to cable correctly can lead to an incorrect system being power cycled or for one sys tem to inappropriately conclude that it has successfully power cycled another cluster member After setting up the power switches perform these tasks to connect them to the cluster systems 1 Connect the power cable for each cluster system to a power switch 2 On each cluster system connect a serial po...

Page 43: ... disk storage subsystem has two power supplies with separate power cords set up two UPS systems and connect one power switch or one cluster system s power cord if not using power switches and one of the storage subsystem s power cords to each UPS system A redundant UPS system configuration is shown in Figure 2 3 Redundant UPS System Configuration Figure 2 3 Redundant UPS System Configuration An al...

Page 44: ...on information 2 4 4 Configuring Shared Disk Storage In a cluster shared disk storage is used to hold service data and two quorum partitions Because this storage must be available to both cluster systems it cannot be located on disks that depend on the availability of any one system See the vendor documentation for detailed product and installation information There are some factors to consider wh...

Page 45: ... unique SCSI identification number See Section A 2 SCSI Bus Configuration Requirements for more information In addition it is strongly recommended to connect the storage enclosure to redundant UPS systems for a highly available source of power See Section 2 4 3 Configuring UPS Systems for more information See Setting Up a Single Initiator SCSI Bus in Section 2 4 4 and Setting Up a Fibre Channel In...

Page 46: ...r detailed information about terminating host bus adapters and configuring a single ini tiator bus To set up a single initiator SCSI bus configuration the following is required Enable the on board termination for each host bus adapter Enable the termination for each RAID controller Use the appropriate SCSI cable to connect each host bus adapter to the storage enclosure Setting host bus adapter ter...

Page 47: ...ps for Setting Up and Connecting the Cluster Hardware 47 Figure 2 6 Single Controller RAID Array Connected to Single Initiator SCSI Buses Figure 2 7 Dual Controller RAID Array Connected to Single Initiator SCSI Buses ...

Page 48: ...a RAID array that has multiple host ports and the RAID array provides simultaneous access to all the shared logical units from the host ports on the storage enclosure set up two single initiator Fibre Channel interconnects to connect each cluster system to the RAID array If a logical unit can fail over from one controller to the other the process must be transparent to the operating system Figure ...

Page 49: ...ns Two raw devices on shared disk storage must be created for the primary quorum partition and the backup quorum partition Each quorum partition must have a minimum size of 10 MB The amount of data in a quorum partition is constant it does not increase or decrease over time The quorum partitions are used to hold cluster state information Periodically each cluster system writes its status either UP...

Page 50: ...uorum partitions Place both quorum partitions on the same RAID set or on the same disk if RAID is not employed because both quorum partitions must be available in order for the cluster to run Do not put the quorum partitions on a disk that contains heavily accessed service data If possible locate the quorum partitions on disks that contain service data that is rarely accessed See Partitioning Disk...

Page 51: ...r p to create a primary partition Command m for help n Command action e extended p primary partition 1 4 3 Specify the partition number required Partition number 1 4 3 4 Press the Enter key or specify the next available cylinder First cylinder 289 2213 default 289 289 5 Specify the partition size that is required Last cylinder or size or sizeM or sizeK 289 2213 default 2213 2000M Note that large p...

Page 52: ...ng See Creating File Systems in Section 2 4 4 for more information Linux supports raw character devices that are not hard coded against specific block devices Instead Linux uses a character major number currently 162 to implement a series of unbound raw devices in the dev raw directory Any block device can have a character raw device front end even if the block device is loaded later at runtime To...

Page 53: ...d on a 512 byte boundary For more information on using the raw command refer to the raw 8 manual page Creating File Systems Use the mkfs command to create an ext2 file system on a partition Specify the drive letter and the partition number For example mkfs t ext2 b 4096 dev sde3 For optimal performance of shared filesystems a 4 KB block size was specified in the above example Note that it is neces...

Page 54: ...54 Chapter 2 Hardware Installation and Operating System Configuration ...

Page 55: ...Manager currently available This package is installed by default in Red Hat Linux Advanced Server so it is typically not necessary to manually install this individual package To initialize and start the cluster software perform the following tasks 1 Edit the etc sysconfig rawdevices file on both cluster systems and specify the raw device special files and character devices for the primary and back...

Page 56: ...ng tasks on the other cluster system Run the sbin cluconfig init raw_file command where raw_file specifies the primary quorum partition The script will use the information specified for the first cluster sys tem as defaults For example cluconfig init dev raw raw1 4 Check the cluster configuration Invoke the cludiskutil utility with the t option on both cluster systems to ensure that the quorum par...

Page 57: ...ev raw raw2 8 5 dev raw raw1 dev sdb1 dev raw raw2 dev sdb2 See Configuring Quorum Partitions in Section 2 4 4 for more information about setting up the quorum partitions See Creating Raw Devices in Section 2 4 4 for more information on using the raw command to bind raw character devices to block devices Note The rawdevices configuration must be performed on both cluster members 3 1 2 Configuring ...

Page 58: ...d remotely monitored by the cluster GUI This is currently the only security provision controlling cluster monitoring access The cluster GUI is only capable of performing monitoring requests and cannot make any active configuration changes Answering no disables Cluster GUI access completely 3 1 4 Example of the cluconfig Utility This section details an example of the cluconfig cluster configuration...

Page 59: ...about Channel 0 Channel type net or serial net Enter hostname of the cluster member on heartbeat channel 0 storage0 storage0 Looking for host storage0 may take a few seconds Information about Quorum Partitions Enter Primary Quorum Partition dev raw raw1 dev raw raw1 Enter Shadow Quorum Partition dev raw raw2 dev raw raw2 Information About the Power Switch That Power Cycles Member storage0 Choose o...

Page 60: ...APCMASTER o WTI_NPS Power switch RPS10 RPS10 Enter the serial port connected to the power switch dev ttyS0 dev ttyS0 Cluster name Development Cluster Cluster alias IP address 10 0 0 154 Cluster alias netmask 255 255 254 0 Serial port connected to the power switch dev ttyS0 dev ttyS0 Cluster name Development Cluster Cluster alias IP address 10 0 0 154 Cluster alias netmask 255 255 254 0 Member 0 In...

Page 61: ... IP address or hostname storage1 Type RPS10 Login or port dev ttyS0 Password 10 Save the cluster member information yes no yes Writing to configuration file done Configuration information has been saved to etc cluster conf Setting up Quorum Partitions Running cludiskutil I to initialize the quorum partitions done Saving configuration information to quorum partitions done Do you wish to enable moni...

Page 62: ...anager command on each cluster system to display the revision of the installed cluster RPM The following section explains the cluster utilities in further detail 3 2 1 Testing the Quorum Partitions The quorum partitions must refer to the same physical device on both cluster systems Invoke the cludiskutil utility with the t command to test the quorum partitions and verify that they are accessible I...

Page 63: ...on On each cluster system examine the system startup messages at the point where the system probes the SCSI subsystem to determine the bus configuration Verify that both cluster systems identify the same shared storage devices and assign them the same name Verify that a cluster system is not attempting to mount a file system on the quorum partition For example make sure that the actual device for ...

Page 64: ...switch appears below clustonith S Unable to determine power switch type Unable to determine default power switch type The above error could be indicative of the following types of problems For serial attached power switches Verify that the device special file for the remote power switch connection serial port for ex ample dev ttyS0 is specified correctly in the cluster database as established via ...

Page 65: ...tion clustonith r clu3 Successfully power cycled host clu3 3 2 3 Displaying the Cluster Software Version Invoke the rpm qa clumanager command to display the revision of the installed cluster RPM Ensure that both cluster systems are running the same version 3 3 Configuring syslog Event Logging It is possible to edit the etc syslog conf file to enable the cluster to log events to a file that is diff...

Page 66: ... for the cluster daemons are warning and higher Examples of log file entries are as follows May 31 20 42 06 clu2 clusvcmgrd 992 info Service Manager starting May 31 20 42 06 clu2 clusvcmgrd 992 info mount ksh info dev sda3 is not mounted May 31 20 49 38 clu2 clulog 1294 notice stop_service ksh notice Stopping service dbase_home May 31 20 49 39 clu2 clusvcmgrd 1287 notice Service Manager received a...

Page 67: ...ke the killall HUP syslogd command or restart syslog with a command similar to etc rc d init d syslog restart In addition it is possible to modify the severity level of the events that are logged by the individual cluster daemons See Section 8 6 Modifying Cluster Event Logging for more information 3 4 Using the cluadmin Utility The cluadmin utility provides a command line user interface that enabl...

Page 68: ...n command line options are as follows d or debug Displays extensive diagnostic information h or help Displays help about the utility and then exits n or nointeractive Bypasses the cluadmin utility s top level command loop processing This option is used for cluadmin debugging purposes t or tcl Adds a Tcl command to the cluadmin utility s top level command interpreter To pass a Tcl command directly ...

Page 69: ... See Section 8 6 Modifying Cluster Event Logging for information cluster loglevel cluquorumd 7 reload Forces the cluster daemons to re read the cluster configuration database See Section 8 8 Reloading the Cluster Database for information cluster reload name Sets the name of the cluster to the specified name The cluster name is included in the output of the clustat cluster monitoring command See Se...

Page 70: ...se from the specified file See Section 8 5 Backing Up and Restoring the Cluster Database for information cluster re storefrom clus ter_backup conf service add Adds a cluster service to the cluster database The command prompts you for information about service resources and properties See Section 4 1 Configuring a Service for information service add modify Modifies the resources or properties of th...

Page 71: ... service You must enable a service to make it available again See Section 4 3 Disabling a Service for information service disable dbservice enable Starts the specified disabled service See Section 4 4 Enabling a Service for information service enable dbservice delete Deletes the specified service from the cluster configuration database See Section 4 7 Deleting a Service for information service del...

Page 72: ... letter at the prompt and then pressing the Tab key displays the commands that begin with the specified letter Specifying a command and then pressing the Tab key displays a list of all the subcommands that can be specified with that command Users can additionally display the history of cluadmin commands by pressing the up arrow and down arrow keys at the prompt The command history is stored in the...

Page 73: ...erties See Section 4 1 1 Gathering Service Information for information 3 Set up the file systems or raw devices that the service will use See Section 4 1 3 Configuring Service Disk Storage for information 4 Ensure that the application software can run on each cluster system and that the service script if any can start and stop the service application See Section 4 1 4 Verifying Application Softwar...

Page 74: ...ve a unique name A service name can consist of one to 63 characters and must consist of a combination of letters either uppercase or lowercase integers underscores periods and dashes However a service name must begin with a letter or an underscore Preferred member Specify the cluster system if any on which the service will run unless failover has occurred or unless the service is manually relocate...

Page 75: ...system types mount options NFS export options and Samba shares If using a file system specify the type of file system the mount point and any mount options Mount options available to specify are the standard file system mount options that are described in the mount 8 manual page It is not necessary to provide mount information for raw devices if used in a service The ext2 and ext3 file systems are...

Page 76: ...orresponding database start script The format of the service scripts conforms to the conventions followed by the System V init scripts This convention dictates that the scripts have a start stop and status clause These should return an exit status of 0 on success The cluster infrastructure will stop a cluster service that fails to successfully start Inability of a service to start will result in t...

Page 77: ... recommendations For optimal performance use a 4 KB block size when creating file systems Note that some of the mkfs file system build utilities default to a 1 KB block size which can cause long fsck times To facilitate quicker failover times it is recommended that the ext3 filesystem be used Refer to Creating File Systems in Section 2 4 4 for more information For large file systems use the mount ...

Page 78: ...vice Status To display service configuration information invoke the cluadmin utility and specify the service show config command For example cluadmin service show config 0 dummy 1 nfs_pref_clu4 2 nfs_pref_clu3 3 nfs_nopref 4 ext3 5 nfs_eng 6 nfs_engineering c cancel Choose service 6 name nfs_engineering disabled no preferred node clu3 relocate yes IP address 0 172 16 33 164 device 0 dev sdb11 moun...

Page 79: ...ervice for more information To temporarily stop a service A running service can be disabled making it unavailable to clients without having to completely delete the service To disable a running service invoke the cluadmin utility and specify the service disable service_name command For example cluadmin service disable user_home Are you sure yes no y notice Stopping service user_home notice Service...

Page 80: ...min service modify web1 Service properties and resources can also be modified as needed The cluster will check the service modifications and allow correction of any mistakes The cluster will verify the submitted service modification and then start the service unless prompted to keep the service disabled If changes are not submitted the service will be started if possible using the original configu...

Page 81: ... the disabled state if it is unable to successfully start the service A disabled state can be caused by various problems such as a service start did not succeed and the subsequent service stop also failed Be sure to carefully handle failed services If service resources are still configured on the owner sys tem starting the service on the other cluster system may cause significant problems For exam...

Page 82: ...e service to fix incorrect information in the cluster database for example an incorrect start script or you may need to perform manual tasks on the owner system for example unmounting file systems 4 Repeat the attempt to enable or disable the service on the owner system If repeated attempts fail to correct the problem and enable or disable the service reboot the owner system 5 If still unable to s...

Page 83: ...riables used in the service scripts depend on the specific Oracle configuration the example may aid in setting up a service for individual environments See Section 5 2 Tuning Oracle Services for information about improving service performance In the example that follows The service includes one IP address for the Oracle clients to use The service has two mounted file systems one for the Oracle sof...

Page 84: ...sed to start and stop the Oracle service Note that the script is run as user oracle instead of root bin sh Cluster service script to start stop oracle cd home oracle case 1 in start su oracle c startdbi su oracle c startdb stop su oracle c stopdb su oracle c stopdbi esac The following is an example of the startdb script which is used to start the Oracle Database Server instance bin sh Script to st...

Page 85: ...cle software product and administrative file structure export ORACLE_BASE u01 app oracle ORACLE_HOME Specifies the directory containing the software for a given release The Oracle recommended value is ORACLE_BASE product release export ORACLE_HOME u01 app oracle product ORACLE_RELEASE LD_LIBRARY_PATH Required when using Oracle products that use shared libraries export LD_LIBRARY_PATH u01 app oracl...

Page 86: ...tialization parameters file specified u01 app oracle product ORACLE_RELEASE bin svrmgrl EOF spool home oracle startdb log connect internal startup pfile u01 app oracle admin db1 pfile initTESTDB ora open spool off EOF exit 0 The following is an example of the stopdb script which is used to stop the Oracle Database Server instance bin sh Script to STOP the Oracle Database Server instance ORACLE_REL...

Page 87: ...tive file structure export ORACLE_BASE u01 app oracle ORACLE_HOME Specifies the directory containing the software for a given release The Oracle recommended value is ORACLE_BASE product release export ORACLE_HOME u01 app oracle product ORACLE_RELEASE LD_LIBRARY_PATH Required when using Oracle products that use shared libraries export LD_LIBRARY_PATH u01 app oracle product ORACLE_RELEASE lib PATH V...

Page 88: ...of the startdbi script which is used to start a networking DBI proxy daemon bin sh This script allows are Web Server application perl scripts to work in a distributed environment The technology we use is base upon the DBD Oracle DBI CPAN perl modules This script STARTS the networking DBI Proxy daemon export ORACLE_RELEASE 8 1 6 export ORACLE_SID TESTDB export ORACLE_BASE u01 app oracle export ORAC...

Page 89: ...e is base upon the DBD Oracle DBI CPAN perl modules This script STOPS the required networking DBI Proxy daemon PIDS ps ax grep usr bin dbiproxy awk print 1 for pid in PIDS do kill 9 pid done exit 0 The following example shows how to use cluadmin to add an Oracle service cluadmin service add oracle The user interface will prompt you for information about the service Not all information is required ...

Page 90: ... elete or s how an IP address or are you f inished adding IP addresses f Do you want to add a disk device to the service yes no yes Disk Device Information Device special file e g dev sda1 dev sda1 Filesystem type e g ext2 reiserfs ext3 or None ext2 Mount point e g usr mnt service1 or None None u01 Mount options e g rw nosuid Return Forced unmount support yes no no yes Do you want to a dd m odify ...

Page 91: ...ing user script home oracle oracle start notice Server starting Added oracle cluadmin 5 2 Tuning Oracle Services The Oracle database recovery time after a failover is directly proportional to the number of outstanding transactions and the size of the database The following parameters control database recovery time LOG_CHECKPOINT_TIMEOUT LOG_CHECKPOINT_INTERVAL FAST_START_IO_TARGET REDO_LOG_FILE_SI...

Page 92: ... database to accommodate network access by clients of the database service This IP address will automatically be migrated among the cluster members as the service fails over In the example below the IP address is 10 1 16 12 The script that is used to start and stop the MySQL database is the standard System V init script which has been modified with configuration parameters to match the file system...

Page 93: ...mon_user root Run mysqld as this user export PATH mode 1 if test w determine if we should look at the root config file then or user config file conf etc my cnf else conf HOME my cnf Using the users config file fi The following code tries to get the variables safe_mysqld needs from the config file This isn t perfect as this ignores groups but it should work as the options doesn t conflict with anyt...

Page 94: ...ase mode in start Start daemon if test x bindir safe_mysqld then Give extra arguments to mysqld with the my cnf file This script may be overwritten at next upgrade bindir safe_mysqld user mysql_daemon_user pid file pid_file datadir datadir else echo Can t execute bindir safe_mysqld fi stop Stop daemon We use a signal here to avoid having to know the root password if test f pid_file then mysqld_pid...

Page 95: ...ngle character command at a prompt to do one of the following c Cancel and return to the top level cluadmin command r Restart to the initial prompt while keeping previous responses p Proceed with the next prompt Currently defined services databse1 apache2 dbase_home mp3_failover Service name mysql_1 Preferred member None devel0 Relocate when the preferred member joins the cluster yes no no yes Use...

Page 96: ...are you f inished adding device information f Disable service yes no no yes name mysql_1 disabled yes preferred node devel0 relocate yes user script etc rc d init d mysql server IP address 0 10 1 16 12 netmask 0 None broadcast 0 None device 0 dev sda1 mount point device 0 var mysql mount fstype device 0 ext2 mount options device 0 rw force unmount device 0 yes Add mysql_1 service as shown yes no y...

Page 97: ...on the mount point created in Step 3 For example devel0 mount t ext2 dev sda3 db2home 5 On the first cluster system devel0 mount the DB2 cdrom and copy the setup response file included in the distribution to root For example devel0 mount t iso9660 dev cdrom mnt cdrom devel0 cp mnt cdrom IBM DB2 db2server rsp root 6 Modify the setup response file db2server rsp to reflect local configuration setting...

Page 98: ...e first cluster system For example devel0 su db2inst1 devel0 db2stop devel0 exit devel0 su db2as devel0 db2admin stop devel0 exit 10 Unmount the DB2 instance and administration data partition on the first cluster system For ex ample devel0 umount db2home 11 Mount the DB2 instance and administration data partition on the second cluster system devel1 For example devel1 mount t ext2 dev sda3 db2home ...

Page 99: ...e parameter DB2COMM CANCEL Start Administration Serve CANCEL 15 Test the database installation by invoking the following commands first on one cluster system and then on the other cluster system mount t ext2 dev sda3 db2home su db2inst1 db2start db2 connect to sample db2 select tabname from syscat tables db2 connect reset db2stop exit umount db2home 16 Create the DB2 cluster start stop script on t...

Page 100: ...le For example db 234 once etc rc db2 dev console 2 1 Autostart DB2 Services Use the cluadmin utility to create the DB2 service Add the IP address from Step 1 the shared partition created in Step 2 and the start stop script created in Step 16 To install the DB2 client on a third system invoke these commands display mount t iso9660 dev cdrom mnt cdrom display cd mnt cdrom IBM DB2 display db2setup d...

Page 101: ... 4 Setting Up a DB2 Service 101 To test the database from the DB2 client system invoke the following commands db2 connect to db2 user db2inst1 using ibmdb2 db2 select tabname from syscat tables db2 connect reset ...

Page 102: ...102 Chapter 5 Database Services ...

Page 103: ... a few requirements which must be met by each cluster server Note these requirements do not pertain to NFS client systems These require ments are as follows Kernel support for the NFS server must be enabled NFS can be either configured statically or as a module Both NFS V2 and NFS V3 are supported The kernel support for NFS provided with Red Hat Linux Advanced Server 2 1 incorporates en hancements...

Page 104: ...the acting NFS server the client systems should not use the cluster member s hostname as the IP address by which a service is mounted Rather clustered NFS services are assigned floating IP addresses which are distinct from the cluster server s IP addresses This float ing IP address is then configured on whichever cluster member is actively serving the NFS export Following this approach the NFS cli...

Page 105: ...specified portion subdirectory of a mounted filesystem can be mounted instead of the entire filesystem By exporting subdirectories of a mountpoint different access rights can be allocated to different sets of NFS clients Export client names this parameter defines which systems will be allowed to access the file system as NFS clients Under this method individual systems can be designated e g fred a...

Page 106: ...rom that of both cluster members clu3 and clu4 The default netmask and broadcast address will be used Mount Information dev sdb10 which refers to the partition on the shared storage RAID box on which the file system will be physically stored ext3 referring to the file system type which was specified when the file system was created mnt users accounting speci fies the file system mount point rw nos...

Page 107: ...ile e g dev sdb4 dev sdb10 Filesystem type e g ext2 ext3 or None ext3 Mount point e g usr mnt service1 None mnt users accounting Mount options e g rw nosuid sync rw nosuid sync Forced unmount support yes no yes Would you like to allow NFS access to this filesystem yes no no yes You will now be prompted for the NFS export configuration Export directory name mnt users accounting Authorized NFS clien...

Page 108: ...itor interval 30 IP address 0 10 0 0 10 netmask 0 None broadcast 0 None device 0 dev sdb10 mount point device 0 mnt users accounting mount fstype device 0 ext3 mount options device 0 rw nosuid sync force unmount device 0 yes NFS export 0 mnt users accounting Client 0 burke rw Client 1 stevens rw Client 2 needle rw Client 3 dwalsh rw Add nfs_eng service as shown yes no yes Added nfs_eng cluadmin 6 ...

Page 109: ...ment Preferred Member clu3 In this example cluster the member names are clu3 and clu4 Note that here clu3 is specified because the other cluster service nfs_accounting has clu4 specified as its preferred server IP Address 10 0 0 11 There is a corresponding hostname of clunfseng associated with this IP address by which NFS clients mount the filesystem Note that this IP address is distinct from that...

Page 110: ...n Device special file e g dev sdb4 dev sdb11 Filesystem type e g ext2 ext3 or None ext2 Mount point e g usr mnt service1 None mnt users engineering Mount options e g rw nosuid sync rw nosuid sync Forced unmount support yes no yes Would you like to allow NFS access to this filesystem yes no no yes You will now be prompted for the NFS export configuration Export directory name mnt users engineering ...

Page 111: ...ify d elete or s how DEVICES or are you f inished adding DEVICES f Disable service yes no no name nfs_engineering disabled no preferred node clu3 relocate yes user script None monitor interval 30 IP address 0 10 0 0 11 netmask 0 None broadcast 0 None device 0 dev sdb11 mount point device 0 mnt users engineering mount fstype device 0 ext2 mount options device 0 rw nosuid sync force unmount device 0...

Page 112: ...plementation stores file locking information in system files These system files representing NFS locking state are not replicated across the cluster The implication is that locks may be regranted subsequent to the failover operation 6 2 Setting Up a High Availability Samba Service Highly available network file services are one of the key strengths of the clustering infrastructure Advantages of hig...

Page 113: ...nd stopped by the cluster infrastructure on a per service basis Consequently the Samba configuration information should not be specified in the conventional etc samba smb conf The automated system startup of the Samba daemons smbd and nmbd should be disabled in init d run levels For example chkconfig del smb Since the cluster infrastructure stops the cluster related Samba daemons appropriately sys...

Page 114: ...inistrator is required to copy the etc samba smb conf sharename files onto both cluster members After the initial configuration time should any changes be made to any smb conf sharename file it is necessary to also copy this updated version to the other cluster member To facilitate high availability Samba functionality each individual Samba service configured within the cluster via cluadmin will h...

Page 115: ...ts from knowing which specific clus ter member is the acting Samba server the client systems should not use the cluster member s hostname as the IP address by which a service is accessed Rather clustered Samba services are assigned floating IP addresses which are distinct from the cluster server s IP addresses This floating IP address is then configured on which ever cluster member is actively ser...

Page 116: ...hare within an individual smb conf sharename file There can be at most 1 samba configuration specified per service which must be specified with the first device For example if you have multiple disk devices and corresponding file system mounts within a single service then specify a single sharename for the service Then within the etc samab smb conf sharename file designate multiple individual samb...

Page 117: ...should be selected Monitoring Interval 90 seconds IP Address 10 0 0 10 There is a corresponding hostname of cluacct associated with this IP address by which Windows based clients access the share Note that this IP address is distinct from that of both cluster members clu3 and clu4 The default netmask and broadcast address will be used Mount Information dev sdb10 refers to the partition on the shar...

Page 118: ... None ext2 Mount point e g usr mnt service1 None mnt users accounting Mount options e g rw nosuid sync rw nosuid sync Forced unmount support yes no yes Would you like to allow NFS access to this filesystem yes no no no Would you like to share to Windows clients yes no no yes You will now be prompted for the Samba configuration Samba share name acct The samba config file etc samba smb conf acct doe...

Page 119: ...f highly available Samba services It is beyond the scope of this document to completely describe all of the fields within a Samba configuration file There have been no additional field names added in support of clustering and the file format follows the normal Samba conventions Shown below is an example smb conf sharename file which was automatically generated by cluadmin in response to the servic...

Page 120: ...n the smb conf sharename file Note that it is possible to designate more than one share within this file provided that the directories described within it are within the service s filesystem mounts lock directory Dictates the name of the directory in which the Samba daemons smbd nmbd will place their locking files This must be set to var cache samba sharename where sharename varies based on the pa...

Page 121: ...nd then copying over the result ing etc samba smbpasswd file to the other system For example to enable a Windows client system named sarge to access a Samba share served by the cluster members run the following com mand on both cluster members taking care to specify the same username and password each time smbpasswd a sarge On a Windows client the Samba share can then be accessed in the convention...

Page 122: ...rast poorly behaved Windows applications will result in error messages in the event of a failover or relocation indicating the inability to access the share It may be necessary to retry the operation or restart the application in order to enable Windows client systems to reattach to a Samba share for applications that do not correctly behave during failover or service relocation The behavior of a ...

Page 123: ...r the Apache service This will ensure that the corresponding IP address and filesystem mounts are active on only one cluster member at a time When adding an Apache service a floating IP address must be assigned to it The cluster infra structure binds this IP address to the network interface on the cluster system that is currently running the Apache service This IP address ensures that the cluster ...

Page 124: ...ter systems in order for the service to fail over correctly The following example shows a basic Apache Web server installation with no third party modules or performance tuning To install Apache with modules or to tune it for better performance see the Apache documentation that is located in the Apache installation directory or on the Apache Web site http httpd apache org docs project On both clus...

Page 125: ...d apache org docs project 2 The standard Apache start script etc rc d init d httpd will also be used within the cluster framework to start and stop the Apache server on the active cluster member Accordingly when configuring the service specify that script when prompted for the User script Note Depending on the release version the default Apache service script etc rc d init d httpd may not correctl...

Page 126: ...n to the top level cluadmin command r Restart to the initial prompt while keeping previous responses p Proceed with the next prompt Preferred member None devel0 Relocate when the preferred member joins the cluster yes no no yes User script e g usr foo script or None None etc rc d init d httpd Do you want to add an IP address to the service yes no yes IP Address Information IP address 10 1 16 150 N...

Page 127: ...ask 0 255 255 255 0 broadcast 0 10 1 16 255 device 0 dev sde3 mount point device 0 var www html mount fstype device 0 ext3 mount options device 0 rw sync force unmount device 0 yes owner device 0 nobody group device 0 nobody Add apache service as shown yes no y Added apache cluadmin Note The Red Hat Cluster Manager GUI can not be used on clusters where high availability Apache services are configu...

Page 128: ...128 Chapter 7 Apache Services ...

Page 129: ... cluster system on which an administrator is running a tool To obtain comprehensive cluster status run a tool on all cluster systems Cluster and service status includes the following information Cluster member system status Power switch status Heartbeat channel status Service status and which cluster system is running the service or owns the service Service monitoring status of the cluster system ...

Page 130: ...configuration does not include power switches Initializing The switch is in the process of being initialized and its definitive status has not been concluded Table 8 3 Heartbeat Channel Status Heartbeat Channel Status Description OK The heartbeat channel is operating properly Wrn Could not obtain channel status Err A failure or error has occurred ONLINE The heartbeat channel is operating properly ...

Page 131: ... service stop fails or the error state if the status of the service resources cannot be determined stopping The service is in the process of being stopped The stopping state is a transient state The service remains in the stopping state until the service stop succeeds or fails From this state the service can enter the stopped state if the service stop succeeds the running state if the service stop...

Page 132: ... the cluster start command located in the System V etc rc d init directory For example sbin service cluster start Stop the cluster software on a cluster system by invoking the cluster stop command located in the System V etc rc d init directory For example sbin service cluster stop The previous command will cause the cluster system s services to failover to the other cluster sys tem 8 3 Removing a...

Page 133: ...tion 8 2 Starting and Stopping the Cluster Software Then invoke the cluconfig utility and specify the correct information at the prompts After running the utility restart the cluster software 8 5 Backing Up and Restoring the Cluster Database It is recommended to regularly back up the cluster database especially before making any significant changes to the cluster configuration To back up the clust...

Page 134: ...ogged by the clupowerd cluquo rumd cluhbd and clusvcmgrd daemons This is done so that the daemons on the cluster systems will log messages at the same level To change a cluster daemon s logging level on all the cluster systems invoke the cluadmin utility and specify the cluster loglevel command the name of the daemon and the severity level Specify the severity level by using the name or the number...

Page 135: ...her to use the existing cluster database specify yes 4 Stop the cluster software on the second cluster system to be update by invoking the cluster stop command located in the System V init directory At this point no services are available 5 Run cluconfig on the first updated cluster system When prompted whether to use the existing cluster database specify yes The cluster configuration prompts will...

Page 136: ... to reinitialize the cluster systems services and database Be sure to back up the cluster database before reinitializing the cluster See Section 8 5 Backing Up and Restoring the Cluster Database for information To completely reinitialize the cluster follow these steps 1 Disable all the running cluster services 2 Stop the cluster daemons on both cluster systems by invoking the cluster stop command ...

Page 137: ...luster You can then reboot the system or run the cluster start command located in the System V init directory For example sbin service cluster start 8 12 Diagnosing and Correcting Problems in a Cluster To ensure the proper diagnosis of any problems in a cluster event logging must be enabled In addition if problems arise in a cluster be sure to set the severity level to debug for the cluster daemon...

Page 138: ...te a SCSI bus In addition be sure that no devices are connected to a SCSI bus using a stub that is longer than 0 1 meter See Section 2 4 4 Configuring Shared Disk Storage and Section A 3 SCSI Bus Termination for information about terminating different types of SCSI buses SCSI bus length greater than maximum limit SCSI errors appear in the log file Each type of SCSI bus must adhere to restrictions ...

Page 139: ...e completion SCSI errors appear in the log file The prioritized arbitration scheme on a SCSI bus can result in low priority devices being locked out for some period of time This may cause commands to time out if a low priority storage device such as a disk is unable to win arbitration and complete a command that a host has queued to it For some workloads this problem can be avoided by assigning lo...

Page 140: ...m Partitions in Section 2 4 4 for more information These messages could also indicate that the underlying block device special file for the quorum partition has been erroneously used for non cluster purposes Service file system is unclean A disabled service cannot be enabled Manually run a checking program such as fsck Then enable the service Note that the cluster infrastructure does by default ru...

Page 141: ... the etc sysconfig rawde vices file See Configuring Quorum Partitions in Section 2 4 4 for more information These messages could also indicate that yes was not chosen when prompted by the cluconfig utility to initialize the quorum partitions To correct this problem run the utility again Cluster service operation fails Messages indicating the operation failed to appear on the console or in the log ...

Page 142: ...he service and enable forced unmount for the file system This enables the cluster service to unmount a file system even if it is being accessed by an application or user Incorrect entry in the cluster database Cluster operation is impaired The cluadmin utility can be used to examine and modify service configuration Additionally the cluconfig utility is used to modify cluster parameters Incorrect E...

Page 143: ...ation by running the cluconfig utility as specified in Section 8 4 Modifying the Cluster Configuration and correct the problem Heartbeat channel problem Heartbeat channel status is OFFLINE Examine the current settings and modify the cluster configuration by running the cluconfig utility as specified in Section 8 4 Modifying the Cluster Configuration and correct the problem Verify that the correct ...

Page 144: ...144 Chapter 8 Cluster Administration ...

Page 145: ...ARNING The IBM JRE is included and installed by default on Red Hat Linux Advanced Server The installation and use of the Sun JRE with Red Hat Linux Advanced Server is not supported The information in Section 9 1 2 Setting up the Sun JRE is provided only as a convenience to users who wish to deploy it 9 1 1 Setting up the IBM JRE The IBM JRE is automatically installed on the cluster members in the ...

Page 146: ... that the JRE s installation instructions are different for release 6 of Netscape Communicator The following example describes the setup step necessary to configure the Mozilla browser to enable the java plugin ln s usr java jre1 3 1_02 plugin i386 ns600 libjavaplugin_oji so usr lib mozilla plugins 9 2 Configuring Cluster Monitoring Parameters When the cluconfig utility is run to configure the clu...

Page 147: ... cluster member at a time Correct operation of the Cluster Manager GUI requires that Apache s document root remain at the default setting of var www html as this is where the directory cluster and its corresponding web content is installed 9 4 Starting the Red Hat Cluster Manager GUI After configuring the java browser plugin start the Cluster Manager GUI by specifying the appropri ate URL to the b...

Page 148: ...en By double clicking on the cluster name within the tree view the right side of the GUI will then fill with cluster statistics as shown in Figure 9 2 Red Hat Cluster Manager GUI Main Screen These statistics depict the status of the cluster members the services running on each member and the heart beat channel status ...

Page 149: ...will load a dialog allowing modification of the default update interval 9 4 1 Viewing Configuration Details After initiating cluster monitoring it is possible to obtain detailed configuration information by dou ble clicking on any of the cluster status items Following the prior example double click on the nfs_samba1 service and the Service Information window will appear as seen in Figure 9 3 Red H...

Page 150: ...I Configuration Details Screen notice that the detailed device information appears after clicking on the individual device parameters In addition to obtaining detailed configuration information related to cluster services it is also possible to view the configuration of individual cluster members and heartbeat channels by double clicking within the relevant section of the GUI ...

Page 151: ...etup switches on both power switches as follows Table A 1 Setting up RPS 10 Power Switches Switch Function Up Position Down Position 1 Data rate X 2 Toggle delay X 3 Power up default X 4 Unused X Ensure that the serial port device special file for example dev ttyS1 that is specified in the etc cluster conf file corresponds to the serial port to which the power switch s serial cable is connected Co...

Page 152: ...ed Since there is no independent means whereby the cluster software can verify that each cluster member system has been plugged into the appropriate plug on the back of the NPS power switch please take care to ensure correct setup Failure to do so will cause the cluster software to incorrectly conclude that a successful power cycle has occurred When setting up the NPS switch the following configur...

Page 153: ...lug of the NPS 115 gets plugged into a separate power source presumably a separate UPS For cluster members with dual power supplies plug their power cords into an outlet in each bank Then when configuring the NPS 115 and assigning ports simply assign the same name to outlets in each bank that has been plugged the corresponding cluster member For example suppose the cluster members were clu3 and cl...

Page 154: ...in menu then selecting Outlet Control From there it is possible to turn on individual outlets for example on 1 on 2 etc When setting up the Baytech switch the following configuration guidelines should be followed When configuring the Baytech power switch itself 1 Using a serial connection assign the IP address related parameters 2 Under the Access Network access menu ensure that both Prompt for us...

Page 155: ...tlet 5 6 Outlet 6 7 Outlet 7 8 Outlet 8 A 1 4 Setting up Watchdog Power Switches A description of the usage model for watchdog timers as a cluster data integrity provision appears in Section 2 1 3 Choosing the Type of Power Controller As described in that section there are two variants of watchdog timers Hardware based and software based The following details the configuration tasks required in or...

Page 156: ...r Enabling the NMI Watchdog Timer If you are using the software watchdog timer as a data integrity provision it is also recommended to enable the Non Maskable Interrupt NMI watchdog timer to enhance the data integrity guarantees The NMI watchdog timer is a different mechanism for causing the system to reboot in the event of a hang scenario where interrupts are blocked This NMI watchdog can be used...

Page 157: ...ine Here is an example etc grub conf grub conf default 0 timeout 10 splashimage hd0 0 grub splash xpm gz title HA Test Kernel 2 4 9 10smp root hd0 0 This is the kernel s command line kernel vmlinuz 2 4 9 10smp ro root dev hda2 nmi_watchdog 1 end of grub conf On systems using lilo instead of grub add nmi_watchdog 1 to the append section in etc lilo conf For example lilo conf prompt timeout 50 defau...

Page 158: ...ribed previously Again check proc interrupts after the system boots If NMI is non zero the NMI watchdog has been configured properly If NMI is zero your system does not support the NMI watchdog timer Configuring a Hardware Watchdog Timer The kernel provides driver support for various types of hardware watchdog timers Some of these timers are implemented directly on the system board whereas others ...

Page 159: ... place a cor responding entry into the etc modules conf file For example if an Intel 810 based TCO WDT is to be used the following line should be added to etc modules conf alias watchdog i810 tco A 1 5 Other Network Power Switches The cluster software includes support for a range of power switch types This range of power switch module support originated from developers at Mission Critical Linux In...

Page 160: ...ller configuring a cluster without any power switch provisions is not recommended due to data corruption vulnerabilities under certain failover scenarios In order to setup a cluster that does not include any power switch provisions simply select the type NONE when prompted for power switch type from the cluconfig utility Note Usage of power switch type NONE is discouraged because it does not prote...

Page 161: ...ge enclosure and RAID controller termination refer to the vendor documentation See Section A 3 SCSI Bus Termination and Section A 5 SCSI Identification Numbers for more infor mation See http www scsita org and the following sections for detailed information about SCSI bus require ments A 3 SCSI Bus Termination A SCSI bus is an electrical path between two terminators A device host bus adapter RAID ...

Page 162: ...ype Buses that do not adhere to these restrictions will not operate properly The length of a SCSI bus is calculated from one terminated end to the other and must include any cabling that exists inside the system or storage enclosures A cluster supports LVD low voltage differential buses The maximum length of a single initiator LVD bus is 25 meters The maximum length of a multi initiator LVD bus is...

Page 163: ...information The prioritized arbitration scheme on a SCSI bus can result in low priority devices being locked out for some period of time This may cause commands to time out if a low priority storage device such as a disk is unable to win arbitration and complete a command that a host has queued to it For some workloads it is possible to avoid this problem by assigning low priority SCSI identificat...

Page 164: ...lity Onboard termination is disabled when the power is off Set the onboard termination to automatic the default Use the internal SCSI connector for private non cluster storage Qlogic QLA1080 Ultra2 wide LVD VHDCI external connector One channel Set the onboard termination by using the BIOS utility Onboard termination is disabled when the power is off unless jumpers are used to enforce termination S...

Page 165: ...d termination is enabled if there is only one cable connected to the segment Termination is disabled when the power is off Use the internal SCSI connector for private non cluster storage Adaptec 29160 Ultra160 HD68 external connector One channel with two bus segments Set the onboard termination by using the BIOS utility Termination is disabled when the power is off unless jumpers are used to enfor...

Page 166: ... used to enforce termination Set the onboard termination to automatic the default Use the internal SCSI connector for private non cluster storage Adaptec 39160 Qlogic QLA12160 Ultra160 Two VHDCI external connectors Two channels Set the onboard termination by using the BIOS utility Termination is disabled when the power is off unless jumpers are used to enforce termination Set onboard termination t...

Page 167: ...tion even when the module power is off Use jumpers to disable the automatic termination Set onboard termination to automatic the default Use the internal SCSI connectors for private non cluster storage Adaptec AIC 7896 on the Intel L440GX motherboard as used on the VA Linux 2200 series One Ultra2 wide LVD port and one Ultra wide port Onboard termination is permanently enabled so the adapter must b...

Page 168: ...es how to tune configurable parameters associated with the cluqourumd dae mon These parameters control the amount of time that a healthy cluster member will grant a failed cluster member prior to concluding that a failure has occurred After this time interval has elapsed the healthy cluster member will power cycle the failed cluster member depending on the type of power switch in use and resume se...

Page 169: ...her cluster member has failed After a period of pingInterval sameTimeNetdown which by default is 14 seconds total the failover will com mence In order to tune this failover interval the cludb utility is used For example to modify the failover interval down to a duration of 10 seconds rather than the default of 14 seconds the value of the same TimeNetdown parameter should be set to 5 as follows clu...

Page 170: ...170 Appendix A Supplementary Hardware Information ...

Page 171: ...e primary quorum partition If this partition is corrupted the cluster systems read the infor mation from the backup quorum partition and simultaneously repair the primary partition Data consistency is maintained through checksums and any inconsistencies between the partitions are automatically corrected If a cluster system reboots but cannot write to both quorum partitions the system will not be a...

Page 172: ... system status information from the primary quorum partition or if the primary partition is corrupted from the backup partition Heartbeat daemon On each cluster system the cluhbd heartbeat daemon issues pings across the point to point Eth ernet and serial lines to which both cluster systems are connected Power daemon On each cluster system the clupowerd power daemon monitors the remote power switc...

Page 173: ...m restarts any services that were running on the hung system 4 If the previously hung system reboots and can join the cluster that is the system can write to both quorum partitions services are re balanced across the member systems according to each service s placement policy In a cluster configuration that does not use power switches if a system hangs the cluster behaves as follows 1 The function...

Page 174: ...ms according to each service s placement policy B 3 3 Inaccessible Quorum Partitions Inaccessible quorum partitions can be caused by the failure of a SCSI or Fibre Channel adapter that is connected to the shared disk storage or by a SCSI cable becoming disconnected to the shared disk storage If one of these conditions occurs and the SCSI bus remains terminated the cluster behaves as follows 1 The ...

Page 175: ...vices that were running on the system that was shut down are restarted on the remaining cluster system 4 If the system reboots and can join the cluster that is the system can write to both quorum partitions services are re balanced across the member systems according to each service s placement policy B 3 5 Remote Power Switch Connection Failure If a query to a remote power switch connection fails...

Page 176: ...till communicating over the heartbeat channels 2 The functional cluster system restarts any services that were running on the cluster system whose quorum daemon has failed Under the unlikely event of catastrophic failure both cluster systems may be running services simultaneously which can cause data corruption B 3 7 Heartbeat Daemon Failure If the heartbeat daemon fails on a cluster system servic...

Page 177: ...er configuration When you run cluconfig the site specific information is entered into fields within the members section of the database The following is each cluster member field and its subsequent description start member0 start chan0 device serial_port type serial end chan0 Specifies the tty port that is connected to a null model cable for a serial heartbeat channel For example the serial_port c...

Page 178: ...ou specify is entered into the fields within the services section in the database The following details each cluster service field and its subsequent description start service0 name service_name disabled yes_or_no userScript path_name Specifies the name of the service whether the service should be disabled after it is created and the full path name of any script used to start and stop the service ...

Page 179: ...vice0 Specifies the owner of the device the group to which the device belongs and the access mode for the device B 5 Using Red Hat Cluster Manager with Piranha A cluster can be used in conjunction with the Piranha load balancing features to deploy a highly avail able e commerce site that has complete data integrity and application availability in addition to load balancing capabilities Figure B 1 ...

Page 180: ...addition if a Web server fails it can be removed from the set This Piranha configuration is particularly suitable if the Web servers serve only static Web content which consists of small amounts of infrequently changing data such as corporate logos that can be easily duplicated on the Web servers However this configuration is not suitable if the Web servers serve dynamic content which consists of ...

Page 181: ...ine merchandise ordering through a URL Client requests to the URL pass through the firewall to the active Piranha load bal ancing system which then forwards the requests to one of the three Web servers The Red Hat Cluster Manager systems serve dynamic data to the Web servers which forward the data to the requesting client system ...

Page 182: ...182 Appendix B Supplementary Software Information ...

Page 183: ... the cluster database 133 changing the cluster name 136 diagnosing and correcting problems in a cluster 137 disabling the cluster software 137 displaying cluster and service status 129 modifying cluster event logging 134 modifying the cluster configuration 133 reinitializing the cluster 136 reloading the cluster database 136 removing a cluster member 132 restoring the cluster database 133 starting...

Page 184: ...to start 81 modifying a service 80 MySQL service setting up 92 NFS caveats 111 NFS client access 108 NFS server requirements 103 NFS service configuration example 105 NFS service setting up 103 Oracle service setting up 83 Oracle tuning 91 relocating a service 80 Samba operating model 113 Samba server requirements 113 Samba service configuration example 117 Samba service configuration parameters 1...

Page 185: ...configuring 65 examples cluconfig 58 minimum cluster configuration 27 NFS service configuration 105 no single point of failure configuration 28 oracle script 84 Samba service configuration 117 sample script to start and stop the MySQL database 92 startdb script 84 startdbi script 88 stopdb script 86 stopdbi script 89 using cluadmin to a MySQL service 95 using cluadmin to add an Oracle service 89 F...

Page 186: ...ations 13 shared storage requirements 14 hardware information supplementary 151 hardware installation operating system configuration 13 hardware watchdog timer Configuring 158 hardware watchdog timers 158 hardware watchdog timers table 159 heartbeat 7 heartbeat channel status table 130 heartbeat channels configuring 40 heartbeat daemon 172 host bus adapter features and configuration requirements 1...

Page 187: ...configuration 109 caveats 111 client access 108 server requirements 103 service configuration example 105 service configuration parameters 104 setting up service 103 NMI watchdog timer enabling 156 no single point of failure configuration 28 Non Maskable Interrupt NMI watchdog timers 156 O operating system configuration hardware installation 13 Oracle adding an Oracle service 89 oracle script exam...

Page 188: ...ster Manager GUI 145 147 Java Runtime Environment JRE 145 service configuration screen 149 Splashscreen 147 Red Hat Linux installation and configuration 33 relocating a service 80 remote monitoring configuring 58 S Samba operating model 113 server requirements 113 service configuration example 117 service configuration parameters 115 setting up service 112 smb conf sharename file fields 119 Window...

Page 189: ...installing the basic system hardware 31 member status 129 minimum cluster configuration components 27 network hardware 24 no single point of failure configuration 28 point to point Ethernet heartbeat channel hardware 25 point to point serial heartbeat channel hardware 26 power switch hardware 20 power switch status 130 QLA2200 features and configuration requirements 168 RPS 10 power switch 151 ser...

Page 190: ...190 Index UPS systems configuring 42 W watchdog timers hardware configuring 158 hardware based 17 NMI enabling 156 setting up 155 software 156 configuration 156 software based 17 ...