Red Hat GLOBAL FILE SYSTEM 4.5, Manual

Page 1: ...Global File System 4 5 Red Hat Global File System ISBN N A Publication date ...

Page 2: ...This book provides information about installing configuring and maintaining Red Hat GFS Red Hat Global File System Global File System ...

Page 3: ...bited without the explicit permission of the copyright holder Distribution of the work or derivative of the work in any standard paper book form for commercial purposes is prohibited unless prior permission is obtained from the copyright holder Red Hat and the Red Hat Shadow Man logo are registered trademarks of Red Hat Inc in the United States and other countries All other trademarks referenced h...

Page 4: ...Global File System ...

Page 5: ...age Network 10 5 Fibre Channel Storage Devices 10 6 Network Power Switches 11 7 Console Access 11 8 Installing GFS 11 3 Getting Started 13 1 Prerequisite Tasks 13 2 Initial Setup Tasks 13 4 Managing GFS 15 1 Making a File System 15 2 Mounting a File System 18 3 Unmounting a File System 20 4 GFS Quota Management 21 4 1 Setting Quotas 21 4 2 Displaying Quota Limits and Usage 23 4 3 Synchronizing Quo...

Page 6: ...antum 36 10 Suspending Activity on a File System 37 11 Displaying Extended GFS Information and Statistics 38 12 Repairing a File System 39 13 Context Dependent Path Names 41 A Upgrading GFS 45 Index 49 Global File System vi ...

Page 7: ...se Linux refer to the following resources Red Hat Enterprise Linux Installation Guide Provides information regarding installation of Red Hat Enterprise Linux Red Hat Enterprise Linux Introduction to System Administration Provides introductory information for new Red Hat Enterprise Linux system administrators Red Hat Enterprise Linux System Administration Guide Provides more detailed information ab...

Page 8: ... other Red Hat documents are available in HTML and PDF versions online at the following location http www redhat com docs 3 Document Conventions Certain words in this manual are represented in different fonts styles and weights This highlighting indicates that the word is part of a specific category The categories include the following Courier font Courier font represents commands file names and p...

Page 9: ...u need to understand the behavior of the system Tip A tip is typically an alternative way of performing a task Important Important information is necessary but possibly unexpected such as a configuration change that will not persist after a reboot Caution A caution indicates an act that would violate your support agreement such as recompiling the kernel Warning A warning indicates potential data l...

Page 10: ... 2002 Provides detailed technical information on cluster file system and cluster volume manager design Storage Area Networks SANs Designing Storage Area Networks A Practical Reference for Implementing Fibre Channel and IP SANs Second Edition by Tom Clark Addison Wesley 2003 Provides a concise summary of Fibre Channel and IP SAN Technology Building SANs with Brocade Fabric Switches by C Beauchamp J...

Page 11: ...Table 1 Recommended References Table Recommended References xi ...

Page 12: ...xii ...

Page 13: ...iated information as background to help you understand GFS It contains the following sections Section 1 New and Changed Features Section 2 Performance Scalability and Economy Section 3 GFS Functions Section 4 GFS Software Subsystems Section 5 Before Setting Up GFS 1 New and Changed Features This section lists new and changed features included with the initial release of Red Hat Enterprise Linux 4 ...

Page 14: ... 2 2 Performance Scalability Moderate Price Section 2 3 Economy and Performance Note The deployment examples in this chapter reflect basic configurations your needs might require a combination of configurations shown in the examples 2 1 Superior Performance and Scalability You can obtain the highest shared file performance when applications access storage directly The GFS SAN configuration in Figu...

Page 15: ...e perspective of a client application storage is accessed as if it were directly attached to the server in which the application is running Stored data is actually on the SAN Storage devices and data can be equally shared by network client applications File locking and sharing functions are handled by GFS for each network client Note Clients implementing ext2 and ext3 file systems can be configure...

Page 16: ...torage shows how Linux client applications can take advantage of an existing Ethernet topology to gain shared access to all block storage devices Client data files and file systems can be shared with GFS on each client Application failover can be fully automated with Red Hat Cluster Suite Chapter 1 GFS Overview 4 ...

Page 17: ... Suite Volume management is managed through CLVM Cluster Logical Volume Manager For information about Red Hat Cluster Suite refer to Configuring and Managing a Red Hat Cluster For information about using CLVM refer to LVM Administrator s Guide Note CLVM is a cluster wide implementation of LVM enabled by the CLVM daemon clvmd running in a Red Hat Cluster Suite cluster The daemon makes it possible t...

Page 18: ...ftware subsystems and their components Software Subsystem Components Description GFS gfs ko Kernel module that implements the GFS file system and is loaded on GFS cluster nodes gfs_fsck Command that repairs an unmounted GFS file system gfs_grow Command that grows a mounted GFS file system gfs_jadd Command that adds journals to a mounted GFS file system gfs_mkfs Command that creates a GFS file syst...

Page 19: ...d as a local file system only It plugs into the lock harness lock_harness ko and provides local locking Table 1 1 GFS Software Subsystem Components 5 Before Setting Up GFS Before you install and set up GFS note the following key characteristics of your GFS file systems Number of file systems Determine how many GFS file systems to create initially More file systems can be added later File system na...

Page 20: ...des are needed Note the hostname and IP address of each GNBD server node for setting up GNBD clients later For information on using GNBD with GFS see the Using GNBD with Global File System document Storage devices and partitions Determine the storage devices and partitions to be used for creating logical volumes via CLVM in the file systems Chapter 1 GFS Overview 8 ...

Page 21: ...nts shows the platform requirements for GFS Operating System Hardware Architecture RAM Red Hat Enterprise Linux AS ES or WS Version 4 or later ia64 x86 64 x86 SMP supported 256 MB minimum Table 2 1 Platform Requirements 2 Red Hat Cluster Suite Red Hat GFS runs with Red Hat Cluster Suite 4 0 or later The Red Hat Cluster Suite software must be installed on the cluster nodes before you can install an...

Page 22: ...nts Note When a small number of nodes is used it may be possible to connect the nodes directly to ports on the storage device Note FC drivers may not work reliably with FC hubs Table 2 2 Fibre Channel Network Requirements 5 Fibre Channel Storage Devices Table 2 3 Fibre Channel Storage Device Requirements shows requirements for Fibre Channel devices that are to be connected to a GFS cluster Require...

Page 23: ...Make sure that you have console access to each GFS node Console access to each node ensures that you can monitor nodes and troubleshoot problems 8 Installing GFS Installing GFS consists of installing Red Hat GFS RPMs on nodes in a Red Hat cluster Before installing the RPMs make sure of the following The cluster nodes meet the system requirements described in this chapter You have noted the key cha...

Page 24: ...12 ...

Page 25: ...Red Hat Enterprise Linux distribution In addition if you are using GNBD multipath make sure that you understand GNBD multipath considerations For information on GNBD multipath see the document Using GNBD with Global Filesystem Note The system clocks in GFS nodes must be within a few minutes of each other to prevent unnecessary inode time stamp updating Unnecessary inode time stamp updating severel...

Page 26: ...als BlockDevice 3 At each node mount the GFS file systems For more information about mounting a GFS file system refer to Section 2 Mounting a File System Command usage mount t gfs BlockDevice MountPoint mount t gfs o acl BlockDevice MountPoint The o aclmount option allows manipulating file ACLs If a file system is mounted without the o acl mount option users are allowed to view ACLs with getfacl b...

Page 27: ...ivity on a File System Section 11 Displaying Extended GFS Information and Statistics Section 12 Repairing a File System Section 13 Context Dependent Path Names 1 Making a File System Once a cluster is set up and running you can create a GFS file system with the gfs_mkfs command A file system is created on an activated CLVM volume The following information is required to run the gfs_mkfs command Lo...

Page 28: ...luster Number Specifies the number of journals to be created by the gfs_mkfs command One journal is required for each node that mounts the file system More journals than are needed can be specified at creation time to allow for future expansion BlockDevice Specifies a volume Examples In this example lock_dlm is the locking protocol that the file system uses The cluster name is alpha and the file s...

Page 29: ...se Recognized cluster locking protocols include lock_dlm The standard locking module lock_gulm The locking module compatible with earlier versions of GFS lock_nolock May be used when GFS is acting as a local file system one node only O Prevents the gfs_mkfs command from asking for confirmation before writing the file system q Quiet Do not display anything r MegaBytes Specifies the size of the reso...

Page 30: ... Managing a Red Hat Cluster After those requirements have been met you can mount the GFS file system as you would any Linux file system To manipulate file ACLs you must mount the file system with the o acl mount option If a file system is mounted without the o acl mount option users are allowed to view ACLs with getfacl but are not allowed to set them with setfacl Usage Mounting Without ACL Manipu...

Page 31: ...pecific o option values that can be passed to GFS at mount time Option Description acl Allows manipulating file ACLs If a file system is mounted without the acl mount option users are allowed to view ACLs with getfacl but are not allowed to set them with setfacl hostdata HostIDInfo This field provides host the computer on which the file system is being mounted identity information to the lock modu...

Page 32: ... table to use with the file system oopses_ok This option allows a GFS node to not panic when an oops occurs By default a GFS node panics when an oops occurs causing the file system used by that node to stall for other GFS nodes A GFS node not panicking when an oops occurs minimizes the failure on other GFS nodes using the file system that the failed node is using There may be circumstances where y...

Page 33: ...ting can allow users or groups to slightly exceed the set limit To minimize this GFS dynamically reduces the synchronization period as a hard quota limit is approached GFS uses its gfs_quota command to manage quotas Other Linux quota facilities cannot be used with GFS 4 1 Setting Quotas Two quota settings are available for each user ID UID or group ID GID a hard limit and a warn limit A hard limit...

Page 34: ...be either a group name from the group file or the GID number Size Specifies the new value to limit or warn By default the value is in units of megabytes The additional k s and b flags change the units to kilobytes sectors and file system blocks respectively MountPoint Specifies the GFS file system to which the actions apply Examples This example sets the hard limit for user Bert to 1024 megabytes ...

Page 35: ...splaying Quota Limits for a Group gfs_quota get g Group f MountPoint Displaying Entire Quota File gfs_quota list f MountPoint User A user ID to display information about a specific user It can be either a user name from the password file or the UID number Group A group ID to display information about a specific group It can be either a group name from the group file or the GID number MountPoint Sp...

Page 36: ...GIDs into names if the n option is added to the command line Space allocated to GFS s hidden files can be left out of displayed values for the root UID and GID by adding the d option to the command line This is useful when trying to match the numbers from gfs_quota with the results of a du command Examples This example displays quota information for all users and groups that have a limit set or ar...

Page 37: ... the gfs_quota sync command to synchronize the quota information from a node to the on disk quota file between the automatic updates performed by GFS Usage Synchronizing Quota Information gfs_quota sync f MountPoint MountPoint Specifies the GFS file system to which the actions apply Tuning the Time Between Synchronizations gfs_tool settune MountPoint quota_quantum Seconds MountPoint Specifies the ...

Page 38: ...fault Disabling is not persistent across unmounts Usage gfs_tool settune MountPoint quota_enforce 0 1 MountPoint Specifies the GFS file system to which the actions apply quota_enforce 0 1 0 disabled 1 enabled Comments A value of 0 disables enforcement Enforcement can be enabled by running the command with a value of 1 instead of 0 as the final command line parameter Even when GFS is not enforcing ...

Page 39: ...ter to 1 Usage fs_tool settune MountPoint quota_account 0 1 MountPoint Specifies the GFS file system to which the actions apply quota_account 0 1 0 disabled 1 enabled Comments To enable quota accounting on a file system the quota_account parameter must be set back to 1 Afterward the GFS quota file must be initialized to account for all current disk usage for users and groups on the file system The...

Page 40: ...has been added The gfs_grow command must be run on a mounted file system but only needs to be run on one node in a cluster All the other nodes sense that the expansion has occurred and automatically start using the new space To verify that the changes were successful use the gfs_grow command with the T test and v verbose flags Running the command with those flags displays the current state of the ...

Page 41: ...tem is mounted Device Specifies the device node of the file system Table 4 3 GFS specific Options Available While Expanding A File System describes the GFS specific options that can be used while expanding a GFS file system Option Description h Help Displays a short usage message q Quiet Turns down the verbosity level T Test Do all calculations but do not write any data to the disk and do not expa...

Page 42: ...d with those flags displays the current state of the mounted GFS file system Usage gfs_jadd j Number MountPoint Number Specifies the number of new journals to be added MountPoint Specifies the directory where the GFS file system is mounted Comments Before running the gfs_jadd command Back up important data on the file system Run a gfs_tool df MountPoint command to display the volume used by the fi...

Page 43: ...lp Displays short usage message J MegaBytes Specifies the size of the new journals in megabytes Default journal size is 128 megabytes The minimum size is 32 megabytes To add journals of different sizes to the file system the gfs_jadd command must be run for each size journal The size specified is rounded down so that it is a multiple of the journal segment size that was specified when the file sys...

Page 44: ...is used regardless of how the file is opened When a file is opened with O_DIRECT or when a GFS direct I O attribute is attached to a file all I O operations must be done in block size multiples of 512 bytes The memory being read from or written to must also be 512 byte aligned One of the following methods can be used to enable direct I O on a file O_DIRECT GFS file attribute GFS directory attribut...

Page 45: ...can be used to assign set a direct I O attribute flag inherit_directio to a GFS directory Enabling the inherit_directio flag on a directory causes all newly created regular files in that directory to automatically inherit the directio flag Also the inherit_directio flag is inherited by any new subdirectories created in the directory The inherit_directio flag can also be cleared Usage Setting the i...

Page 46: ...n to the journal which can be substantially faster than the time it takes to write the file data to the main file system Applications that rely on fsync to sync file data may see improved performance by using data journaling Data journaling can be enabled automatically for any GFS files created in a flagged directory and all its subdirectories Existing files with zero length can also have data jou...

Page 47: ... atime Updates Each file inode and directory inode has three time stamps associated with it ctime The last time the inode status was changed mtime The last time the file or directory data was modified atime The last time the file or directory data was accessed If atime updates are enabled as they are by default on GFS and other Linux file systems then every time a file is read its inode needs to b...

Page 48: ...ted using the gfs_tool command Each GFS node updates the access time based on the difference between its system time and the time recorded in the inode It is required that system clocks of all GFS nodes in a cluster be synchronized If a node s system time is out of synchronization by a significant fraction of the tunable parameter atime_quantum then atime updates are written more frequently Increa...

Page 49: ... on the mount point gfs1 are displayed gfs_tool gettune gfs1 In this example the atime update period is set to once a day 86 400 seconds for the GFS file system on mount point gfs1 gfs_tool settune gfs1 atime_quantum 86400 10 Suspending Activity on a File System You can suspend write activity to a file system by using the gfs_tool freeze command Suspending write activity allows hardware based devi...

Page 50: ...Information and Statistics You can use the gfs_tool command to gather a variety of details about GFS This section describes typical use of the gfs_tool command for displaying statistics space usage and extended status Usage Displaying Statistics gfs_tool counters MountPoint The counters flag displays statistics about a file system If c is used the gfs_tool command continues to run displaying stati...

Page 51: ...fs_tool man page Examples This example reports extended file system usage about file system gfs gfs_tool df gfs This example reports extended file status about file gfs datafile gfs_tool stat gfs datafile 12 Repairing a File System When nodes fail with the file system mounted file system journaling allows fast recovery However if a storage device loses power or is physically disconnected file syst...

Page 52: ...ers no to any queries automatically The option provides a way of trying the command to reveal errors without actually allowing the gfs_fsck command to take effect Refer to the gfs_fsck man page gfs_fsck 8 for additional information about other command options Usage gfs_fsck y BlockDevice y The y flag causes all questions to be answered with yes With the y flag specified the gfs_fsck command does n...

Page 53: ...es a name to represent the real file or directory on the other end of the link For a Variable Symbolic Link ln s VariableLinkName Variable Specifies a special reserved name from a list of values refer to Table 4 5 CDPN Variable Values to represent one of multiple existing files or directories This string is not the name of an actual file or directory itself The real files or directories must be cr...

Page 54: ...ng produced by the output of the following command echo id g Table 4 5 CDPN Variable Values Example In this example there are three nodes with hostnames n01 n02 and n03 Applications on each node uses directory gfs log but the administrator wants these directories to be separate for each node To do this no actual log directory is created instead an hostname CDPN link is created with the name log In...

Page 55: ...fileC Example 43 ...

Page 56: ...44 ...

Page 57: ... lock management to DLM lock management during the upgrade to Red Hat GFS 6 1 However after the upgrade to GFS 6 1 you can change lock managers The following procedure demonstrates upgrading to Red Hat GFS 6 1 from a GFS 6 0 or GFS 5 2 1 configuration with an example pool configuration for a pool volume named argus poolname argus subpools 1 subpool 0 512 1 gfs_data pooldevice 0 0 dev sda1 1 Halt t...

Page 58: ...conf 4 At all GFS 6 1 nodes start ccsd run the lock_gulmd c command and start clvmd as shown in the following example ccsd lock_gulmd c Warning You didn t specify a cluster name before use_ccs Letting ccsd choose which cluster we belong to clvmd Note Ignore the warning message following the lock_gulmd c command Because the cluster name is already included in the converted configuration file there ...

Page 59: ...grade option as shown in the following example mount t gfs o upgrade dev pool argus mnt gfs1 Note This step only needs to be done once on the first mount of the GFS file system Note If static minor numbers were used on pool volumes and the GFS 6 1 nodes are using LVM2 for other purposes root file system there may be problems activating the pool volumes under GFS 6 1 That is because of static minor...

Page 60: ...48 ...

Page 61: ... tuning atime quantum 36 context dependent path names CDPNs 41 data journaling 34 direct I O 32 directory attribute 33 file attribute 32 O_DIRECT 32 growing 28 making 15 mounting 18 quota management 21 disabling enabling quota accounting 27 disabling enabling quota enforcement 26 displaying quota limits 23 setting quotas 21 synchronizing quotas 24 repairing 39 suspending activity 37 unmounting 20 ...

Page 62: ...accounting 27 disabling enabling quota enforcement 26 displaying quota limits 23 setting quotas 21 synchronizing quotas 24 R recommended references table x Red Hat Cluster Suite system requirements 9 references recommended x repairing a file system 39 S setup initial initial tasks 13 suspending activity on a file system 37 system requirements 9 console access 11 fencing 9 fibre channel storage dev...