NFS Server Requirements

In order to create highly available NFS services, there are a few requirements which must be met by each cluster server. (Note: these requirements do not pertain to NFS client systems.) These requirements are as follows:

• Kernel support for the NFS server must be enabled. NFS can be either configured statically or as a module. Both NFS V2 and NFS V3 are supported.

• The kernel support for NFS provided with Red Hat Enterprise Linux AS 2.1 incorporates enhancements (initially developed by Mission Critical Linux Inc.) which allow for transparent relocation of NFS services. These kernel enhancements prevent NFS clients from receiving Stale file handle errors after an NFS service has been relocated. If using kernel sources that do not include these NFS enhancements, then NFS can still be configured and run within the cluster; but warning messages will appear during service start and stop pointing out the absence of these kernel enhancements.

• The NFS daemons must be running on all cluster servers. This is accomplished by enabling the NFS init.d run level script. For example:

  /sbin/chkconfig --level 345 nfs on

• The RPC portmap daemon must be enabled. For example:

  /sbin/chkconfig --level 345 portmap on

  NFS services will not start unless the following NFS daemons are running: nfsd, rpc.mountd, and rpc.statd.

• Filesystem mounts and their associated exports for clustered NFS services should not be included in /etc/fstab or /etc/exports. Rather, for clustered NFS services, the parameters describing mounts and exports are entered via the cluadmin configuration utility.

Gathering NFS Service Configuration Parameters

In preparation of configuring NFS services, it is important to plan how the filesystems will be exported and failed over. The following information is required in order to configure NFS services:

• Service Name — A name used to uniquely identify this service within the cluster.

• Preferred Member — Defines which system will be the NFS server for this service if more than one cluster member is operational.

• Relocation Policy — whether to relocate the service to the preferred member if the preferred member wasn't running at the time the service was initially started. This parameter is useful as a means of load balancing the cluster members as NFS servers by assigning half the load to each.

• IP Address — NFS clients access filesystems from an NFS server which is designated by its IP Address (or associated hostname). In order to abstract NFS clients from knowing which specific cluster member is the acting NFS server, the client systems should not use the cluster member's hostname as the IP address by which a service is mounted. Rather, clustered NFS services are assigned floating IP addresses which are distinct from the cluster server's IP addresses. This floating IP address is then configured on whichever cluster member is actively serving the NFS export. Following this approach, the NFS clients are only aware of the floating IP address and are unaware of the fact that clustered NFS server has been deployed. When entering an NFS service's IP address, an administrator will also be prompted to enter an associated netmask and broadcast address. If None (which is the default) is selected, then the assigned netmask and broadcast will be the same as what the network interface is currently configured to.

• Mount Information — for non-clustered filesystems, the mount information is typically placed in /etc/fstab. By contrast, clustered filesystems must not be placed in /etc/fstab. This is necessary to ensure that only one cluster member at a time has the filesystem mounted. Failure to do so will result in filesystem corruption and likely system crashes.

  • Device special file — The mount information designates the disk's device special file and the directory on which the filesystem will be mounted. In the process of configuring an NFS service, this information will be prompted for.

  • Mount point directory — An NFS service can include more than one filesystem mount. In this manner, the filesystems will be grouped together as a single failover unit.

  • Mount options — The mount information also designates the mount options. Note: by default, the Linux NFS server does not guarantee that all write operations are synchronously written to disk. In order to ensure synchronous writes, specify the sync mount option. Specifying the sync mount option favors data integrity at the expense of performance. Refer to mount(8) for detailed descriptions of the mount related parameters.

  • Forced unmount — As part of the mount information, there will be a prompt as to whether forced unmount should be enabled or not. When forced unmount is enabled, if any applications running on the cluster server have the designated filesystem mounted when the service is being disabled or relocated, then that application will be killed to allow the unmount to proceed.

• Export Information — for non-clustered NFS services, export information is typically placed in /etc/exports. In contrast, clustered NFS services should not place export information in /etc/exports; rather there will be a prompt for this information during service configuration. Export information includes:

  • Export directory — the export directory can be the same as the mount point specified with the mount information. In this case, the entire filesystem is accessible through NFS. Alternatively, a specified portion (subdirectory) of a mounted filesystem can be mounted instead of the entire filesystem. By exporting subdirectories of a mountpoint, different access rights can be allocated to different sets of NFS clients.

  • Export client names — this parameter defines which systems will be allowed to access the file system as NFS clients. Under this method, individual systems can be designated (e.g. fred), as well as wildcards to allow groups of systems (e.g. *.wizzbang.com). Entering a client name of * allows any client to mount the filesystem.

  • Export client options — this parameter defines the access rights afforded to the corresponding client(s). Examples include ro (read only), and rw (read write). Unless explicitly specified otherwise, the default export options are ro,async,wdelay,root_squash.

  Refer to exports(5) for detailed descriptions of the export parameter syntax.

When running the cluadmin utility to configure NFS services:

• Take extra care to correctly enter the service parameters. The validation logic associated with NFS parameters is currently insufficient.

• In response to most of the prompts, you can enter the [?] character to obtain descriptive help text.

Example NFS Service Configuration

In order to illustrate the configuration process for an NFS service, an example configuration is described in this section. This example consists of setting up a single NFS export which houses the home directories of 4 members of an accounting department. NFS client access will be restricted to these four user's systems.

The following are the service configuration parameters which will be used as well as some descriptive commentary.

	Note
	Prior to configuring an NFS service using cluadmin, it is required that the cluster daemons are running.

• Service Name — nfs_accounting. This name was chosen as a reminder of the service's intended function to provide exports to the members of the accounting department.

• Preferred Member — clu4. In this example cluster, the member names are clu3 and clu4.

• User Script — The cluster infrastructure includes support for NFS services. Consequently, there is no need to create a User Script when configuring an NFS service. For this reason, when prompted to specify a User Script, the default value of None should be selected.

• IP Address — 10.0.0.10. There is a corresponding hostname of clunfsacct associated with this IP address, by which NFS clients mount the filesystem. Note that this IP address is distinct from that of both cluster members (clu3 and clu4). The default netmask and broadcast address will be used.

• Mount Information — /dev/sdb10, which refers to the partition on the shared storage RAID box on which the file system will be physically stored. ext3 — referring to the file system type which was specified when the file system was created. /mnt/users/accounting — specifies the file system mount point. rw,nosuid,sync — are the mount options.

• Export Information - for this example, the entire mounted file system will be made accessible on a read write basis by four members of the accounting department. The names of the systems used by these four members are burke, stevens, needle and dwalsh.

The following is an excerpt of the /etc/hosts file used to represent IP addresses and associated hostnames used within the cluster:

10.0.0.3        clu3          # cluster member

10.0.0.4        clu4          # second cluster member

10.0.0.10       clunfsacct    # floating IP address associated with accounting dept. NFS service

10.0.0.11       clunfseng     # floating IP address associated with engineering dept. NFS service

The following is excerpted from running cluadmin to configure this example NFS service:

cluadmin> service add

Service name: nfs_accounting
Preferred member [None]: clu4
Relocate when the preferred member joins the cluster (yes/no/?) \
	    [no]: yes
Status check interval [0]: 30
User script (e.g., /usr/foo/script or None) [None]: 
Do you want to add an IP address to the service (yes/no/?) [no]: yes

        IP Address Information

IP address: 10.0.0.10
Netmask (e.g. 255.255.255.0 or None) [None]:    
Broadcast (e.g. X.Y.Z.255 or None) [None]: 
Do you want to (a)dd, (m)odify, (d)elete or (s)how an IP address, or 
are you (f)inished adding IP addresses [f]: f
Do you want to add a disk device to the service (yes/no/?) [no]: yes

Disk Device Information

Device special file (e.g., /dev/sdb4): /dev/sdb10
Filesystem type (e.g., ext2, ext3 or None): ext3
Mount point (e.g., /usr/mnt/service1) [None]: /mnt/users/accounting
Mount options (e.g., rw,nosuid,sync): rw,nosuid,sync
Forced unmount support (yes/no/?) [yes]: 
Would you like to allow NFS access to this filesystem (yes/no/?)  [no]: yes

You will now be prompted for the NFS export configuration: 

Export directory name: /mnt/users/accounting

Authorized NFS clients

Export client name [*]: burke
Export client options [None]: rw
Do you want to (a)dd, (m)odify, (d)elete or (s)how NFS CLIENTS, or 
are you (f)inished adding CLIENTS [f]: a

Export client name [*]: stevens
Export client options [None]: rw
Do you want to (a)dd, (m)odify, (d)elete or (s)how NFS CLIENTS, or 
are you (f)inished adding CLIENTS [f]: a

Export client name [*]: needle
Export client options [None]: rw
Do you want to (a)dd, (m)odify, (d)elete or (s)how NFS CLIENTS, or 
are you (f)inished adding CLIENTS [f]: a

Export client name [*]: dwalsh
Export client options [None]: rw
Do you want to (a)dd, (m)odify, (d)elete or (s)how NFS CLIENTS, or 
are you (f)inished adding CLIENTS [f]: f
Do you want to (a)dd, (m)odify, (d)elete or (s)how NFS EXPORTS, or 
are you (f)inished adding EXPORTS [f]: 
Do you want to (a)dd, (m)odify, (d)elete or (s)how DEVICES, 
or are you (f)inished adding DEVICES [f]: 
Disable service (yes/no/?) [no]: 
name: nfs_eng
disabled: no
preferred node: clu4
relocate: yes
user script: None
monitor interval: 30
IP address 0: 10.0.0.10
  netmask 0: None
  broadcast 0: None
device 0: /dev/sdb10
  mount point, device 0: /mnt/users/accounting
  mount fstype, device 0: ext3
  mount options, device 0: rw,nosuid,sync
  force unmount, device 0: yes
NFS export 0: /mnt/users/accounting
  Client 0: burke, rw
  Client 1: stevens, rw
  Client 2: needle, rw
  Client 3: dwalsh, rw
Add nfs_eng service as shown? (yes/no/?) yes
Added nfs_eng.
cluadmin>

NFS Client Access

The NFS usage model for clients is completely unchanged from its normal approach. Following the prior example, if a client system wishes to mount the highly available NFS service, it simply needs to have an entry like the following in its /etc/fstab file:

clunfsacct:/mnt/users/accounting  /mnt/users/  nfs bg 0 0

Active-Active NFS Configuration

In the previous section, an example configuration of a simple NFS service was discussed. This section describes how to setup a more complex NFS service.

The example in this section involves configuring a pair of highly available NFS services. In this example, suppose two separate teams of users will be accessing NFS filesystems served by the cluster. To serve these users, two separate NFS services will be configured. Each service will have its own separate IP address and be preferred to distinct cluster members. In this manner, under normal operating circumstances, when both cluster members are running, each will be NFS exporting one of the filesystems. This enables an administrator to most effectively utilize the capacity of the two server systems. In the event of a failure (or planned maintenance) on either of the cluster members, both NFS services will be running on the active cluster member.

This example configuration will expand upon the NFS service created in the prior section by adding in a second service. The following service configuration parameters apply to this second service:

• Service Name — nfs_engineering. This name was chosen as a reminder of the service's intended function to provide NFS exports to the members of the engineering department.

• Preferred Member — clu3. In this example cluster, the member names are clu3 and clu4. Note that here clu3 is specified because the other cluster service (nfs_accounting) has clu4 specified as its preferred server.

• IP Address — 10.0.0.11. There is a corresponding hostname of clunfseng associated with this IP address, by which NFS clients mount the filesystem. Note that this IP address is distinct from that of both cluster members (clu3 and clu4). Also note that this IP address is different from the one associated with the other NFS service (nfs_accounting). The default netmask and broadcast address will be used.

• Mount Information — /dev/sdb11, which refers to the partition on the shared storage RAID box on which the filesystem will be physically stored. ext2 — referring to the filesystem type which was specified when the filesystem was created. /mnt/users/engineering — specifies the filesystem mount point. rw,nosuid,sync — are the mount options.

• Export Information — for this example, individual subdirectories of the mounted filesystem will be made accessible on a read-write (rw) basis by three members of the engineering department. The names of the systems used by these three team members are ferris, denham, and brown. To make this example more illustrative, notice that each team member will only be able to NFS mount their specific subdirectory.

The following is an example output from running cluadmin to create this second NFS service on the same cluster as used in the prior example when the service nfs_accounting was created.

cluadmin> service add

Service name: nfs_engineering
Preferred member [None]: clu3
Relocate when the preferred member joins the cluster (yes/no/?) [no]: yes
Status check interval [0]: 30
User script (e.g., /usr/foo/script or None) [None]: 
Do you want to add an IP address to the service (yes/no/?) [no]: yes

        IP Address Information

IP address: 10.0.0.11
Netmask (e.g. 255.255.255.0 or None) [None]: 
Broadcast (e.g. X.Y.Z.255 or None) [None]: 
Do you want to (a)dd, (m)odify, (d)elete or (s)how an IP address, or 
are you (f)inished adding IP addresses [f]: f
Do you want to add a disk device to the service (yes/no/?) [no]: yes

Disk Device Information

Device special file (e.g., /dev/sdb4): /dev/sdb11
Filesystem type (e.g., ext2, ext3 or None): ext2
Mount point (e.g., /usr/mnt/service1) [None]: /mnt/users/engineering
Mount options (e.g., rw,nosuid,sync): rw,nosuid,sync
Forced unmount support (yes/no/?) [yes]: 
Would you like to allow NFS access to this filesystem (yes/no/?) \
	    [no]: yes

You will now be prompted for the NFS export configuration: 

Export directory name: /mnt/users/engineering/ferris

Authorized NFS clients

Export client name [*]: ferris
Export client options [None]: rw
Do you want to (a)dd, (m)odify, (d)elete or (s)how NFS CLIENTS, or 
are you (f)inished adding CLIENTS [f]: f
Do you want to (a)dd, (m)odify, (d)elete or (s)how NFS EXPORTS, or 
are you (f)inished adding EXPORTS [f]: a

Export directory name: /mnt/users/engineering/denham

Authorized NFS clients

Export client name [*]: denham
Export client options [None]: rw
Do you want to (a)dd, (m)odify, (d)elete or (s)how NFS CLIENTS, or 
are you (f)inished adding CLIENTS [f]: 
Do you want to (a)dd, (m)odify, (d)elete or (s)how NFS EXPORTS, or 
are you (f)inished adding EXPORTS [f]: a

Export directory name: /mnt/users/engineering/brown

Authorized NFS clients

Export client name [*]: brown
Export client options [None]: rw
Do you want to (a)dd, (m)odify, (d)elete or (s)how NFS CLIENTS, or 
are you (f)inished adding CLIENTS [f]: f
Do you want to (a)dd, (m)odify, (d)elete or (s)how NFS EXPORTS, or 
are you (f)inished adding EXPORTS [f]: a
Do you want to (a)dd, (m)odify, (d)elete or (s)how DEVICES, or 
are you (f)inished adding DEVICES [f]: 
Disable service (yes/no/?) [no]: 
name: nfs_engineering
disabled: no
preferred node: clu3
relocate: yes
user script: None
monitor interval: 30
IP address 0: 10.0.0.11
  netmask 0: None
  broadcast 0: None
device 0: /dev/sdb11
  mount point, device 0: /mnt/users/engineering
  mount fstype, device 0: ext2
  mount options, device 0: rw,nosuid,sync
  force unmount, device 0: yes
NFS export 0: /mnt/users/engineering/ferris
  Client 0: ferris, rw
NFS export 0: /mnt/users/engineering/denham
  Client 0: denham, rw
NFS export 0: /mnt/users/engineering/brown
  Client 0: brown, rw
Add nfs_engineering service as shown? (yes/no/?) yes
Added nfs_engineering.
cluadmin>

NFS Caveats

The following points should be taken into consideration when clustered NFS services are configured.