How to re-enable a failed cluster service?

Sometimes a cluster service fails to restart because the service might have died unexpectedly or was killed without using the cluster tools. In such a case the service could no longer be re-enabled via clusvcadm -e <service_name>.

To recover from this issue you need to perform this procedure:

1. Stop rgmanager on all nodes with service rgmanager stop.
2. Start clumanager on all nodes with service rgmanager start.
3. Disable the service using clusvcadm -d <service_name>. The log messages that disabling the service failed may be ignored.
4. Enable the service using clusvcadm -e <service>.

Beware that the whole cluster services are halted and restarted. If multiple services are running, that could mean stopping all of them for the sake of one service that failed.

You may also try to trick the Cluster Suite into thinking that the service is still there:

1. Start httpd via /etc/init.d/httpd start.
2. Disable the service using clusvcadm -d web.
3. Enable the service using clusvcadm -e web.

This procedure is not guaranteed to work but has smaller impact on the other cluster services. It may be an effective workaround.

How to debug the script of a cluster service?

Sometimes the scripts that are shipped with RHEL do not comply to the Linux Standards Base defined here:

http://refspecs.freestandards.org/LSB_3.1.0/LSB-Core-generic/LSB-Core-generic/iniscrptact.html

Also if you want to write your own scripts you might have difficulties running them on your cluster. In such situations you might want to debug your script.

This can be done with multiple ways. Most obvious you might want to add log_level="7" to the <rm> tag in /etc/cluster/cluster.conf

As an alternative you can run manual tests:

# clusvcadm -d <servicename>
# rg_test test /etc/cluster/cluster.conf start service <servicename>
# rg_test test /etc/cluster/cluster.conf status service <servicename>
# rg_test test /etc/cluster/cluster.conf stop service <servicename>

How to prevent fenced cluster nodes before they boot up?

Sometimes cluster nodes don't have enough time to join the cluster and they are fenced by other cluster nodes. With the post join delay you can control how much time a cluster node will have to successfully connect to the cluster infrastructure. A good default are 30 seconds. If the process takes too long, the node gets fenced. You may adjust the value in the /etc/cluster/cluster.conf. Do not forget to rebuild the initrd afterwards.

How to tune GFS performance?

GFS works fastest with directories containing few, large files that are accessed mostly by single cluster nodes. The more data a cluster node reads and the fewer data it writes the better is the overall performance - especially if the node has independent access to the ressource. However GFS was invented for concurrent access and performance won't be bad if multiple cluster nodes share the same data!

Difficulties may arise if there are lots of small files accessed permanently by many cluster nodes. The concurrent locking race between nodes will drop the performance significantly.

To prevent this situation one should plan the filesystem layout with respect to these cluster filesystem specific barriers. The time invested in the tuning will pay off lateron. If you are in doubt how to analyze your filesystem layout and if you want to achieve maximum performance you should think about using ATIX professional services to inspect and tune your GFS infrastructure.

As GFS uses inter cluster communication for locking and filesystem coordination, GFS is not as fast as local filesystems. However you may tune the performance with the following options:

• On large file systems you should use -r 2048 on gfs_mkfs. The internal structure of GFS includes GFS resource groups (not related in any way to the resource groups of the rgmanager). On large filesystems you will have may resource groups that may slow down GFS unnecessarily. With the default settings, GFS will be created with 256 megabytes resource groups. With bigger resource group size you will need fewer resource group. As they are linked to each other you can significantly speed up filesystem operations like df or gfs_fsck.

• Allocate files. If you know that you will need to allocate space for some files you may decide to do so with "dd if=/dev/zero of=/mnt/gfs bs=1M count=123". So your application may use the preallocated files lateron. You simply move the required timeslot to the present.

• Seperate filesystems when you use huge quantities of files. The overhead used for huge quantities of files may be significant for the overall performance. If you choose to separate your filesystems you will have smaller filesystems with fewer files that are managed by GFS with better performance.

• Disable disk quotas if not needed.

  If you won't need disk quotas you should disable them. Thus GFS must not write quota information and will speed up. Mount with the "noquota" option.

• Use fast network hardware.

  Since the locking is coordinated via the cluster network you should speed up the overall performance with faster hardware. Also lookout if you have made any configuration mistakes that will result in a higher package count that is slowing down the network.

• Tune the number of GFS locks that are cached.

  GFS locks are cached in memory and you may balance out the right amount of memory used for this cache. If you use too much cache you will loose performance because of the management overhead. If you use too few cached locks the misses will result in lock recreation that will drop your performance. You should cache about the same number of locks as you plan to have used on your files regularily. You may change locks via:

  # echo "1234567" > /proc/cluster/lock_dlm/drop_count
  

• Increase "statfs_slots".

  With asynchronous inter cluster locking, GFS fills in stat data as the locks may be aquired. As a default 64 locks will be allocated for this task. With more slots you will have more cluster nodes that can lock files asynchronously at the cost of more network traffic. The following non persistent setting has to be entered to the gfs init script to reenable it after a reboot:

  # gfs_tool settune /mnt/gfs statfs_slots 128
  

• Adjust GFS daemon frequency.

  Various GFS daemons perform maintenance tasks in the background. You may affect performance by tweaking the running frequency.

  Daemon	Function	Frequency	Parameter
  gfs_glockd	reclaim unused glock structures	As needed	Unchangable
  gfs_inoded	reclaim unlinked inodes.	15 secs	inoded_secs
  gfs_logd	journal maintenance	1 sec	logd_secs
  gfs_quotad	write cached quota changes to disk	5 secs	quotad_secs
  gfs_scand	Look for cached glocks and inodes to toss from memory	5 secs	scand_secs
  gfs_recoverd	Recover dead machine's journals	60 secs	recoverd_secs

To change the frequency use and adjust the follwing command:

# gfs_tool settune /mnt/gfs inoded_secs 30

As these values are nonpersistent, you will have to enter them in the gfs init script after the file systems are mounted to enable them after a reboot.