Lab 7 Configuring Heartbeat and Ldirector

Configuring Heartbeat



1. Configuring ha.cf
   There are three files you will need to configure before starting up heartbeat. The first one is ha.cf. This is placed in the /etc/ha.d directory that is created after installation. It tells heartbeat what types of media paths to use and how to configure them. The ha.cf in the source directory contains all the various options you can use, as described below. For convenience of discussion, suppose your domain name is cula.net.
   serial /dev/ttyS0

   Use a serial heartbeat - if you don't use a serial heartbeat, you must use another medium, such as a bcast (ethernet) heartbeat. Replace /dev/ttyS0 with the appropriate device file for your required serial heartbeat.
   watchdog /dev/watchdog

   Optional. The watchdog function provides a way to have a system that is still minimally functioning, but not providing a heartbeat, reboot itself after a minute of being sick. This could help avoid a scenario where the machine recovers its heartbeat after being pronounced dead. If that happened and a disk mount failed over, you could have two nodes mounting a disk simultaneously. If you wish to use this feature, then in addition to this line, you will need to load the "softdog" kernel module and create the actual device file. To do this, first type insmod softdog to load the module. Then, type grep misc /proc/devices and note the number it reports (should be 10). Next, type cat /proc/misc | grep watchdog and note that number (should be 130). Now you can create the device file with that info typing, mknod /dev/watchdog c 10 130. Commnet this out in your experiment.
   bcast eth0

   Specifies to use a broadcast heartbeat over the eth0 interface.
   keepalive 2

   Sets the time between heartbeats to 2 seconds.
   warntime 10

   Time in seconds before issuing a late heartbeat warning in the logs.
   deadtime 30

   Node is pronounced dead after 30 seconds.
   initdead 120

   With some configurations, the network takes some time to start working after a reboot. This is a separate "deadtime" to handle that case. It should be at least twice the normal deadtime.
   hopfudge 1

   Optional. For ring topologies, number of hops allowed in addition to the number of nodes in the cluster.
   baud 19200

   Speed at which to run the serial line (bps).
   udpport 694

   Use port number 694 for udp. This is the default, and the official registered port number.
   udp eth0

   The interfaces to heartbeat over.
   nice_failback on

   Optional. For those familiar with Tru64 Unix, heartbeat acts as if in "favored member" mode. The master holds all the resources until a failover, at which time the slave takes over. Once the master comes back online, it will take everything back from the slave. This option will prevent the master node from re-acquiring cluster resources after a failover.
   node director_host_name1.cula.net

   Mandatory. Hostname of machine in cluster as described by uname -n.
   node director_host_name2.cula.net

   Mandatory. Hostname of machine in cluster as described by uname -n.
   debugfile /var/log/ha-debug

   File to wirte debug messages to.
   logfile /var/log/ha-log

   File to write other messages to.
   The following is a sample ha.cf file.
   

   debugfile /var/log/ha-debug logfile /var/log/ha-log logfacility local0 keepalive 2 deadtime 10 serial /dev/ttyS0 baud 19200 udpport 694 udp eth0 nice_failback on node lvs1.cula.net node lvs2.cula.net

   
   
2. Configuring haresources
   Once you've got your ha.cf set up, you need to configure haresources. This file specifies the services for the cluster and who the default owner is. Note that this file must be the same at both nodes! You need the following line in your haresources file to heartbeat the specified services, using the VIP 192.168.1.176 and primary node name lvs1.cula.net as an example,
   lvs1.cula.net 64.62.163.176/27/eth0 ldirectord
   This line tells heartbeat to start the Linux Director Daemon ldirectord, which in turn will read the file /etc/ha.d/conf/ldirectord.cf to start and monitor the Linux Virtual Server cluster.
   
   
3. Configuring Authkeys
   The third file to configure determines your authentication keys. There are three types of authentication methods available: crc, md5, and sha1. To make things simple, we shall use crc. In your authkeys file, add the entry: auth 1
   1 crc Comment out other authentication methods. This file must have mode 600.
   
   

   Configuring Ldirctord

   
   
4. Configuring ldirectord.cf. After you have configured heartbeat, you have to configure ldirectord which controls the Linux Virtual Server. The only file you need to create and edit is ldirectord.cf. which is in the directory /etc/ha.d/conf. It contains information about its use. You may set the entries as follows, using the virtual IP 192.168.1.176, real IPs 192.168.1.180, and 192.168.1.181, real server names, custer1.cula.net, and custer2.cula.net and services http and https as examples.

   # Global Directives
   checktimeout=3
   checkinterval=1
   fallback=127.0.0.1:80
   #remove down server from cluster
   quiescent=no
   
   
   # A sample virtual with a fallback that will override the goble setting
   # the real must start at least 4 spaces from left margin
   #your "index.html" file must contain the string "happy"
   virtual=192.168.1.176:443
   	real=192.168.1.180:443 ipip 6
   	real=192.168.1.181:443 ipip 6
    	fallback=127.0.0.1:443
           service=https
   	request="index.html"
           receive="happy"
           scheduler=wlc
           persistent=120
           netmask=255.255.255.224
           protocol=tcp
   
   virtual=192.168.1.176:80
           real=192.168.1.180:80 ipip 6
           real=192.168.1.181:80 ipip 6
           fallback=127.0.0.1:80
           service=http
           request="index.html"
           receive="happy"
           scheduler=wlc
           persistent=120
           netmask=255.255.255.0
           protocol=tcp
   
   

5. You can setup your real servers ( custer1.cula.net and custer2.cula.net here ) with the following scripts:

   #!/bin/sh
   echo 1 > /proc/sys/net/ipv4/ip_forward
   /sbin/modprobe ipip
   /sbin/ifconfig tunl0 0.0.0.0 up
   echo 1 > /proc/sys/net/ipv4/conf/all/hidden
   echo 1 > /proc/sys/net/ipv4/conf/tunl0/hidden
   /sbin/ifconfig tunl0 192.168.1.176 netmask 255.255.255.255 broadcast 192.168.1.176 up
   
   

6. You may now start, stop or check the status of heartbeat by the commands /etc/ha.d/heartbeat start
   /etc/ha.d/heartbeat status
   /etc/ha.d/heartbeat stop
   While starting or stopping heartbeat, you may watch the log or debug files from another window by tail -f /var/log/ha-debug
   tail -f /var/log/ha-log
7. After you have started heartbeat, check its status to make sure that it is running.
8. Execute /sbin/ipvsadm to check if the cluster has been formed. Note that the LVS should have been started by heartbeat ( which calls ldirectord to do the work ). If yes, that means you have done everything successfully and you should see something like

   IP Virtual Server version 1.0.7 (size=4096)
   Prot LocalAddress:Port Scheduler Flags
     -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
   TCP  lvs1.cula.net:https wlc persistent 120 mask 255.255.255.224
     -> custer1.cula.net:https         Tunnel  6      0         0       
     -> custer2.cula.net:https         Tunnel  6      0         0       
   TCP  lvs1.cula.net:http wlc persistent 120 mask 255.255.255.224
     -> custer1.cula.net:http          Tunnel  6      0         0       
     -> custer2.cula.net:http          Tunnel  6      0         0       
   
   

9. If you do not see this, there may be errors in your scripts. Check also the log file /var/log/ldirectord.log. If it says something like "return code 1" or "return code 2", its likely that you have syntax errors in the file ldirectord.cf. Make sure that the definition of real= in the file must be at least four spaces from the left margin and your comments '#' should start at the left margin ( it would be real nice if the config file could be written in xml format but as it stands, you need to be careful about the white spaces in the file ).
10. Start heartbeat in the backup node. Execute ipvsadm. You should not see that cluster. Now shutdown or stop the heartbeat of the primary machine. Wait one minute. Check the backup machine with ipvsadm. If things have been setup properly, the cluster should have shifted to the backup machine.
    
11. Turn off a real machine ( or shutdown its http server ) of the cluster and check at the director with ipvsadm. You should see that the machine has been deleted from the cluster. Now turn the machine ( or http server ) on and check again. You should see that it joins the cluster again.