Lab 4 Setting Up LVS/Tunnel Cluster

1. A networked machine uses ARP ( Address Resolution Protocol ) to resolve the hardware location/address of another machine on the same local network. The hardware address of a machine is the address of its network card ( ethernet card ) and is in general referred to as a MAC ( Medium Access Control ) address. Machines on the Internet are generally known by their names which resolve to IP addresses. This is how a machine on the cula.net network is able to communicate with another machine which is on the cula.net network. An IP address, though, cannot tell you the physical location of a machine. An IP address is a 32-bit number while a MAC address is a 48-bit number. In the Internet an IP address is bound to a MAC address. The ARP is used to translate an IP address to a MAC address so that the physical location of the machine can be located.
   
   ARP uses broadcasting mechanisms to locate a machine in a local network. When one wants to find another machine called x, it will yell, "Where are you, x?" All machines in the network listen and receive the message but only machine x will reply, "I am here." For instance, consider a very simple example. On a local network there are two machines called man and happiness with IP addresses 192.168.1.18 and 192.168.1.28 respectively. Now man wants to ping happiness to see that she is alive, but alas, man has no idea where happiness is. So when man decides to ping happiness he will need to send out an ARP request. This ARP request is akin to man shouting out on the network "happiness (192.168.1.28), where are you?" As a result of this every machine on the network will hear man shouting, but only happiness (192.168.1.28) will respond. happiness will then send an ARP reply directly back to man which is akin happiness saying, "man (192.168.1.18) I am here at 00:e0:18:6c:16:07 ( a 48-bit number expressed in hexadecimal )" After this simple transaction that's used to locate his friend on the network, man is able to communicate with happiness until he (his arp cache) forgets where happiness is (typically after 15 minutes on Unix).
   
   
2. Now let's see how this works. You can view your machines current arp/neighbor cache/table like so:
   
   [root@ns fore]# ip neigh show
   206.55.157.1 dev eth0 lladdr 00:d0:79:ac:dc:00 nud reachable
   As you can see the machine ns ( 206.55.157.2 ) knows where to find 206.55.157.1.
   
   What do you get when you execute the command ip neigh show on your machine? Also try arp -a which shows all the machines in the arp cache. What do you get?
   
   
3. Now let's add another machine to the arp cache.

   [root@ns fore]# ping -c 4 206.55.157.7
   PING 206.55.157.7 (206.55.157.7) from 206.55.157.2 : 56(84) bytes of data.
   64 bytes from 206.55.157.7: icmp_seq=1 ttl=255 time=0.476 ms
   64 bytes from 206.55.157.7: icmp_seq=2 ttl=255 time=0.456 ms
   64 bytes from 206.55.157.7: icmp_seq=3 ttl=255 time=0.442 ms
   64 bytes from 206.55.157.7: icmp_seq=4 ttl=255 time=0.436 ms
   
   --- 206.55.157.7 ping statistics ---
   4 packets transmitted, 4 received, 0% loss, time 2997ms
   rtt min/avg/max/mdev = 0.436/0.452/0.476/0.026 ms
   
   
   [root@ns fore]# ip neigh show
   206.55.157.1 dev eth0 lladdr 00:d0:79:ac:dc:00 nud reachable
   206.55.157.7 dev eth0 lladdr 00:e0:18:6c:16:07 nud reachable
   
   

   The hardware address/location ( MAC address ) of 206.55.157.7 has now been added to the arp/neighbor cache to ns which was trying to contact it. So until the entry for 206.55.157.7 times out (as a result of no communication between the two) ns knows where to find it and has no need to send an ARP request.
   Now let's delete 206.55.157.7 from our arp cache:

   [root@ns fore]# ip neigh delete 206.55.157.7 dev eth0
   [root@ns fore]# ip neigh show
   206.55.157.1 dev eth0 lladdr 00:d0:79:ac:dc:00 nud reachable
   206.55.157.7 dev eth0  nud failed
   
   

   Now ns has again forgotten where to find 206.55.157.7 and will need to send another ARP request the next time he needs to communicate with it. You may also see an output that an arp entry has been changed to the "stale" state. This means that the location shown is still valid, but it will have to be confirmed at the first transaction to that machine. Try the commands and record your results.
   
   What would happen if two machines in the network have the same IP 206.55.157.7? In this case, both of them may reply and this could be a mess and lead to undesired results. This is the arp problem that we need to deal with in LVS/Tunnel.
   
   ARP problem in LVS/Tunnel
   
4. With LVS-Tunnel, all the machines (director, realservers) in the LVS have an extra IP, the VIP. Here's a test setup where all machines and IPs are on the same physical network (i.e. are using the same link layer and can hear each other's broadcasts). ( Note : VIP is Virtual IP, RIP is Real server IP, DIP is Director IP. )

   ________ | | | client | |________| | | (router) | | | __________ | DIP | | |------| director | | VIP |__________| | | | ------------------------------------ | | | | | | RIP1, VIP RIP2, VIP RIP3, VIP ______________ ______________ ______________ | | | | | | | realserver1 | | realserver2 | | realserver3 | |______________| |______________| |______________|

   When the client requests a connection to the VIP, it must connect to the VIP on the director and not to the VIP on the realservers.

   The director box acts as an IP router, accepting packets destined for the VIP and then sending them on to a realserver (where the real work is done and a reply is generated). When the client (or router) puts out the arp request "who has VIP, tell client", the client/router must receive the MAC address of the director for the LVS to work. After receiving the arp reply, the client will send the connect request to the director. (The director will update its ipvsadm tables to keep track of the connections that it's in charge of and then forward the connect request packet to the chosen realserver).

   If instead, the client gets the MAC address of one of the realservers, then the packets will be sent directly to that realserver, bypassing the LVS action of the director and defeat the purpose of clustering. Getting the MAC address of the director (instead of the realservers) to the client when the client/router does an arp request is the key to solving the arp problem.
   We shall use a hidden interface approach to solve the arp problem.
   
   

5. Construct the tunnel interface. Execute the commands 1-3 on a real server, not the director.
   1. /sbin/modprobe ipip
   2. ifconfig tunl0 0.0.0.0 up
   3. ifconfig tunl0 192.168.1.176 netmask 255.255.255.255 broadcast 192.168.1.176 up
      
   4. Perform the following from another machine ( not your real server ).
      Before you hide the real server's VIP, ping 192.168.1.176 from another machine, then run arp -a. You should see that the MAC address for the VIP ( 192.168.1.176 ) matches that for tunl0 on the realserver.
      Clear the entry for the VIP with arp -d 192.168.1.176, and show that the arp entry is gone for the IP 192.168.1.176 (with arp -a)
      ping the VIP and look for the reappearance of the arp entry for the VIP ( with arp -a ).
   5. Hide the tunl0 interface with the following commands,
      # echo 1 > /proc/sys/net/ipv4/conf/all/hidden
      # echo 1 > /proc/sys/net/ipv4/conf/tunl0/hidden
   6. ping the VIP again from an outside machine. The VIP will most likely reply to the ping since the entry for the VIP is still in the arp table of the outside machine.
   7. Clear the arp entry (arp -d VIP) of the outside machine and ping the VIP again - this time you'll get no reply. This means that the VIP of the real server has been hidden.
   
   
   
6. Repeat what you did in lab 3 to construct a LVS cluster. You should be able to ssh into the cluster from an outside machine. Check the status with the command /sbin/ipvsadm. Also check using command w to see which machine you have logged in.