The very nature of the floating IPs can lead to some classical quirks in a distributed system network. This discussion mainly focuses on IPv6, and how its duplicate IP detection mechanism can clash with the floating IP technique.
Floating IPs are a common scenario in Highly Available or Scaled-out Distributed Systems. The basic idea behind it is to have a transient IP address that can move from one node to another, keeping the change of serving-node transparent on the access-side of the network. For instance, if there are two server machines, each represented by an unique IP, and one of them goes down, then its IP address “floats” to the other server which will henceforth process the client requests. This technique is widely used to provide seamless transition from one serving-node to another in case of failures. One such implementation is present in OpenStack Nova.
On the other hand, Duplicate Address Detection (DAD) is a mechanism to identify if same IP is assigned to multiple nodes in a local network. It is implemented using the Neighbor Discovery Protocol (NDP) under IPv6. It uses the Neighbor Solicitation (NS) and Neighbor Advertisement (NA) messages. The operation is applicable to all the IPs that are link-local. More specifically:
- all the IPs that fall under the link-local address-family
- all the IPs that fall under global address-family but are present on the same LAN (one hop away on the link)
This connotation of the “link-local” is not obvious right away from the RFC 4862 that describes IP assignment and DAD. In other words, DAD can run for any IP that is valid candidate for the Neighbor Cache Table. Also, DAD runs for both auto-generated and statically-assigned IPs.
The implications of DAD failure (duplicate IP present) are a bit harsh! RFC 4862 prescribes following actions in such an event, quote:
- not send any IP packets from the interface, - silently drop any IP packets received on the interface, and - not forward any IP packets to the interface (when acting as a router or processing a packet with a Routing header)
This is, specifically, what Linux Kernel IPv6 implementation does:
- If an auto-assigned link-local duplicate IP is detected, either the new auto-assigned IP generation is triggered again (accept_dad times), or IPv6 functionality is disabled on that interface
- If an user-assigned link-local duplicate IP is detected — unlike the previous case — the IPv6 functionality stays enabled, even the interface state stays UP and RUNNING, but there are no more IP operations happening as the IPv6 autoconf state-machine gets stuck in the "TENTETIVE" state.
This is visible from user-space through the ip utility from bash. Below we see the interface eth16a has IP 2011:db6:0:1::1/0 with flag tentative set. In normal scenario, this flag vanishes in seconds as the DAD is successful. But it is persistently visible when the DAD fails, implying the IP configuration is stuck in the tentative state.
$ ip addr show eth16a 14: eth16a: <BROADCAST,MULTICAST> mtu 1500 qdisc mq qlen 1000 link/ether 00:de:13:98:87:3c brd ff:ff:ff:ff:ff:ff inet6 2011:db6:0:1::1/0 scope global tentative valid_lft forever preferred_lft forever
A peek into the Linux kernel code confirms the behavior and implementation. In the file net/ipv6/addrconf.c, the addrconf_dad_failure() defines the action for auto-assigned IP.
void addrconf_dad_failure(struct inet6_ifaddr *ifp)
It either regenerates the address depending on configuration, or disables the IP operations for the auto-assigned IPs.
idev->cnf.disable_ipv6 = 1;
This is same as accessing the sysctl parameter with the same name, except now it is done by the kernel, without the user’s volition. More on this at the end.
For the user-assigned IP, the following function is invoked.
static void addrconf_dad_stop(struct inet6_ifaddr *ifp, int dad_failed)
The user-assigned IP is identified by the IFA_F_PERMANENT flag. When the above function detects this flag, it adds in the IFA_F_TENTATIVE flag. The IFA_F_TENTATIVE flag is indicative that the DAD protocol has not (yet) approved this IP. Thus, this IP is stuck in this state forever. Further, if the DAD has failed, this function triggers removal of all the routes corresponding to this specific IP.
To summarize it concisely, the Duplicate Address Detection (DAD) protocol deludes the system into misinterpreting the Floating IP behavior which is a completely valid and an extremely practical scenario. Someone’s specification; someone else’s constraint.
IPv6 is like a peacock: beautiful and enchanting; at the same time, intricate, gaudy and ostentatious!
The scenario discussed is a classic example of enforcing policy instead of merely providing a mechanism. A software should do good enough with respect to what the user wants, but not so much to over-anticipate user-expectations.
Disable DAD before configuring the floating IP and enable it back when it is out of the tentative state. This could be done using the sysctl system call from libc, through sysctl utility or through procfs. Details are in the next section.
sysclt parameters for DAD
sysctl command run on bash to read and write to the kernel variables, respectively.
sysctl net.ipv6.conf.default.<dad_option> sysctl -w net.ipv6.conf.default.<dad_option>=<value>
The default can be replaced by interface name to make configuration specific to a particular interface. These are also accessible through procfs: /proc/sys/net/ipv6/. The dad_option refers to the parameters as below:
dad_transmits - INTEGER The amount of Duplicate Address Detection probes to send. Default: 1 accept_dad - INTEGER Whether to accept DAD (Duplicate Address Detection). 0: Disable DAD 1: Enable DAD (default) 2: Enable DAD, and disable IPv6 operation if MAC-based duplicate link-local address has been found. idgen_delay - INTEGER Controls the delay in seconds after which time to retry privacy stable address generation if a DAD conflict is detected. Default: 1 (as specified in RFC7217) idgen_retries - INTEGER Controls the number of retries to generate a stable privacy address if a DAD conflict is detected. Default: 3 (as specified in RFC7217) optimistic_dad - BOOLEAN Whether to perform Optimistic Duplicate Address Detection(RFC 4429) 0: disabled (default) 1: enabled use_optimistic - BOOLEAN If enabled, do not classify optimistic addresses as deprecated during source address selection. Preferred addresses will still be chosen before optimistic addresses, subject to other ranking in the source address selection algorithm. 0: disabled (default) 1: enabled disable_ipv6 - BOOLEAN Disable IPv6 operation. If accept_dad is set to 2, this value will be dynamically set to TRUE if DAD fails for the link-local address. Default: FALSE (enable IPv6 operation) When this value is changed from 1 to 0 (IPv6 is being enabled), it will dynamically create a link-local address on the given interface and start Duplicate Address Detection, if necessary. When this value is changed from 0 to 1 (IPv6 is being disabled), it will dynamically delete all address on the given interface.