IPv6: Floating IPs and Duplicate Address Detection

The very nature of the floating IPs can lead to some classical quirks in a distributed system network. This discussion mainly focuses on IPv6, and how its duplicate IP detection mechanism can clash with the floating IP technique.

Floating IPs are a common scenario in Highly Available or Scaled-out Distributed Systems. The basic idea behind it is to have a transient IP address that can move from one node to another, keeping the change of serving-node transparent on the access-side of the network. For instance, if there are two server machines, each represented by an unique IP, and one of them goes down, then its IP address “floats” to the other server which will henceforth process the client requests. This technique is widely used to provide seamless transition from one serving-node to another in case of failures. One such implementation is present in OpenStack Nova.

On the other hand, Duplicate Address Detection (DAD) is a mechanism to identify if same IP is assigned to multiple nodes in a local network. It is implemented using the Neighbor Discovery Protocol (NDP) under IPv6. It uses the Neighbor Solicitation (NS) and Neighbor Advertisement (NA) messages. The operation is applicable to all the IPs that are link-local. More specifically:

  • all the IPs that fall under the link-local address-family
  • all the IPs that fall under global address-family but are present on the same LAN (one hop away on the link)

This connotation of the “link-local” is not obvious right away from the RFC 4862 that describes IP assignment and DAD. In other words, DAD can run for any IP that is valid candidate for the Neighbor Cache Table. Also, DAD runs for both auto-generated and statically-assigned IPs.

The implications of DAD failure (duplicate IP present) are a bit harsh! RFC 4862 prescribes following actions in such an event, quote:

- not send any IP packets from the interface,
- silently drop any IP packets received on the interface, and
- not forward any IP packets to the interface (when acting as a router 
  or processing a packet with a Routing header)

This is, specifically, what Linux Kernel IPv6 implementation does:

  • If an auto-assigned link-local duplicate IP is detected, either the new auto-assigned IP generation is triggered again (accept_dad times), or IPv6 functionality is disabled on that interface
  • If an user-assigned link-local duplicate IP is detected — unlike the previous case — the IPv6 functionality stays enabled, even the interface state stays UP and RUNNING, but there are no more IP operations happening as the IPv6 autoconf state-machine gets stuck in the "TENTETIVE" state.

This is visible from user-space through the ip utility from bash. Below we see the interface eth16a has IP 2011:db6:0:1::1/0 with flag tentative set. In normal scenario, this flag vanishes in seconds as the DAD is successful. But it is persistently visible when the DAD fails, implying the IP configuration is stuck in the tentative state.

$ ip addr show eth16a
14: eth16a: <BROADCAST,MULTICAST> mtu 1500 qdisc mq qlen 1000
    link/ether 00:de:13:98:87:3c brd ff:ff:ff:ff:ff:ff
    inet6 2011:db6:0:1::1/0 scope global tentative
       valid_lft forever preferred_lft forever

A peek into the Linux kernel code confirms the behavior and implementation. In the file net/ipv6/addrconf.c, the addrconf_dad_failure() defines the action for auto-assigned IP.

void addrconf_dad_failure(struct inet6_ifaddr *ifp)

It either regenerates the address depending on configuration, or disables the IP operations for the auto-assigned IPs.

idev->cnf.disable_ipv6 = 1;

This is same as accessing the sysctl parameter with the same name, except now it is done by the kernel, without the user’s volition. More on this at the end.

For the user-assigned IP, the following function is invoked.

static void addrconf_dad_stop(struct inet6_ifaddr *ifp, int dad_failed)

The user-assigned IP is identified by the IFA_F_PERMANENT flag. When the above function detects this flag, it adds in the IFA_F_TENTATIVE flag. The IFA_F_TENTATIVE flag is indicative that the DAD protocol has not (yet) approved this IP. Thus, this IP is stuck in this state forever. Further, if the DAD has failed, this function triggers removal of all the routes corresponding to this specific IP.

Conclusion

To summarize it concisely, the Duplicate Address Detection (DAD) protocol deludes the system into misinterpreting the Floating IP behavior which is a completely valid and an extremely practical scenario. Someone’s specification; someone else’s constraint.

        IPv6 is like a peacock: beautiful and enchanting; at the same time, intricate,                     gaudy and ostentatious!

The scenario discussed is a classic example of enforcing policy instead of merely providing a mechanism. A software should do good enough with respect to what the user wants, but not so much to over-anticipate user-expectations.

Solution

Disable DAD before configuring the floating IP and enable it back when it is out of the tentative state. This could be done using the sysctl system call from libc, through sysctl utility or through procfs. Details are in the next section.

sysclt parameters for DAD

sysctl command run on bash to read and write to the kernel variables, respectively.

sysctl net.ipv6.conf.default.<dad_option>
sysctl -w net.ipv6.conf.default.<dad_option>=<value>

The default can be replaced by interface name to make configuration specific to a particular interface. These are also accessible through procfs: /proc/sys/net/ipv6/. The dad_option refers to the parameters as below:

dad_transmits - INTEGER
    The amount of Duplicate Address Detection probes to send.
    Default: 1

accept_dad - INTEGER
    Whether to accept DAD (Duplicate Address Detection).
    0: Disable DAD
    1: Enable DAD (default)
    2: Enable DAD, and disable IPv6 operation if MAC-based 
       duplicate link-local address has been found.

idgen_delay - INTEGER
    Controls the delay in seconds after which time to retry
    privacy stable address generation if a DAD conflict is
    detected.
    Default: 1 (as specified in RFC7217)

idgen_retries - INTEGER
    Controls the number of retries to generate a stable privacy
    address if a DAD conflict is detected.
    Default: 3 (as specified in RFC7217)

optimistic_dad - BOOLEAN
    Whether to perform Optimistic Duplicate Address Detection(RFC 4429)
    0: disabled (default)
    1: enabled

use_optimistic - BOOLEAN
    If enabled, do not classify optimistic addresses as deprecated 
    during source address selection.  Preferred addresses will still be
    chosen before optimistic addresses, subject to other ranking in the
    source address selection algorithm.
    0: disabled (default)
    1: enabled

disable_ipv6 - BOOLEAN
    Disable IPv6 operation.  If accept_dad is set to 2, this value
    will be dynamically set to TRUE if DAD fails for the link-local
    address.
    Default: FALSE (enable IPv6 operation)
    When this value is changed from 1 to 0 (IPv6 is being enabled),
    it will dynamically create a link-local address on the given
    interface and start Duplicate Address Detection, if necessary.
    When this value is changed from 0 to 1 (IPv6 is being disabled),
    it will dynamically delete all address on the given interface.
References:

[1] RFC 4862: https://tools.ietf.org/html/rfc4862
[2] Linux kernel sources (http://www.kernel.org) Documentation/networking/ip-sysctl.txt
[3] OpenStack Nova: http://docs.openstack.org/developer/nova/

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s