[afnog] A heads up on a nasty IPv6 bug

[Mukom Akong T.]

Very succulent description. It still doesn't explain why host behind CPE gets two different default gateways as Andrew is reporting. Unless in reality, between provisioning the various LAN-side /64s (or some other yet to be identified event), the link local address of the LAN interface changes. 
Static Prefix will fix the problem of reachability due to an no-longer valid prefix. The different default gateways issue will still be present. Right?




		_____________________________
From: Jan Zorz <zorz at isoc.org>
Sent: Monday, August 15, 2016 8:35 AM
Subject: Re: [afnog] A heads up on a nasty IPv6 bug
To: Andrew Alston <andrew.alston at liquidtelecom.com>, Mukom Akong T. <mukom.tamon at gmail.com>
Cc:  <afnog at afnog.org>


Hey,

Yes, this dynamic way of assigning IPv6 PDs is causing much trouble to
operators around the world until they decide to swallow the initial pain
and change to static PD assignments... ;)

Please see my comments inline...

On 14/08/16 16:40, Andrew Alston wrote:
> If you are automatically allocating usernames for PPPoE authentication
> its relatively simple to tie that username provisioning to static
> assignments.
> 
> As an example, if your username ends in a numeric on your auto
> provisioning system its relatively simple to use some basic maths and
> hex conversion to produce a static subnet that’s tied.

Yes, usually some math formula is created to tie the PD and username and
then the script populates additional field in your radius database and
after that - the user always gets the same IPv6 PD "for life".

If you have multiple aggregation or termination points then some
observation is needed prior to this so you can group users on same
termination points to have PD from same aggregated prefix, but this is
trivial.

> With regards to the subnet issue on the RA, I’ll respond to that later,
> though perhaps Jan would also like to make some comments on this, since
> his understanding of it is admittedly better than mine until I do more
> testing and labbing

Ok, let's un-dust the old saying:

"In theory there is no difference between theory and practice. In
practice there is."

For sake of simplicity, let's say we have only 3 components:

+--------+       wan +---------+ lan        +-----------+
|  ISP   |-----------|   CPE   |------------|   host    |
+--------+           +---------+            +-----------+

ISP can be any access equipment you are using, for example BRAS or
anything else.

CPE has for simplicity just WAN access and LAN segment behind the CPE
for home network.

host is any device that we use on our home network - it can be computer,
laptop, tablet, mobile phone, printer - anything that connects to our
network and can autoconfigure IPv6.

Theory:
- CPE connects to ISP and gets the Prefix Delegation (PD) from ISP.
- ISP installs a route for that PD segment towards the CPE wan interface
- CPE provisions /64 out of PD to LAN interface and starts sending out
RA (Router Advertisements) packets with prefix information to LAN
- host connects to LAN network and sends out na RS (Router Solicitation)
packet that is responded by RA packet containing prefix information.
- host accepts the packet, generates IPv6 address(es), does the DAD
process and if all good - sets up the IPv6 addresses and sets the
default route to source IPv6 address of RA message - that is a
link-local address of a CPE LAN interface.
- now IPv6 traffic can start flowing.
- ISP decides that PD must change, or something is wrong with wan link
and the PD assignment process restarts (for example pppoe client restarts)
- in this event CPE gets a different PD from ISP and need to delete the
old IPv6 address from LAN port that is no longer in assigned PD.
- ISP installs a route for that PD segment towards the CPE wan interface
and removes the route for old PD towards that CPE
- CPE adds a new IPv6 address from new /64 from new PD to LAN, deletes
the old IPv6 address and sends to LAN link the RA packet with old prefix
information with lifetime 0
- ISP removes the route
- all hosts that receives RA packet with lifetime 0 must remove the old
IPv6 address and stop using it.
- now we have CPE with new IPv6 PD and all hosts on LAN link with just
IPv6 addresses from new PD and world is beautiful and nice and a safe place.

This was the theory, now let's see some practice and real world - what
can go (and will) go wrong?

We have 2 failure modes here:
- host never receives RA packet with lifetime 0 and ends up with IPv6
addresses from old and new IPv6 PD
- host receives RA packet with lifetime 0, but doesn't care much because
it's implemented wrong (and this is happening, given the wide variety of
end user devices that are in use on this earth)

In both cases we end up with a device that has the option of using the
wrong IPv6 address as a source address for sent packet.

Source address selection mechanism is broken in this case and there is
an ongoing discussion at IETF how to fix that, but there will be some
time before any fix becomes standard and even more time before it's
implemented.

Currently hosts selects the source IPv6 address for packets in quite
variety of ways - some randomly, some "address that was last allocated",
some of them "first one until it's valid", and all other possible ways,
depending on OS and vendor.

Problem is, that after changing the PD - ISP removed the route back to
CPE for that old PD and some of those ISPs that went the "less optimal"
path down the IPv6 road and started with dynamic PD assignments did a
quick fix to put the old PD in "quarantene" and keep the old route back
to CPE for additional 24 hours so all old IPv6 addresses in LAN segment
behind the CPE expires and vanishes. This is a quick hack so you get a
quazi functional access network and can dedicate your time to start
planing a process to make PD assignments properly static (also over BRAS
reboots).

So, this is a pain that many of operators went through and the solution
is quite obvious: go with static IPv6 PD assignments. Your help desk
will appreciate you.

Again: "In theory there is no difference between theory and practice. In
practice there is."

I hope I shed some light on the issue with the above explanation.

See you all in Mauritius for Afrinic-25 where we can discuss IPv6
deployment challenges at length ;)

Cheers and thnx, Jan Zorz

-- 
Jan Zorz
Internet Society
mailto:<zorz at isoc.org>
http://www.internetsociety.org/deploy360/
--
"Engineering is always positive in results..." N. Tesla



	
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.afnog.org/pipermail/afnog/attachments/20160815/563e9f77/attachment.html>