[afnog] Fwd: [afripv6-discuss] IPv6 rollout…

Fri Aug 3 16:33:09 UTC 2012

Thanks for sharing....Thanks Andrew for this writeup.. stared for future
reference!

Cheers!

OJ
sent from google nexus
On Aug 3, 2012 2:14 PM, "Adiel Akplogan" <adiel at afrinic.net> wrote:

> this May be of interest to some folk here.
>
> - a.
>
> Begin forwarded message:
>
> *From: *Andrew Alston <alston.networks at gmail.com>
> *Date: *July 31, 2012 23:55:22 PM GMT+04:00
> *To: *IPv6 in Africa <afripv6-discuss at afrinic.net>
> *Subject: **Re: [afripv6-discuss] IPv6 rollout…*
> *Reply-To: *IPv6 in Africa <afripv6-discuss at afrinic.net>
>
> Hi Guys,
>
> Ok a bit more info now that I have time to sit down and write, sorry
> things have been rather hectic.
>
> Here is how this came about, and a bit more of the full story.
>
> The university in question was running a network without a real topology,
> in essence, it was a flat network, v4 only, and a massive one at that.
>  This was causing REAL issues, and it was the result of years and years of
> legacy.    The decision was taken that IPv6 would be required, but, simply
> put, we had to fix the network first.  So first step, how do you migrate
> from a flat network running a single /16 flat, to a segmented network, and
> do it on a live campus environment that has 38 thousand users on it every
> day using the network?  The answer... very very carefully, with a lot of
> planning, and some very careful segment and IP planning.
>
> So, the following decisions were made:
>
> A.) We get rid of NAT in entirety - if we were going to do this, we would
> do it properly, and the dual stack would be on live IPs in identical
> topology v4 and v6 network wide.
> B.) We divide the network into a three tier network, core, distribution
> and edge.
> C.) Core/Distribution would be routed, would have to be scalable, and we'd
> use SP style protocols to do this.
> D.) Edge would remain layer 2, but we would choose not to span any L2
> across distributions.  Since Core/Distribution in future will be MPLS
> enabled, if we need L2 between to points, we can EoMPLS it.
>
> So we did our planning, and discovered (to our horror), that changing the
> topology, eliminating NAT, and rolling out the wireless infrastructure we
> had planned, we'd need a LOT more IPv4 space.  So, we applied and were
> granted a second /15 PI space from AfriNIC.   We then started
> implementation.
>
> First step:  Pick a network segment - we chose the student residences (not
> because we hate the students, but because we wouldn't break any critical
> research if it all went horribly wrong).  Then, we moved all the student
> residences between a single distribution, our residence distribution.  At
> this point, Vlan 1 was still spanned to the residence distribution, so
> NOTHING had broken in doing this, all was working.  Then, we created a
> point to point link between that distribution and the core.  On the point
> to point, we put a /30 v4 and a /126 v6.  (Sadly, the gear we are using
> doesn't support either /31 or /127).
>
> Then, we enabled ospf for v4 and ospf3 for v6 across the point to point.
>  There is a slight difference in the topologies at this point because the
> ospf for v4 was configured to ONLY carry the point to point and the
> loopbacks from the distributions, the rest is covered by iBGP, where as in
> v6, the hardware didn't do ipv6 bgp, so we had to do full route
> distribution of v6 in OSPF3.
>
> Note at this point, we had still had zero downtime.
>
> Then, for each residence between the distribution, we created a vlan, and
> assigned it an IP segment and an IPv6 segment.  To avoid mass routes in our
> routing tables, these segments were all taken out of supernets we had
> dedicated to each distribution, and the supernets were what we pushed into
> the routing table on both v4 and v6 level.  So, the vlans were now created
> on the distribution, the routing was working.  Fixed up the DHCP for v4 as
> well, so that was in place and ready to go with the correct scopes.  (Note,
> we are using RA for v6 at this point, we haven't gotten around to DHCPv6
> yet, so most people are still hitting the DNS servers on v4 addresses,
> since we can't push DNS via RA).
>
> Note: Still ZERO downtime to anyone
>
> Then, we took the created vlan's on the distribution, trunked them down to
> the edge switches, waited till after hours, and moved the edge ports into
> the correct vlan's.  (Different vlans for student pcs and wireless aps
> etc).  The actual move into the correct vlans was like, a single command on
> each switch.  Then simple forced a port flap one very port as we went.  The
> port flap was to force a DHCP reallocation on v4.
>
> Bang, the residences came up on the new topology with v4 and v6 - total
> downtime to the clients - less than 30 seconds.
>
> Then, we did a rinse and repeat job through the various distributions
> (we're still busy doing some of them, 6 outta 11 done so far, and probably
> around 300 or 400 edge switches tagged correctly).
>
> Once we were sure the topology was working, and the IPv6 was working, next
> step was to enable the ipv6 on the proxy servers, so that they could fetch
> via IPv6.  We did this, and instantly saw around 30% of the traffic coming
> in via v6, primarily google, youtube, facebook and akamai.  Note however,
> at this point the clients were still seeing the proxies via v4, though the
> proxies were fetching via v6.  So next step, put in quad-a records for the
> proxy servers and for the pac file round robin.  Suddenly, everyone who had
> a v6 address was fetching from the proxy servers via v6, irrespective of if
> the proxies were fetching v4 or v6.
>
> Suddenly, we had a situation where 50% of the traffic coming in was via
> IPv6, and we infact peaked at well over 100mbit of IPv6 traffic today
> coming in off the Internet.
>
> Our next steps of course are to migrate the rest of the distributions and
> edge to the new network, and infact in the next 10 minutes we'll be moving
> another thousand edge ports into this.  Once this is done, we'll start
> looking closely at the server infrastructure and how we go about putting
> the rest of the production servers both into the new topology and IPv6
> enabling them.  We expect this to be the most problematic part, since we
> know there are certain services which have issues with IPv6, but we'll work
> around those when we get there.
>
> In summary - it is entirely possible to take a network with around 15
> thousand wired network points, a few hundred wireless access points, a few
> thousand VOIP phones and completely redeploy it both on a v4 and a v6 level
> with almost no downtime if the planning is correct.  The traffic levels
> also prove, there is IPv6 content out there, lots of it, and we're happy to
> use it!  It just takes some planning, some forethought and some people
> willing to work really hard at strange hours getting it done.
>
> For interests sake, graphs can be seen here:
>
>
> http://graphs.tenet.ac.za/iris/browser/browse?username=UFS&selectedmnemonicgroup=TSN81
>
> The graph marked vl1081 is the IPv4 interface to TENET (The South African
> NREN), the graph marked vl3081 is the IPv4 interface to TENET.  We
> specifically asked them to provide v4 and v6 on separate interfaces as it
> did allow for us to see the traffic on a more individual basis as well,
> which was useful.
>
> Hope this answers some of the questions I have been sent off list and
> provides hope for those who believe that IPv6 migration is impossible -
> never forget - we did it on both v4 and v6 *at the same time*, on a live
> network, with no downtime, so if anyone doubts it can be done, we're proof
> that it can.
>
> Thanks
>
> Andrew Alston
> Network Consultant
>
> NOTE: I write the above as a private individual and private consultant and
> have gained specific permission from my client (The University of the Free
> State) to relay this story.  I would like to say a special thank you to
> them for allowing me to share this with you as well.
>
>
> On 31 Jul 2012, at 3:58 PM, Maye diop <mayediop at gmail.com> wrote:
>
> Dear Andrew,
> Congratulations.
> I look forward more details to share with my universities.
> Best Regards.
> Le 31 juil. 2012 07:11, "Andrew Alston" <alston.networks at gmail.com> a
> écrit :
> >
> > Hi Guys,
> >
> > So, while i'll be sending out a lot more data soon, with a lot more
> information on exactly what we did and how we did it etc, I thought  I
> would share some news that I for one found rather exciting.
> >
> > Yesterday evening starting at around 7pm one of the South African
> universities turned up IPv6, in a fairly consistent manner.  Now, I'm not
> talking about turning up IPv6 on a few servers, I'm talking about
> integrating it into every part of their network.  By 2:30am this morning it
> was running on all their proxy servers, all their residence networks, the
> wireless networks, all the lab PC's and a good portion of the staff
> network.  The topology used was identical to that of the IPv4, and as the
> rest of the network is migrated to the new IPv4 topology V6 will be
> implemented on everything in dual stack along side that as well.
> >
> > Now, here is where things get interesting, another network dual stacked
> is no real news, so lets talk about traffic levels.
> >
> > The University in question is now running anywhere between 30 to 50
> percent of its internet traffic on IPv6, and its working flawlessly so far.
>  So flawlessly infact that even with Apple's default implementation of
> Happy Eyeballs that tests RTT and defaults to v4 if the v6 latency is
> higher, the apples we tested on running lion and mountain lion were still
> choosing ipv6 most of the time.
> >
> > I am not going to say this little rollout has been easy though, we had
> to rearchitecture the entire network (that had to happen anyway for various
> reasons), and we added the v6 as part of that project.  It would not have
> been possible to do that without first getting our hands on another /15
> worth of IPv4 space though to allow that rearchitecturing to happen
> properly.
> >
> > As I said though, in coming days we'll write up what we did with a lot
> more detail and send through some graphs and other information, I just had
> to share the fact that we're seeing at points half the traffic on a
> standard university coming in from the internet over ipv6!
> >
> > Thanks
> >
> > Andrew Alston
> > Network Consultant_______________________________________________
> > afripv6-discuss mailing list
> > afripv6-discuss at afrinic.net
> > https://lists.afrinic.net/mailman/listinfo.cgi/afripv6-discuss
>  _______________________________________________
> afripv6-discuss mailing list
> afripv6-discuss at afrinic.net
> https://lists.afrinic.net/mailman/listinfo.cgi/afripv6-discuss
>
>
> _______________________________________________
> afripv6-discuss mailing list
> afripv6-discuss at afrinic.net
> https://lists.afrinic.net/mailman/listinfo.cgi/afripv6-discuss
>
>
>
> _______________________________________________
> afnog mailing list
> http://afnog.org/mailman/listinfo/afnog
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://afnog.org/pipermail/afnog/attachments/20120803/40803cfa/attachment-0001.html>