[afnog] Fwd: [afripv6-discuss] IPv6 rollout…

Mon Aug 6 10:25:46 UTC 2012

Thank you for sharing, you are solving the similar project which have
just started from my university, we are currently having 21,000
on-campus students with very challenging network infrastracture.
Hope we will share the experience, planning to implement both v4 and v6

On 8/3/12, Seun Ojedeji <seun.ojedeji at gmail.com> wrote:
> Thanks for sharing....Thanks Andrew for this writeup.. stared for future
> reference!
>
> Cheers!
>
> OJ
> sent from google nexus
> On Aug 3, 2012 2:14 PM, "Adiel Akplogan" <adiel at afrinic.net> wrote:
>
>> this May be of interest to some folk here.
>>
>> - a.
>>
>> Begin forwarded message:
>>
>> *From: *Andrew Alston <alston.networks at gmail.com>
>> *Date: *July 31, 2012 23:55:22 PM GMT+04:00
>> *To: *IPv6 in Africa <afripv6-discuss at afrinic.net>
>> *Subject: **Re: [afripv6-discuss] IPv6 rollout…*
>> *Reply-To: *IPv6 in Africa <afripv6-discuss at afrinic.net>
>>
>> Hi Guys,
>>
>> Ok a bit more info now that I have time to sit down and write, sorry
>> things have been rather hectic.
>>
>> Here is how this came about, and a bit more of the full story.
>>
>> The university in question was running a network without a real topology,
>> in essence, it was a flat network, v4 only, and a massive one at that.
>>  This was causing REAL issues, and it was the result of years and years
>> of
>> legacy.    The decision was taken that IPv6 would be required, but,
>> simply
>> put, we had to fix the network first.  So first step, how do you migrate
>> from a flat network running a single /16 flat, to a segmented network,
>> and
>> do it on a live campus environment that has 38 thousand users on it every
>> day using the network?  The answer... very very carefully, with a lot of
>> planning, and some very careful segment and IP planning.
>>
>> So, the following decisions were made:
>>
>> A.) We get rid of NAT in entirety - if we were going to do this, we would
>> do it properly, and the dual stack would be on live IPs in identical
>> topology v4 and v6 network wide.
>> B.) We divide the network into a three tier network, core, distribution
>> and edge.
>> C.) Core/Distribution would be routed, would have to be scalable, and
>> we'd
>> use SP style protocols to do this.
>> D.) Edge would remain layer 2, but we would choose not to span any L2
>> across distributions.  Since Core/Distribution in future will be MPLS
>> enabled, if we need L2 between to points, we can EoMPLS it.
>>
>> So we did our planning, and discovered (to our horror), that changing the
>> topology, eliminating NAT, and rolling out the wireless infrastructure we
>> had planned, we'd need a LOT more IPv4 space.  So, we applied and were
>> granted a second /15 PI space from AfriNIC.   We then started
>> implementation.
>>
>> First step:  Pick a network segment - we chose the student residences
>> (not
>> because we hate the students, but because we wouldn't break any critical
>> research if it all went horribly wrong).  Then, we moved all the student
>> residences between a single distribution, our residence distribution.  At
>> this point, Vlan 1 was still spanned to the residence distribution, so
>> NOTHING had broken in doing this, all was working.  Then, we created a
>> point to point link between that distribution and the core.  On the point
>> to point, we put a /30 v4 and a /126 v6.  (Sadly, the gear we are using
>> doesn't support either /31 or /127).
>>
>> Then, we enabled ospf for v4 and ospf3 for v6 across the point to point.
>>  There is a slight difference in the topologies at this point because the
>> ospf for v4 was configured to ONLY carry the point to point and the
>> loopbacks from the distributions, the rest is covered by iBGP, where as
>> in
>> v6, the hardware didn't do ipv6 bgp, so we had to do full route
>> distribution of v6 in OSPF3.
>>
>> Note at this point, we had still had zero downtime.
>>
>> Then, for each residence between the distribution, we created a vlan, and
>> assigned it an IP segment and an IPv6 segment.  To avoid mass routes in
>> our
>> routing tables, these segments were all taken out of supernets we had
>> dedicated to each distribution, and the supernets were what we pushed
>> into
>> the routing table on both v4 and v6 level.  So, the vlans were now
>> created
>> on the distribution, the routing was working.  Fixed up the DHCP for v4
>> as
>> well, so that was in place and ready to go with the correct scopes.
>> (Note,
>> we are using RA for v6 at this point, we haven't gotten around to DHCPv6
>> yet, so most people are still hitting the DNS servers on v4 addresses,
>> since we can't push DNS via RA).
>>
>> Note: Still ZERO downtime to anyone
>>
>> Then, we took the created vlan's on the distribution, trunked them down
>> to
>> the edge switches, waited till after hours, and moved the edge ports into
>> the correct vlan's.  (Different vlans for student pcs and wireless aps
>> etc).  The actual move into the correct vlans was like, a single command
>> on
>> each switch.  Then simple forced a port flap one very port as we went.
>> The
>> port flap was to force a DHCP reallocation on v4.
>>
>> Bang, the residences came up on the new topology with v4 and v6 - total
>> downtime to the clients - less than 30 seconds.
>>
>> Then, we did a rinse and repeat job through the various distributions
>> (we're still busy doing some of them, 6 outta 11 done so far, and
>> probably
>> around 300 or 400 edge switches tagged correctly).
>>
>> Once we were sure the topology was working, and the IPv6 was working,
>> next
>> step was to enable the ipv6 on the proxy servers, so that they could
>> fetch
>> via IPv6.  We did this, and instantly saw around 30% of the traffic
>> coming
>> in via v6, primarily google, youtube, facebook and akamai.  Note however,
>> at this point the clients were still seeing the proxies via v4, though
>> the
>> proxies were fetching via v6.  So next step, put in quad-a records for
>> the
>> proxy servers and for the pac file round robin.  Suddenly, everyone who
>> had
>> a v6 address was fetching from the proxy servers via v6, irrespective of
>> if
>> the proxies were fetching v4 or v6.
>>
>> Suddenly, we had a situation where 50% of the traffic coming in was via
>> IPv6, and we infact peaked at well over 100mbit of IPv6 traffic today
>> coming in off the Internet.
>>
>> Our next steps of course are to migrate the rest of the distributions and
>> edge to the new network, and infact in the next 10 minutes we'll be
>> moving
>> another thousand edge ports into this.  Once this is done, we'll start
>> looking closely at the server infrastructure and how we go about putting
>> the rest of the production servers both into the new topology and IPv6
>> enabling them.  We expect this to be the most problematic part, since we
>> know there are certain services which have issues with IPv6, but we'll
>> work
>> around those when we get there.
>>
>> In summary - it is entirely possible to take a network with around 15
>> thousand wired network points, a few hundred wireless access points, a
>> few
>> thousand VOIP phones and completely redeploy it both on a v4 and a v6
>> level
>> with almost no downtime if the planning is correct.  The traffic levels
>> also prove, there is IPv6 content out there, lots of it, and we're happy
>> to
>> use it!  It just takes some planning, some forethought and some people
>> willing to work really hard at strange hours getting it done.
>>
>> For interests sake, graphs can be seen here:
>>
>>
>> http://graphs.tenet.ac.za/iris/browser/browse?username=UFS&selectedmnemonicgroup=TSN81
>>
>> The graph marked vl1081 is the IPv4 interface to TENET (The South African
>> NREN), the graph marked vl3081 is the IPv4 interface to TENET.  We
>> specifically asked them to provide v4 and v6 on separate interfaces as it
>> did allow for us to see the traffic on a more individual basis as well,
>> which was useful.
>>
>> Hope this answers some of the questions I have been sent off list and
>> provides hope for those who believe that IPv6 migration is impossible -
>> never forget - we did it on both v4 and v6 *at the same time*, on a live
>> network, with no downtime, so if anyone doubts it can be done, we're
>> proof
>> that it can.
>>
>> Thanks
>>
>> Andrew Alston
>> Network Consultant
>>
>> NOTE: I write the above as a private individual and private consultant
>> and
>> have gained specific permission from my client (The University of the
>> Free
>> State) to relay this story.  I would like to say a special thank you to
>> them for allowing me to share this with you as well.
>>
>>
>> On 31 Jul 2012, at 3:58 PM, Maye diop <mayediop at gmail.com> wrote:
>>
>> Dear Andrew,
>> Congratulations.
>> I look forward more details to share with my universities.
>> Best Regards.
>> Le 31 juil. 2012 07:11, "Andrew Alston" <alston.networks at gmail.com> a
>> écrit :
>> >
>> > Hi Guys,
>> >
>> > So, while i'll be sending out a lot more data soon, with a lot more
>> information on exactly what we did and how we did it etc, I thought  I
>> would share some news that I for one found rather exciting.
>> >
>> > Yesterday evening starting at around 7pm one of the South African
>> universities turned up IPv6, in a fairly consistent manner.  Now, I'm not
>> talking about turning up IPv6 on a few servers, I'm talking about
>> integrating it into every part of their network.  By 2:30am this morning
>> it
>> was running on all their proxy servers, all their residence networks, the
>> wireless networks, all the lab PC's and a good portion of the staff
>> network.  The topology used was identical to that of the IPv4, and as the
>> rest of the network is migrated to the new IPv4 topology V6 will be
>> implemented on everything in dual stack along side that as well.
>> >
>> > Now, here is where things get interesting, another network dual stacked
>> is no real news, so lets talk about traffic levels.
>> >
>> > The University in question is now running anywhere between 30 to 50
>> percent of its internet traffic on IPv6, and its working flawlessly so
>> far.
>>  So flawlessly infact that even with Apple's default implementation of
>> Happy Eyeballs that tests RTT and defaults to v4 if the v6 latency is
>> higher, the apples we tested on running lion and mountain lion were still
>> choosing ipv6 most of the time.
>> >
>> > I am not going to say this little rollout has been easy though, we had
>> to rearchitecture the entire network (that had to happen anyway for
>> various
>> reasons), and we added the v6 as part of that project.  It would not have
>> been possible to do that without first getting our hands on another /15
>> worth of IPv4 space though to allow that rearchitecturing to happen
>> properly.
>> >
>> > As I said though, in coming days we'll write up what we did with a lot
>> more detail and send through some graphs and other information, I just
>> had
>> to share the fact that we're seeing at points half the traffic on a
>> standard university coming in from the internet over ipv6!
>> >
>> > Thanks
>> >
>> > Andrew Alston
>> > Network Consultant_______________________________________________
>> > afripv6-discuss mailing list
>> > afripv6-discuss at afrinic.net
>> > https://lists.afrinic.net/mailman/listinfo.cgi/afripv6-discuss
>>  _______________________________________________
>> afripv6-discuss mailing list
>> afripv6-discuss at afrinic.net
>> https://lists.afrinic.net/mailman/listinfo.cgi/afripv6-discuss
>>
>>
>> _______________________________________________
>> afripv6-discuss mailing list
>> afripv6-discuss at afrinic.net
>> https://lists.afrinic.net/mailman/listinfo.cgi/afripv6-discuss
>>
>>
>>
>> _______________________________________________
>> afnog mailing list
>> http://afnog.org/mailman/listinfo/afnog
>>
>

-- 
MATOGORO Jabhera
Instructor Cum Technical Staff, University Computing Centre (2008-Aug.2009)
Computer Programmer/Systems Analyst I, UDOM-ICT Unit (Sept.2009-Dec.2011)
ARIS Database Administrator-UDOM (2010-To Date)
Assistant Lecturer,UDOM-School of Informatics (14 Dec.2011-To Date)
The University of Dodoma-UDOM