If you haven't already read the last part of this series, you should do that first, you can find that here.
The network has been running well for a while and it is time to expand. I can either grow at the datacenter and add another rack, or I could add a whole new location. I decided to go with the new location method and looked into getting a rack closer to the best city in the world, Chicago.
Reached out to a few datacenters, got quotes and decided to go with Netrality's Chicago location at 717 S Wells for the next POP. Now that I had the power and space ready, I had to reach out to network providers to get connectivity at the datacenter. For redundancy reasons, we need to get at lest 2 network providers, that way if one is down, we can use the the other one. We run both at the same time to balance the traffic and let BGP determine which path to use.
Now that we had a datacenter and network connectivity solutions in place, we need IPs for the new region. Blocks of IPv4 addresses must be announced to our transit providers with a size of /24(256 ips). So we can't just use part of the IPs from our San Fransisco region in Chicago, since we'd need to announce the IPs in both regions, and then find a way to get the traffic to the right end destination(probably a tunnel of some sort as an actual link between the 2 locations would be costly). Instead we chose to try and get on the waitlist for more IPv4 blocks.
During the middle of the application for the waitlist, we decided to go ahead and buy a block of ipv4 addresses as well, while expensive, it felt justified. We learned that if we buy any, we would get kicked off the waitlist, so we cancelled our application to get on the waitlist, and just bought the IPs. Now we've got IPs.
Now we need hardware for the new region. All we really need is networking gear, since we don't need to have company owned servers here(though I did personally buy my own server to colocate here, since who's going to give me a better deal than, well me). I decided to buy the same Mikrotik setup as we had in San Fransisco.
After about 2 months, we started seeing some issues with the Mikrotik hardware in both regions and decided to do a swap to Arista gear, and we went with the 7050SX series of switches, and 7280SR series of routers. The switches have 4 40G ports and 48 10G SFP+ ports and the routers have 6x 100G ports and 48x 10G ports should should be plenty for our current needs. We bought 2 switches and a router for Chicago. Installed them and swapped over to them. Everything went fine and no more weird Mikrotik issues. So we bought the same for San Fransisco. That upgrade hasn't happened yet, but we have started moving some ports to the switch(the router is sitting under my bed as of now, pending configuration).
On top of that, we've also started looking into European sites. I shipped a router out to Stockholm, Sweden a while ago and are pending installation of that. Up next is probably NIKHEF in the Netherlands. Don't plan on doing any colo there, just some network connectivity.
We also connected to FD-IX and Megaport Internet Exchange in Chicago(though FD-IX we are still waiting on the IX to get turned up, and the Megaport IX we are still in the middle of configuring, which will probably happen sometime after the holidays).
And that brings us to the current day. The current network is in a state of mid upgrades and has 2 pretty well built up sites which are slowly increasing on the traffic needs and getting upgrades.
