Network and Telecom Strategies Blog

Google is Building its own ___________?

Those of us who follow the telecom industry have no doubt heard the rumor that Google is going to build its own routers.  It was first reported in the SD Times who quoted a "super-secret" source.  Super-secret in this case is almost redundant; everything inside Google is super-secret.  But this isn't the first time one of the Google employees has leaked a tidbit about the operation, nor is it the first time Google has struck out on its own:

Google Builds its own Servers for Efficiency

Google's Secret 10GbE Switch

Google Builds its own Virtual Machine for Android

Google builds a CDN for its own content

Google Launching its own Navy

I especially like that last one.

I've spent very little time in and around Google, but it's obvious, even to me, that Google has both the money and the intellectual horsepower to pull something like this off.  Walking around their campus and looking at the nameplates on the doors is like a trip to  your favorite tech bookstore, only it's not the books that are on display behind the glass-lined offices, it's the authors themselves.  Can Google build a router, you bet. 

So, I may not know what goes on behind the doors at Google but I've been closely involved with router companies, close enough to know goes into building a high-end edge router:

The right people

$30-$50 million dollars 

12-18 months

Put these three things together with motivation (a winning idea, or a networking problem only Google could have) and just about anything is possible.  

For Google, $30-$50 million isn't really all that much money.  Even so, we might shave a few million dollars from that estimate if Google gets someone to provide the ASICs - they were rumored to be using Broadcom for their 10GbE switches.  Google may not be ready to fab their own silicon but they'll need custom silicon if they want to achieve the forwarding rates found in a modern router such as a Juniper (that is unless their developers are "other-worldly" - they're too far from Area 51 and I didn't see anyone with dark eyes and huge craniums in my visits).  We can shave off a few million dollars more because Google won't need marketing and sales to sell this new device, they won't be kneeling at the altar of Verizon and AT&T trying to get large telecom companies to take notice of their new hardware.   In other words, this is probably a $20-$30 million problem for Google and they have the ability to solve it themselves.

So, who cares if Google builds a router?  Well, their current supplier of high-end routers for one.  Is it really fair, however, to lay this 'failure' at the feet of Juniper?  Wall Street bounced Juniper a little bit on the news last week, but then, Wall Street has all of the macho of my 91 year-old grandmother these days.  No, Juniper isn't to blame, nor is this anything more than a short-term threat to Juniper's revenue (and it's hard to say just how much revenue, all we know is that Google isn't Juniper's largest reported client - Verizon Business is with 10% of reported revenue).  On a side note, it would have been interesting to see the market's reaction to this event if the supplier where Cisco, not Juniper.  

In the long term, unless Google has had this project under wraps for some time I'd be shocked to see a device in less than 12 months, and even more shocked to see Google get that device in place, tested, functioning and manufactured in numbers in less than 24 months.    Even at that, this device will never make it out into the wild, it won't be competing with the likes of Juniper or Cisco.  In each of the cases where Google chose to build rather than buy, the only effect has been to remove Google from the list of potential customers for servers, switches and now, perhaps, routers.  The server market didn't collapse, the market for 10GbE switches didn't collapse, the market for routers won't collapse either.  

If I were a betting man, I'd bet that Juniper will be fighting hard to stay involved with this project.  Google is a unique animal in this industry - so unique in fact that it may not be profitable to be their router or switch company.  That is not to say, however that it won't be beneficial to learn from what they're doing.  Juniper may be uniquely positioned to offer their silicon and in exchange, learn from what Google is doing.  It's all speculation, but if I had to bet on it, I believe it may be better for Juniper than first thought.  

Security (or lack of) in the Internet Routing System

About a week ago, a small ISP located in Pakistan blocked access to a few IP addresses belonging to YouTube. Unfortunately, they blocked the sites for the entire Internet. The media coverage was full of today's buzzword fodder: Islam, censorship, Internet, Network Neutrality and so forth; most stories were focused on "Internet Censorship." In the end, the story died down, Internet routing returned to normal and Google/YouTube pulled the offending content off of their servers. What wasn't widely reported? The mechanism used by the ISP, why it affected the larger YouTube audience instead of affecting only the customers of the ISP inside Pakistan, and why those that have a corporate presence on the Internet (or rely on the infrastructure) should care.

As with all blunders, (this one wasn't all that large by comparison) it took a chain of events that went unbroken to achieve the result. First, the ISP acted in response to a directive from the Pakistani Government. The directive (available here) instructed that a certain URL be blocked and went on to indicate which servers (by IP address) contained the offensive content. I'm guessing that Pakistan doesn't send out these directives all that frequently; if they did their ISP's would have gotten it right the first time, the rest of the world wouldn't have noticed and only a few poor souls inside Pakistan would have gone without their daily YouTube. By comparison, think of the Great Firewall of China where such censorship occurs constantly, without affecting anyone external to the Great Wall. Second, the network engineer (we may never learn his or her name) who implemented the policy used the equivalent of a sledgehammer where a pair of tweezers would have been sufficient; my sense is that he or she never intended to provoke an international incident. Third, as we'll see, the upstream provider to this ISP (PCCW) can be held as much to blame for its laziness as the Pakistani engineer might be blamed for a sin of omission (or incompetence). Most alarming of all, it is very likely that your ISP, under the same circumstances, would perform in the very same (inept) way.

If you'd like to see a technical breakdown of the incident, you can find a very good analysis from Danny McPherson, founding member of Arbor Networks and respected contributor to the North American Network Operator's Group (NANOG). To summarize the blunder, for those who are familiar with Internet routing, the ISP in Pakistan chose to comply with the government directive by using an IP routing entry. That routing entry -- a static route in the ISP's router -- was intended to black hole any traffic to or from the YouTube servers destined to or from the ISP's customers. ISP's often enter static (unchanging) routes into ISP routers on behalf of their customers so the mechanism would have been familiar. The key difference is that customer static routes should be advertised (or aggregated and advertised) to the world, black hole routes should not.

The engineer that entered the routes (they first appear on 3/14/08 at 01:00 UTC, RIPE has the history) even saved him or herself a little time; rather than enter three static routes (one for each server) he entered one route, a /24 to cover them all. Ironically, routes with a /24 prefix are the smallest routes that an ISP will accept; had the engineer used routes specific to each server, the routes would never have been accepted by their upstream provider (PCCW), the traffic to and from the servers would be blocked in Pakistan and no one else would be the wiser. Speaking of the upstream ISP in the chain of blunders; PCCW is a tier 1 provider of Internet services. Tier 1 means that PCCW carries a full routing table (all of the Internet routes) inside their routers and don't rely on any other providers for that information. PCCW might easily have stopped the Pakistani ISP from advertising routes that belonged to YouTube. In the first place, the Pakistani ISP normally advertises 4 routes to PCCW, on the morning of 3/14 they were advertising almost 30 routes. ISP's routinely put limits on the number of routes that each customer can advertise but there are easier ways to verify the routing information sent to an ISP by their customers.

Routing registries are databases of routes that an autonomous system (AS) can advertise. A number of organizations (RIPE, RADB, ARIN, and so on) run routing registries. Some of the registries are free, some require membership fees. Larger ISP's can ever run their own registry. To use a routing registry, once an ISP or an enterprise has been assigned IP address space and wants to advertise that space to the Internet, the space is registered. ISP's can use the registry information to create access control lists (ACL's) that limit the information that the ISP will accept from a downstream customer. In the case of the Pakistani ISP, had PCCW entered the four routes that this ISP normally advertised to the Internet in a routing registry, they would have rejected the YouTube routes that caused all of the trouble.

This might sound like a bit of Monday morning quarterback on my part until you realize that routing registries -- the ability for an ISP to certify that their customers are presenting valid routes to the Internet -- have been in use since the mid-1990's. Routing registries require a bit of extra work on the part of the ISP, the routes must be registered and the information in the database turned into routing ACL's (these processes have long been automated for the major routing platforms). There is no defense for an ISP that doesn't verify the routes it learns from customers and smaller ISP's. Yet as they say, those who don't learn from history are doomed to repeat it. In fact, one of the more memorable Internet routing blunders, similar to last weeks blunder from Pakistan, loosed more than 70,000 invalid routes on the Internet through another tier 1 carrier that wasn't using routing registries, UUNet. You can read about that blunder in the NANOG archives; it occurred in 1997.

So what does the future look like for the Internet if routing information isn't verified? The opportunity for denial of service (DoS) is always present. Anyone organization that can inject information into the routing system unchecked can deny service to another organization. Bad publicity and the realization that the denial of service will only continue until an ISP intercedes on behalf of the rightful owner of the address space keeps this type of DoS attack in check. But what if the rightful owner of the address space is in doubt?

Remember that the Internet is running out of IPv4 address space. Our best estimates from ARIN and RIPE indicate that we have between two and four years before these organizations can no longer distribute new IP addresses. Of course, by that time all of our networks will have transitioned to IPv6, right? Probably not. For those networks that are still using IPv4 come 2012, we might imagine a market for IPv4 address space. The creation of a market for address space will certainly bring disputes, so exactly how will ISP's sort out a dispute? What happens if some unscrupulous individual sells address space that has been assigned to another organization? To your organization? What if your address space is part of a block that has been sold? Can your organization survive without an Internet presence until a dispute is resolved?

Having your own information verified against entries in a routing registry won't particularly help your organization, but if enough organizations demand it, the Internet will be a much safer place to do business. Let something good come from the blunder in Pakistan. Do some homework, ask your ISP how they verify routing information from their customers.