Network and Telecom Strategies Blog

SNMP Security

Posted by: Eric Siegel

A question recently arose about best practices for SNMP security, so I feel that a quick blog entry about it would be useful.

Here is the simple rule: if you're running SNMP version 1 or SNMP version 2c (the current variant of SNMP version 2; the original version was abandoned because of a flawed security system), then there isn't any security. So don't allow anyone to write to your devices using SNMP, period. Completely disable all facilities on the devices that respond to SNMP write (SET) commands. If you have to write to the devices, then use a secure pathway, such as secure shell (SSH) or a secure XML flow, thereby avoiding SNMP entirely for writing.

For reading/GET functions in SNMPv1 and v2, I'd just give up and allow unencrypted community strings and data, but I still might secure the aggregated data in collectors or a central management platform. I'd use TCP and encryption for all communications between management platforms and between management platforms and SNMP collectors/pollers. I'd disallow any use of SNMP to sniff traffic, because that might reveal user passwords, etc. (And such sniffing is inefficient and awkward anyway!) I'd also see if I could restrict the data sent by sensitive devices (e.g., firewalls) in response to GET commands; e.g., I'd might not let them send their version numbers or configuration data.

An intruder could do a denial-of-service attack by issuing an overwhelming number of read/GET commands to an SNMPv1 or v2 device under these rules, so maybe you should require that devices have a way to throttle incoming SNMP requests to ensure that they aren't overwhelmed. (Many devices do this by ensuring that SNMP processes run at a relatively low priority.)

Having devices and the management agent check IP addresses and hostnames is not a guarantee against intruders, although it makes intrusion more difficult. Therefore, to decrease the probability of an attack on managed devices, the managed devices should be configured to accept commands only from sources in designated subnets (e.g., a management subnet) or from designated IP addresses or address ranges.

And, of course, you must ensure that devices do not include "hidden" community strings (i.e., strings that are embedded in the device, are typically used by the manufacturer, and are usually undocumented and not revealed to the enterprise purchaser of the equipment); these can create severe vulnerabilities.

You should have firewalls block all SNMP commands that are incoming to protected areas. Use remote collectors/pollers to gather SNMP data and push it through the firewall on a secure TCP flow if you have to go through a firewall.

SNMP version 3 is the most secure, robust solution. Unfortunately, its security features use a new security model that was designed to be independent of other security infrastructures. That helped ensure that SNMPv3 could function in a broken network, but the complexity has made many organizations reluctant to deploy SNMPv3. The IETF's isms Working Group is now trying to create an SNMPv3 security model based on SSH over TCP. When it becomes commercially available, the Integrated Security Model for SNMP will be the recommended security solution for SNMPv3.

Remote Access During A Pandemic?

Posted by: Eric Siegel

(Edited Oct 18, 2009, with probably more edits in the future!)

It's late at night; it's a good time for worrying.

Current worry: pandemics. Swine (H1N1) Flu.

What happens if there's a pandemic, and your staff has to work from home?

First, see the U.S. Government website: http://www.pandemicflu.gov

But second, how are we going to handle the communications problem? If your staff is going to work from home, using their consumer-grade Internet connections (DSL, cable), they may be in for a bad surprise. I worry that everybody is going to have that same idea, and your staff will have to compete for the access paths with millions of people downloading movies and music and YouTube and Facebook. Remember, the first groups of people who will be sent home will almost certainly be schoolchildren, and I'd guess that they'll be heavy users of high-bandwidth applications.

Unless the government or the service providers insist on traffic restrictions; for example, bandwidth-limiting YouTube at the YouTube servers (which might interfere with videos that are important for flu sufferers) or bandwidth-limiting peer-to-peer file sharing, I'm guessing that a lot of oversubscribed consumer-grade Internet access will be a congested mess. A lot of kids (and adults) doing a lot of video-watching and file sharing during the day, combined with a lot of working-from-home parents, may overload the access networks. And it will be a really tempting time for criminals to launch DDoS attacks; service provider staffs will be thin and will be overstretched.

Indeed, a U.S. Department of Homeland Security study published in December 2007 ("Pandemic Influenza Impact on Communications Networks Study") using a pandemic model based on Chicago and a telecommuting exercise by the Federal Reserve Board showed that "In the 40 percent absenteeism pandemic scenario, the telecommuting strategy is expected to be significantly impacted for most telecommuters during the peak of the pandemic," and "In the high absenteeism [90 percent; e.g., local quarantine] scenario, the telecommuting strategy is expected to be unusable for the majority of telecommuters during the peak of the pandemic."

So what should we do?

One option is clearly to have business-class dedicated lines or business-class VSAT to some selected members of the staff.

Another option would be plain old dial-up modem service. It won't work for the beginning of a crisis; there will be too much use of cellular and land telephone lines. But after that, I'd guess that the public switched telephone network will be more stable and less congested and more available than any consumer-grade Internet access. Just dust off those old modem racks in your basement, and be sure you have connectivity. And be sure your users have a modem at home or built in to their laptops. "Application remoting" (e.g., Citrix XenApp or Microsoft Terminal Services) can help provide good performance over dial-up modem links; the new software WAN optimization products (e.g., from Blue Coat, Cisco, Citrix, Expand, Riverbed, Stampede) can also help.

And I have more worries: will your server room's Internet access, firewalls, and VPN equipment have enough capacity to handle all of those incoming Internet and POTS remote users?

That's assuming that you have your server rooms functioning, with some staff available to handle all of the dial-in users and their service desk problems! But that's a different question, for another night's worry. I'll do some more digging on this topic and post more blog entries. And, of course, I'd love your comments!

Final Catalyst WAN Performance Optimization Notes

Posted by: Eric Siegel

OK, two final notes on WAN Performance Optimization from the Catalyst Conference.

(By the way, I'll be rewriting my 90-page, hyper-boring paper "Optimizing WAN Performance" starting in a week or so; it should be published in a couple of months. Ask me if you need a draft copy sooner than that. And post a reply to this blog entry if you have any suggestions for me or for anyone about WAN performance optimization!)

FIRST, be sure to test using realistic user scenarios. Don't just rely on vendor promises or magazine articles or your buddy's experiences or some simple "drag and drop across the Windows desktop" copies! I've been saying this for years, but it really bears repetition.

The classical example is assuming that all vendors handle Microsoft's remote file access protocol, CIFS (or its latest Vista version, SMB 2.0) in the same way. They don't. And there's a difference between string-based compression (e.g., Riverbed) and Wide Area File Services (WAFS)-based compression (e.g., Cisco WAAS and Expand Networks). If you just do a sequential copy (e.g., drag-and-drop across the Windows desktop from a remote file share to your local disk), all of the CIFS systems look good. BUT if you open a file from INSIDE Microsoft Word, Excel, or some other application (is that what your users do?), then you'll see major differences. An application doesn't usually do only sequential reads and writes; it'll do "random" reads and writes within the file, and it'll create temporary files. So the intelligence of the WAN performance optimization solution becomes relevant: it can greatly improve performance when using remote file shares, possibly using the inherent caching ability of CIFS (i.e., the "oplock" capabilities), but it'll also be more complex and vulnerable to possible failure if the programming is sloppy.

SECOND, don't underestimate the impact of Web caching. Even major vendors, such as Microsoft, don't set their cache tags for performance. You can make simply astounding improvements in performance by setting Web cache tags appropriately, and your application front end (AFE) or load balancer can even do that for you. Be sure to include an "Expires" header, or most caches will not cache your pages, or they'll have to check back with the server to check the expiration status of every single object (unless you're using a specialized Web accelerator). See my Burton Group paper "Optimizing WAN Performance" and http://www.mnot.net/cacheability/, http://www.squid-cache.org/, http://www.web-caching.com/, and http://webmaster.info.aol.com/caching.html for more information. (Also look at http://developer.yahoo.com/performance/rules.html, which Nick Fiekowsky recommends, for more caching hints and general performance suggestions for the Web.)

Building Applications for the WAN

Posted by: Eric Siegel

Our final session at Catalyst this year was a joint presentation between Network and Telecom Strategies and Richard Watson of Burton Group's Application Platform Strategies group. We talked about how programmers and network engineers can work together to improve application performance and problem diagnosis.

This session was based on the NTS report "Designing Applications for the WAN," which will be updated next week -- so go read it when the update comes out (it's out now!), but there are a two points that came out in the session that I think were rather surprising to us network folks and that should be emphasized here.

First, the applications developers are now using frameworks to enable them to develop applications more quickly. Those application frameworks completely hide ("abstract") the network. For example, an application developer will ask a "method" for a "data object." The developer doesn't even realize that the network is involved, or that there may be some SQL queries used by that method to create the data object. That's why when you have a crisis, and you put Wireshark or a Sniffer on the communications link and show the resulting trace to the developer, he or she has no idea what the heck is going on. I'll say it again: the developer didn't even realize an SQL query was generated by the method request, and has no idea whatsoever what that stuff in the query string, trapped by the trace, really means.

No wonder there's frustration and wasted time in diagnosis! The time has arrived for the network organization, the operations group, and the development organization to cooperate and purchase a diagnostic tool that will trace individual transactions through the production system for diagnosis. (An example is Quest Software's PerformaSure; there are others.) The programmers will understand what they're seeing, the network staff won't need to run around doing Wireshark or Sniffer traces for days on end, and the problem will be handled much more quickly.

And, as the second point, realize that performance is poor because of the "abstraction" of the network layer. The application framework has a lot of parameters that set timeouts, window sizes, buffer sizes, compression, whether to disable the Nagle Algorithm, etc. The programmers understand none of this, so they use the defaults. But the defaults aren't usually set by the vendor to ensure optimum performance; they're set to create the fewest trouble calls to the vendor regardless of the network characteristics. That may not be the optimum setting for your particular situation, and that's why the people who install and set up the framework should talk to the network staff to jointly decide how to set the networking parameters for the networks that the framework will be running over. That'll help quite a bit in creating good performance.

TCP stack Performance Tuning for Windows

Posted by: Eric Siegel

I held a Birds of a Feather (BoF) session on WAN performance optimization at Burton Group's Catalyst Conference this week in San Diego, and we had approximately 14 attendees, which was quite good. (Attendance at Catalyst is almost exactly the same as last year: approximately 1500 people! We're VERY happy.)

We started the BoF with a talk by Nick Fiekowsky of Wyeth; he also has a talk coming up on WAN optimization, but the BoF was a great opportunity for us to have a more unstructured discussion (it lasted over an hour) about some of the detailed technical issues.

Nick made some great new observations about TCP performance on Windows; they're very interesting and also seem to have a large impact on performance. So here, with his permission, are some of them:

First, as Nick says, "Latency doesn't end at the RJ-45." When you're computing the required buffer size (receive window, RWIN) for long fat networks (i.e., large bandwidth and large latency), we always think of just the latency on the path and the bandwidth. I hadn't previously thought about the time that the received packet spends inside the receiving station's TCP buffers, waiting to be processed and then waiting to be read by the application. But that's an important factor in the latency; after all, the RWIN can open until there's space in the receive buffer, and the receive buffer won't have space until the application has read something out of it!

So Nick looked at what actually happened, and he discovered that although the bandwidth delay product (and therefore the receive buffer size) was approximately half a megabyte for a particular path, there were still long pauses waiting for acknowledgements to be returned to the sender -- throughput was lower than it could be. He had to increase the receive buffer size to almost THREE MEGABYTES before the sniffer (actually, Wireshark) trace showed that the bandwidth was being fully utilized. Wow.

And that led to Nick's other major point: it's important to optimize Windows performance to keep it from stalling while processing those incoming packets:

• "Non-Paged Pool" (NPP), where the buffers are located, is a precious resource. Nick says that in a 32-bit Windows OS, the NPP = RAM/16. That's not very large, especially if you need large receive buffers. (Look at the Task Manager to see how much of the NPP is being used. Or you can just notice that the TCP stack is requesting a small RWIN when a new connection comes in, because it's under memory pressure.) The only way to increase that is to increase the total RAM in the processor. In the 64-bit Windows OS, the NPP allocation is handled much better, TCP auto-tuning in Vista, Windows 7 and Windows Server 2008 makes better use of NPP, and finally the 64-bit OS also makes the entire system run better because system processes are faster. So the 64-bit OS is a big improvement here.
• Defrag the hard drive to be sure that fragmentation isn't slowing down the processes and the stack.
• Use updated NIC drivers. Nick says that recently he's noticed that there's a new emphasis on performance in the drivers, possibly being driven by iSCSI and datacenter requirements. Also increase the number of transmit and receive descriptors in the driver configuration.

Nick had a lot of other recommendations, such as the big improvement made by ensuring that Web cache expiration tags are properly used, and I'll try to get to them in a later blog.

Meanwhile, thanks, Nick!

Backup for WAN performance optimizers at remote offices

Posted by: Eric Siegel

(This is the first of a few blog entries that I'm going to post to share some ideas that came out of the Birds of a Feather (BoF) session on WAN performance optimization at Burton Group's Catalyst Conference this week in San Diego.)

A primary advantage of WAN performance optimization appliances is that they cut down the bandwidth needs of a remote branch office -- usually by 50% or more. And they reduce response time to make remote file shares, etc., actually usable instead of being a miserable, irritating experience that destroys productivity.

But what happens when the optimizer breaks? Bandwidth requirements spike upwards, and response time instantly becomes lousy.

My suggestions in the past involved QoS in the branch office boundary router to ensure that critical traffic would still flow despite the worsened bandwidth restrictions, and recommendations that the enterprise make arrangements with vendors or resellers to have spares available in-country (to avoid delays in customs and in transit) and configuration information available on a USB flash drive or by downloading as an email attachment. There would be a day or so of worse-than-normal performance as the compression directories are rebuilt, and then things would be OK.

But Nick Fiekowsky, while sharing his slides at the BoF, had a great idea: use the new software client optimizers, such as Blue Coat's Proxy Client, Cisco's WAAS Mobile, Expand Network's Mobile Accelerator Client, and Riverbed's Steelhead Mobile to serve as emergency backups. These software optimizers run on the users' laptops and provide the function of the optimization appliance. They're not as good, because (except for the new Expand Networks Mobile Accelerator Client! More later...) they don't share their compression directories and caches. But they're a lot better than nothing, and they also provide the ability for the user to use acceleration from home or on the road. This is especially great if the software optimizer system licenses are inexpensive or can be turned on and off easily from the central server location, with billing only when they're used.

My updated recommendation is therefore to install the WAN accelerator vendor's mobile client in branch office desktops. If the office's WAN accelerator fails, the desktop clients will step in to reduce WAN traffic and provide an acceptable user experience. It is necessary to have enough license capacity to cover all the client devices that may become active when a single branch office's WAN accelerator fails.

XML Compression

Posted by: Eric Siegel

XML compression came up recently in one of my conversations with a client. It's an interesting topic that a lot of us will need to deal with as XML replaces other forms of process-to-process messaging.

XML is inefficient in its use of bandwidth, a tradeoff that was made to gain ease of use and generality. However, that need for a lot of bandwidth can be a problem, especially on paths with restricted bandwidth or high error rates, such as mobile wireless communications.

Some vendors, such as Stampede and Accelenet (ViaSat/ICT), provide advanced compression solutions that can be extremely effective -- and they also improve the transport protocol, which can help on wireless links. These are quite different from classical "zip" compression. They're proprietary and there are no "standards" involved; you can't pair up compression from two different vendors and expect them to interoperate. However, in exchange for the proprietary nature of these things, you get great compression. These systems maintain compression dictionaries at both end of the path, substituting short pointers for data strings. You can also pre-load these dictionaries with XML phrases; e.g., Stampede's Web 2.0 Performance Series has special features for SOA, including pre-loading of XML compression dictionaries, persistent TCP connections despite the tendency of AJAX (asynchronous Java and XML) to disconnect and reconnect frequently, and extensive security that includes XML schema validation, message anomaly detection, and denial of service attack detection. Indeed, most WAN performance optimization vendors provide advanced compression solutions that can assist with XML.

An alternative is to use binary XML, such as the Efficient XML Interchange format (EXI), which is being specified and evaluated by the Efficient XML Interchange (EXI) Working Group of W3C and is available from vendors such as AgileDelta. AgileDelta says that they provide software that supports standard XML APIs and they can also provide HTTP and TCP Socket proxies to convert XML to Efficient XML (EXI) without any change whatsoever to the application. The software could be used for applications that you can get the vendor to write with EXI; the proxies could be used to convert EXI to standard XML for applications that you don't have control over.

A big question, of course, is whether EXI will become broadly used. Although the EXI standard hasn't been formally approved, it's just one step away from that approval (it's in "Last Call"). The wikipedia article on binary XML has some discussion of these standards -- both EXI and others.

Note, of course, that any encryption or non-standard compression (e.g., proprietary compression, or possibly, EXI) can interfere with network devices that are trying to do "deep packet inspection" of data flows to detect policy violations and attacks, or to route data flows depending on flow contents. And any flow conversion system (compression or protocol acceleration) must usually intercept both flow directions; that can be tricky if flows travel asymmetrically. So use of any encryption, interception, or compression device means you have to think about precisely where you're going to place that device.

My colleague Chris Haddad also says that edge CPU processing speed for XML is important; it can be a problem that binary XML can help. An article referenced by the wikipedia binary XML entry is interesting here: "i-Technology Viewpoint: The Performance Woe of Binary XML" by Jimmy Zhang in SOA World Magazine, August 5, 2008.

QoS Problems With VoIP Over Wireless

Posted by: Eric Siegel

A fascinating article, "Coexistence of VoIP and TCP in Wireless Multihop Networks," was just published in the June 2009 edition of IEEE Communications Magazine (vol 47, issue 6, pp 75-81; the authors are Kyungtae Kim and Dragos Niculescu [NEC Laboratories America] and Sangjin Hong [Stony Brook University - SUNY]).

Burton Group has been warning that using VoIP over wireless LAN (VoWLAN) is tricky because "QoS alone cannot completely regulate the bandwidth provided to flows on a wireless network... a single wireless station on a shared wireless channel can degrade performance for all other stations on the channel...." We've been saying that: "A reasonable alternative is to avoid voice over WLAN altogether, substituting cellular voice, possibly with femtocell or in-building repeater technology." (See the Burton Group reports "Quality of Service (QoS) Alternatives Demystifying Radio Management;along with the Technical Position "Quality of Service".)

This new article shows that the problems created by VoIP over wireless LAN are even worse than I'd thought. The authors did some experiments, and they found that TCP and VoIP do not coexist well on a wireless LAN. TCP's large frame sizes, large receive windows (RWIN), bursty traffic, and retransmission when a frame is lost make it interfere with short, evenly-spaced VoIP frames. The TCP frames will dominate radio transmissions because of the frame lengths and frequency during a burst. (Frames can't be interrupted after they start to transmit; therefore, long, frequent TCP frames will dominate over infrequent, short VoIP frames.)

Worse, the self-interference and half-duplex characteristics of a wireless LAN system increase the probability that the long TCP data bursts will need retransmission, further increasing their burstiness and their domination of the channel when compared to short VoIP frames. (Self-interference occurs when a single flow interferes with itself as it travels over the wireless system. When transmission and reception share the same half-duplex radio channel, a stream may compete with its own acknowledgements for radio time, delaying or destroying some data and acknowledgement packets. If transmission hops over a couple of radio links in sequence, then each hop's transmission may interfere with transmissions on other hops that are within reception range; a flow may interfere with itself.)

The authors of the new article show that no standard type of queuing algorithm in routers adjacent to the wireless network, even Priority Queuing (PQ), can handle these problems. And they also show that five standard TCP variants used in end stations (TCP Reno, TCP Vegas, TCP Westwood, TCP CUBIC [used in Linux], and C-TCP [used in Windows Vista and Server 2008]) do not handle these problems. In all cases, the TCP flows dominate and degrade the VoIP flows.

What should be done? Well, Burton Group has recommended that cellular should be considered instead of VoWLAN. The authors of the study make additional suggestions that sound interesting, but they also warn that none of them is a panacea - they all are only partial solutions, at best. The authors warn that "the methods examined here have straight-forward application only when the wireless capacity is fixed. In reality, the capacity is highly variable." They also warn that P2P, uploading traffic, and interactive streaming, such as Skype, that use TCP for voice will complicate the situation even more.

The paper's most interesting recommendations are:

• Use unusually-small TCP receive windows and eliminate TCP bursts on the wireless LAN. This may need to be done by gateways at the wired:wireless interface. The gateway would use smaller TCP packets and a small receive window; it would also use a buffer to ensure that any bursts were smoothed out and that TCP packets were spaced out to reduce self-interference and interference with VoIP frames.
• Use a traffic shaping system. The article says, "After a packet of size pkt_size goes out over the air, the next packet is scheduled no earlier than pkt_size/R. The gateway chooses R based on the network status to determine how much to send as well as when to send."

Other papers have recommended different methods, such as doing priority queuing at all intermediate wireless nodes and combining that with traffic shaping.

At this time, it seems to me that avoiding sharing the same channel between TCP and voice is still the best solution.

Be a Hero! Cut costs to Zero!

Posted by: Eric Siegel

Is there anyone who hasn't felt the threat of layoffs this last few months? Everyone is digging foxholes, worrying about the future. That's the natural tendency, I know. (Hey, I'm typing this from the warm security of my desk's kneehole, and my cats have gathered around me, prepared to show their fangs to anyone who might endanger their food supply.)

But if you really want to avoid Trouble, the thing to do is to get out there and save your company money, or set it up to make money quickly! Winning enterprises are using this period to prepare to launch at rocket speed out of the recession, leaving their foxhole-dwelling competitors chewing dirt and breathing rocket exhaust. And there's an easy way to help position your company for that launch, at the exact same time that you save megabucks immediately! It's called WAN Performance Optimization.

Yes, the same drum I've been beating for years, this time with extra added incentives!

I'll make this short, so we can all get back to work. But there isn't much to say.

ONE: WAN Optimization really DOES pay for itself in under a year. (In these times, payback in more than a year doesn't impress management at all. You need instant payback, and then big savings.) Those charts showing 50%, 80% and larger bandwidth savings aren't fake. In a surprising number of cases, the bandwidth savings are immense. So you'll be a hero for saving costs, and your procurement department will be singing your praises!

TWO: Your users will be delirious with joy at the performance improvements, especially if they've been suffering with remote file shares or with the pain of sending huge files back and forth among team members. Because it now takes only minutes, not hours, to send project files back and forth, there will be closer teamwork and they'll win more deals and avoid more screwups. Server centralization will be enabled, too, which is more money saved. So you're a hero for improving productivity and morale, and your HR and operations departments and your branch office managers and project managers will be singing your praises!

Just remember to forward all of those love notes to your management; this is not the time to be shy.

The technology can be somewhat complex, so be sure to find some reference accounts with a similar architecture and network design to yours. The vendors will love to show you around the technology, and they'll even loan you equipment to try it out. Different vendors specialize in different optimization situations; it's a good idea to look around. Some leading vendors are:

• Blue Coat
• Certeon
• Cisco (WAAS and AVS)
• Citrix (WanScaler and NetScaler)
• Exinda
• Expand Networks
• F5 (WANJet and WebAccelerator)
• Juniper (WX)
• Riverbed
• Silver Peak
• Stampede

See my previous blog entry, "WAN Optimization Whitelists and Blacklists"  for more information on this topic. Burton Group clients should look at my 80-single-spaced-page hyper-boring "Optimizing WAN Performance" report. And in a few weeks a new report, "Evaluating and Deploying WAN Performance Optimization," will be out in print. (Ask me for a pre-production copy.) It talks about how to pitch WAN optimization to management, how to select and evaluate contenders, and how to go through a rollout without crises. Or go to Burton Group's Catalyst Conference, July 27-31 in San Diego, where we're going to have tutorials and sessions on this, and a Birds-of-a-Feather meeting. Or call me for a free "dialogue." Gee, there are lots of ways to get started; no excuses here!

The Latest Communications Link Failure

Posted by: Eric Siegel

OK, yesterday there was a major disruption of the communications links in the San Francisco Bay Area. Ten fiber optic cables, belonging to AT&T and to Sprint, were cut at four separate locations by saboteurs. (See, for example, The San Francisco Chronicle, "Sabotage attacks knock out phone service" August 10, 2009,and The Mercury News, "Phone service fully restored; AT&T offers 100K reward" August 10, 2009)

All Hell broke loose; telephones were down, ATMs stopped working; 911 calls went nowhere.

There were the usual articles talking about how this should "remind" everyone about link failures, et cetera, and about how everyone was so surprised at this.

There really shouldn't be any surprise. THIS is a major reason for having a truly remote data center! In any geographic area, there are only a few paths for communications links. They're typically next to railroads or other utilities that have a long, narrow right-of-way. It's just too difficult to negotiate with hundreds of landowners for a new right-of-way for hundreds of miles. The result is that most communications links, even from different carriers, end up in the same manholes and tunnels. (Remember the July 2001 CSX train derailment in a Baltimore railroad tunnel? It started a fire and melted the fiber optic cables in that tunnel, causing major problems for at least five ISPs. Or how about the December 2008 cut of a major undersea cable in the Mediterranean, one of many cuts, often caused by boat anchors? [See WIRED.com "Undersea Cables Cut; 14 Countries Lose Web -- Updated" December 19, 2008 ] There are hundreds of examples.)

The rules are:

If you have "redundant" access links to the service provider, using the same or different carriers, you must inquire very carefully about the routing of those links. They will usually share facilities at some point! Be sure each access link arrives at a major service provider switching node, with many physically divergent routes from that point, before it encounters any shared facilities with any other access link. Physical diversity will guard against a single physical event killing both access links into the massive service provider meshes.

Strongly consider using alternative media, such as satellite or radio links (see the Burton Group reports "Fixed Broadband Wireless Access Technology: Unwiring the Last Mile" and "The Wild World of WISPs" ), to connect to a service provider who will be outside the radius of any probable disruptions.

Ensure that your "backup" link is actually working. Share traffic across it, or test it very frequently.

And remember that loss of communications is a major reason for having a redundant data center!