Network and Telecom Strategies Blog

Handling Poor Application Performance over the WAN

Posted by: Eric Siegel

It's the end of my writing quarter, and I'm putting the finishing touches on a paper for management about WAN performance optimization (also called WAN Optimization Controller [WOC]) technologies.

That reminded me to make a note that WAN performance optimization isn't always the best solution, although I do think it's great! (See, for example, my previous blog entries "Final Catalyst WAN Performance Optimization Notes" and "Be a Hero! Cut costs to Zero!"; there are others.) (And, by the way, Catalyst is a great place to meet your colleagues! So let's put in an advert for them: Go To Catalyst!)

Here are some alternatives to WAN performance optimization that should always be considered:

• Application redesign or reselection: In some cases, it's better to replace a few poorly-designed applications instead of trying to alter the WAN characteristics. Backup and file transfer or distribution applications that don't remove long duplicate data strings ("deduplication") or that handle transmission errors or congestion inefficiently are prime examples. The Burton Group report “Designing Applications for the WAN” was written to assist programmers in designing applications properly for operation over the WAN, thereby avoiding the need for optimization. 

• Application remoting: Often called "terminal server" or "Citrix," this solution is best for applications that are tightly-intertwined with some remote service; for example, an application in a remote office that makes frequent calls on a database in the enterprise's central server location. Application remoting can also save money on licensing fees and has other advantages. Application Remoting data flows will probably require network QoS.

• System tuning: In some cases (e.g., inability to use all of the bandwidth in a high-latency path), simply tuning existing software or upgrading to more recent version (e.g., shifting to Microsoft Windows Vista from Windows/XP) can produce massive results at minimal cost. 

• WAN service modification: For some situations, the need for more bandwidth or better network delay or error characteristics is unavoidable or is the most cost-effective solution. In some cases, technology changes (e.g., to satellite from terrestrial links) are also involved. Renegotiation of carrier contracts and changing carriers are also options. 

See the Burton Group report "WAN Performance Optimization Quick Start" to learn about WAN performance optimization, and wait for my new Management Initiative document "WAN Performance Optimization" if you have to discuss things with your management. (As always, if you desperately need it now, and you're a Burton Group client, send me an email and I'll give you a pre-publication copy.)

Congestion Going Into Network Branches = Packet Loss

Posted by: Eric Siegel

End of the month, and I've been going over the list of the "dialogues" I had with Burton Group clients to see if there's something that might be of general interest.

So I uncovered one that I participated in with my colleague Ken Agress of Burton Group's consulting organization, and it's interesting because I think that the topic of QoS congestion on links into remote branches isn't given enough consideration in QoS network design.

The basic symptom of the problem was that despite QoS on the routers, there was excessive packet loss on high-priority flows into the branches on a hub-and-spoke network design.

But that's because the server-room router was trying to push too much data into the network cloud for a specific branch office. The discards weren't occurring at the server room boundary router into the cloud, because the fat access link from the server room will easily accommodate lots of traffic to the branch, especially if the other branches aren't receiving much traffic at that time.

If you have a large bandwidth access link into a cloud, but just a small bandwidth access link out of the cloud to a branch office destination, the cloud will need to discard data packets on high-bandwidth flows or during bursts, because the cloud itself (i.e., the backbone routers) has minimal buffering.

True, a single TCP flow will self-adjust (by "self-clocking") to a bandwidth-restricted exit from the cloud because its acknowledgements will be delayed, but if there are a lot of parallel flows, or there are non-flow-controlled UDP flows, you'll be in trouble. TCP flows will get the "global synchronization problem," in which all TCP flows ramp up in bandwidth concurrently, lose packets concurrently, reset their flow rates downwards concurrently, and then repeat. RED or WRED or FRED mechanisms can help avoid global synchronization, but you'll still lose packets.

So, how should this be handled? If you own the cloud or can signal to the cloud's administrator (e.g., by setting DSCP fields or EXP tags on packets going into a MPLS cloud), then the cloud can pick which packets to discard if the exit bandwidth isn't sufficient at the branch office. And the tags will, we hope, tell the cloud to discard packets from low-importance flows. RED / WRED / FRED on those flows will then get those low-importance flows to cut their bandwidth use. Those low-importance flows will be losing packets and will be miserable, but the higher-importance flows will get through.

A better idea, we think, is to avoid letting the cloud do your packet discards -- especially if you don't actually own/administer the cloud directly. Don't give it more traffic than it can handle; use traffic shaping on subinterface definitions to ensure that you never feed more data into the cloud than can exit at the destination. You will probably do a better job of discriminating among flow types and of smoothing brief traffic spikes (e.g., your routers probably have larger buffers than the cloud's routers have).

For example, see the following Cisco documents:

href="http://www.cisco.com/en/US/docs/ios/qos/configuration/guide/reg_pkt_flow_shaping.html" target=_blank>Regulating Packet Flow Using Traffic Shaping

href="http://www.cisco.com/en/US/tech/tk543/tk545/technologies_tech_note09186a0080114326.shtml" target=_blank>Applying QoS Features to Ethernet Subinterfaces

Or you could install a Blue Coat PacketShaper in the server room's exit path to do the traffic shaping for both TCP and UDP. And the new PacketShapers can also do compression.

Security on Internet Substitution Paths

Posted by: Eric Siegel

I recently had a question come up about the security of a path that crosses the public Internet from one enterprise location to another; i.e., an "Internet substitution" path. These types of connections can be much less expensive and faster to provision than a leased line, but auditors may question their security.

Standard encryption devices can be installed at both ends of the communications path to ensure that traffic on that particular path won't be compromised, and if the Internet connection appears to your encryption devices as a simple leased line (i.e., a circuit-switch, not a packet-switch), then there shouldn't be any difference between the security situation for the Internet or for a leased line.

However, if the Internet connection appears at the encryption device's outside interface as a packet-switched, IP-based connection, then there are further considerations:

• First, the encryption devices must positively authenticate each other before opening the encrypted tunnel between them.

  

• Second, no non-tunneled traffic can be allowed to cross the encryption device. It must act as a firewall, absolutely stopping any traffic that isn't a part of the authenticated, encrypted tunnel. (This function could also be performed by the Internet interface device, but it's easier to guarantee the security of the system if it's the encryption device that's performing this function.)

  

• Third, there must be no way that an intruder from the Internet can alter the configuration of the encryption device to subvert the firewall, authentication, or encryption functions. Ideally, the encryption box shouldn't have an Internet-accessible control address and shouldn't be discoverable by scanning. If there is an Internet-accessible control address, it must be very tightly controlled.

(Thanks to Ken Agress, of Burton Group's consulting organization, and to Eric Maiwald of Burton Group's Security and Risk Management Strategies, for assistance on this blog entry!)

Remote Access During A Pandemic?

Posted by: Eric Siegel

(Edited Oct 18, 2009, with probably more edits in the future!)

It's late at night; it's a good time for worrying.

Current worry: pandemics. Swine (H1N1) Flu.

What happens if there's a pandemic, and your staff has to work from home?

First, see the U.S. Government website: http://www.pandemicflu.gov

But second, how are we going to handle the communications problem? If your staff is going to work from home, using their consumer-grade Internet connections (DSL, cable), they may be in for a bad surprise. I worry that everybody is going to have that same idea, and your staff will have to compete for the access paths with millions of people downloading movies and music and YouTube and Facebook. Remember, the first groups of people who will be sent home will almost certainly be schoolchildren, and I'd guess that they'll be heavy users of high-bandwidth applications.

Unless the government or the service providers insist on traffic restrictions; for example, bandwidth-limiting YouTube at the YouTube servers (which might interfere with videos that are important for flu sufferers) or bandwidth-limiting peer-to-peer file sharing, I'm guessing that a lot of oversubscribed consumer-grade Internet access will be a congested mess. A lot of kids (and adults) doing a lot of video-watching and file sharing during the day, combined with a lot of working-from-home parents, may overload the access networks. And it will be a really tempting time for criminals to launch DDoS attacks; service provider staffs will be thin and will be overstretched.

Indeed, a U.S. Department of Homeland Security study published in December 2007 ("Pandemic Influenza Impact on Communications Networks Study") using a pandemic model based on Chicago and a telecommuting exercise by the Federal Reserve Board showed that "In the 40 percent absenteeism pandemic scenario, the telecommuting strategy is expected to be significantly impacted for most telecommuters during the peak of the pandemic," and "In the high absenteeism [90 percent; e.g., local quarantine] scenario, the telecommuting strategy is expected to be unusable for the majority of telecommuters during the peak of the pandemic."

So what should we do?

One option is clearly to have business-class dedicated lines or business-class VSAT to some selected members of the staff.

Another option would be plain old dial-up modem service. It won't work for the beginning of a crisis; there will be too much use of cellular and land telephone lines. But after that, I'd guess that the public switched telephone network will be more stable and less congested and more available than any consumer-grade Internet access. Just dust off those old modem racks in your basement, and be sure you have connectivity. And be sure your users have a modem at home or built in to their laptops. "Application remoting" (e.g., Citrix XenApp or Microsoft Terminal Services) can help provide good performance over dial-up modem links; the new software WAN optimization products (e.g., from Blue Coat, Cisco, Citrix, Expand, Riverbed, Stampede) can also help.

And I have more worries: will your server room's Internet access, firewalls, and VPN equipment have enough capacity to handle all of those incoming Internet and POTS remote users?

That's assuming that you have your server rooms functioning, with some staff available to handle all of the dial-in users and their service desk problems! But that's a different question, for another night's worry. I'll do some more digging on this topic and post more blog entries. And, of course, I'd love your comments!

Final Catalyst WAN Performance Optimization Notes

Posted by: Eric Siegel

OK, two final notes on WAN Performance Optimization from the Catalyst Conference.

(By the way, I'll be rewriting my 90-page, hyper-boring paper "Optimizing WAN Performance" starting in a week or so; it should be published in a couple of months. Ask me if you need a draft copy sooner than that. And post a reply to this blog entry if you have any suggestions for me or for anyone about WAN performance optimization!)

FIRST, be sure to test using realistic user scenarios. Don't just rely on vendor promises or magazine articles or your buddy's experiences or some simple "drag and drop across the Windows desktop" copies! I've been saying this for years, but it really bears repetition.

The classical example is assuming that all vendors handle Microsoft's remote file access protocol, CIFS (or its latest Vista version, SMB 2.0) in the same way. They don't. And there's a difference between string-based compression (e.g., Riverbed) and Wide Area File Services (WAFS)-based compression (e.g., Cisco WAAS and Expand Networks). If you just do a sequential copy (e.g., drag-and-drop across the Windows desktop from a remote file share to your local disk), all of the CIFS systems look good. BUT if you open a file from INSIDE Microsoft Word, Excel, or some other application (is that what your users do?), then you'll see major differences. An application doesn't usually do only sequential reads and writes; it'll do "random" reads and writes within the file, and it'll create temporary files. So the intelligence of the WAN performance optimization solution becomes relevant: it can greatly improve performance when using remote file shares, possibly using the inherent caching ability of CIFS (i.e., the "oplock" capabilities), but it'll also be more complex and vulnerable to possible failure if the programming is sloppy.

SECOND, don't underestimate the impact of Web caching. Even major vendors, such as Microsoft, don't set their cache tags for performance. You can make simply astounding improvements in performance by setting Web cache tags appropriately, and your application front end (AFE) or load balancer can even do that for you. Be sure to include an "Expires" header, or most caches will not cache your pages, or they'll have to check back with the server to check the expiration status of every single object (unless you're using a specialized Web accelerator). See my Burton Group paper "Optimizing WAN Performance" and http://www.mnot.net/cacheability/, http://www.squid-cache.org/, http://www.web-caching.com/, and http://webmaster.info.aol.com/caching.html for more information. (Also look at http://developer.yahoo.com/performance/rules.html, which Nick Fiekowsky recommends, for more caching hints and general performance suggestions for the Web.)

Backup for WAN performance optimizers at remote offices

Posted by: Eric Siegel

(This is the first of a few blog entries that I'm going to post to share some ideas that came out of the Birds of a Feather (BoF) session on WAN performance optimization at Burton Group's Catalyst Conference this week in San Diego.)

A primary advantage of WAN performance optimization appliances is that they cut down the bandwidth needs of a remote branch office -- usually by 50% or more. And they reduce response time to make remote file shares, etc., actually usable instead of being a miserable, irritating experience that destroys productivity.

But what happens when the optimizer breaks? Bandwidth requirements spike upwards, and response time instantly becomes lousy.

My suggestions in the past involved QoS in the branch office boundary router to ensure that critical traffic would still flow despite the worsened bandwidth restrictions, and recommendations that the enterprise make arrangements with vendors or resellers to have spares available in-country (to avoid delays in customs and in transit) and configuration information available on a USB flash drive or by downloading as an email attachment. There would be a day or so of worse-than-normal performance as the compression directories are rebuilt, and then things would be OK.

But Nick Fiekowsky, while sharing his slides at the BoF, had a great idea: use the new software client optimizers, such as Blue Coat's Proxy Client, Cisco's WAAS Mobile, Expand Network's Mobile Accelerator Client, and Riverbed's Steelhead Mobile to serve as emergency backups. These software optimizers run on the users' laptops and provide the function of the optimization appliance. They're not as good, because (except for the new Expand Networks Mobile Accelerator Client! More later...) they don't share their compression directories and caches. But they're a lot better than nothing, and they also provide the ability for the user to use acceleration from home or on the road. This is especially great if the software optimizer system licenses are inexpensive or can be turned on and off easily from the central server location, with billing only when they're used.

My updated recommendation is therefore to install the WAN accelerator vendor's mobile client in branch office desktops. If the office's WAN accelerator fails, the desktop clients will step in to reduce WAN traffic and provide an acceptable user experience. It is necessary to have enough license capacity to cover all the client devices that may become active when a single branch office's WAN accelerator fails.

Be a Hero! Cut costs to Zero!

Posted by: Eric Siegel

Is there anyone who hasn't felt the threat of layoffs this last few months? Everyone is digging foxholes, worrying about the future. That's the natural tendency, I know. (Hey, I'm typing this from the warm security of my desk's kneehole, and my cats have gathered around me, prepared to show their fangs to anyone who might endanger their food supply.)

But if you really want to avoid Trouble, the thing to do is to get out there and save your company money, or set it up to make money quickly! Winning enterprises are using this period to prepare to launch at rocket speed out of the recession, leaving their foxhole-dwelling competitors chewing dirt and breathing rocket exhaust. And there's an easy way to help position your company for that launch, at the exact same time that you save megabucks immediately! It's called WAN Performance Optimization.

Yes, the same drum I've been beating for years, this time with extra added incentives!

I'll make this short, so we can all get back to work. But there isn't much to say.

ONE: WAN Optimization really DOES pay for itself in under a year. (In these times, payback in more than a year doesn't impress management at all. You need instant payback, and then big savings.) Those charts showing 50%, 80% and larger bandwidth savings aren't fake. In a surprising number of cases, the bandwidth savings are immense. So you'll be a hero for saving costs, and your procurement department will be singing your praises!

TWO: Your users will be delirious with joy at the performance improvements, especially if they've been suffering with remote file shares or with the pain of sending huge files back and forth among team members. Because it now takes only minutes, not hours, to send project files back and forth, there will be closer teamwork and they'll win more deals and avoid more screwups. Server centralization will be enabled, too, which is more money saved. So you're a hero for improving productivity and morale, and your HR and operations departments and your branch office managers and project managers will be singing your praises!

Just remember to forward all of those love notes to your management; this is not the time to be shy.

The technology can be somewhat complex, so be sure to find some reference accounts with a similar architecture and network design to yours. The vendors will love to show you around the technology, and they'll even loan you equipment to try it out. Different vendors specialize in different optimization situations; it's a good idea to look around. Some leading vendors are:

• Blue Coat
• Certeon
• Cisco (WAAS and AVS)
• Citrix (WanScaler and NetScaler)
• Exinda
• Expand Networks
• F5 (WANJet and WebAccelerator)
• Juniper (WX)
• Riverbed
• Silver Peak
• Stampede

See my previous blog entry, "WAN Optimization Whitelists and Blacklists"  for more information on this topic. Burton Group clients should look at my 80-single-spaced-page hyper-boring "Optimizing WAN Performance" report. And in a few weeks a new report, "Evaluating and Deploying WAN Performance Optimization," will be out in print. (Ask me for a pre-production copy.) It talks about how to pitch WAN optimization to management, how to select and evaluate contenders, and how to go through a rollout without crises. Or go to Burton Group's Catalyst Conference, July 27-31 in San Diego, where we're going to have tutorials and sessions on this, and a Birds-of-a-Feather meeting. Or call me for a free "dialogue." Gee, there are lots of ways to get started; no excuses here!

The Latest Communications Link Failure

Posted by: Eric Siegel

OK, yesterday there was a major disruption of the communications links in the San Francisco Bay Area. Ten fiber optic cables, belonging to AT&T and to Sprint, were cut at four separate locations by saboteurs. (See, for example, The San Francisco Chronicle, "Sabotage attacks knock out phone service" August 10, 2009,and The Mercury News, "Phone service fully restored; AT&T offers 100K reward" August 10, 2009)

All Hell broke loose; telephones were down, ATMs stopped working; 911 calls went nowhere.

There were the usual articles talking about how this should "remind" everyone about link failures, et cetera, and about how everyone was so surprised at this.

There really shouldn't be any surprise. THIS is a major reason for having a truly remote data center! In any geographic area, there are only a few paths for communications links. They're typically next to railroads or other utilities that have a long, narrow right-of-way. It's just too difficult to negotiate with hundreds of landowners for a new right-of-way for hundreds of miles. The result is that most communications links, even from different carriers, end up in the same manholes and tunnels. (Remember the July 2001 CSX train derailment in a Baltimore railroad tunnel? It started a fire and melted the fiber optic cables in that tunnel, causing major problems for at least five ISPs. Or how about the December 2008 cut of a major undersea cable in the Mediterranean, one of many cuts, often caused by boat anchors? [See WIRED.com "Undersea Cables Cut; 14 Countries Lose Web -- Updated" December 19, 2008 ] There are hundreds of examples.)

The rules are:

If you have "redundant" access links to the service provider, using the same or different carriers, you must inquire very carefully about the routing of those links. They will usually share facilities at some point! Be sure each access link arrives at a major service provider switching node, with many physically divergent routes from that point, before it encounters any shared facilities with any other access link. Physical diversity will guard against a single physical event killing both access links into the massive service provider meshes.

Strongly consider using alternative media, such as satellite or radio links (see the Burton Group reports "Fixed Broadband Wireless Access Technology: Unwiring the Last Mile" and "The Wild World of WISPs" ), to connect to a service provider who will be outside the radius of any probable disruptions.

Ensure that your "backup" link is actually working. Share traffic across it, or test it very frequently.

And remember that loss of communications is a major reason for having a redundant data center!

Please TEST before inflicting your app on the network!

Posted by: Eric Siegel

Here's another pet peeve of mine: lack of realistic pre-deployment testing.

How many development organizations use zero-latency infinite-bandwidth LANs and top-of-the-line switches in their development shop, then, when their tightly-coupled client-server application goes into production, they blame lousy performance on the network? As the saying goes, "If I had a nickel....." Well, I'd be writing this blog entry from a penthouse suite on Fifth Avenue.

A leading cause of friction between network operations staff and development staff could be removed if all applications were realistically tested before being placed into production. Real production network environments differ greatly from typical development environments; they include high-latency WAN and cellular links, bandwidth restrictions, congestion, link errors, packet duplication and misordering, and packet fragmentation caused by devices in the communications path, encryption, and virtual private network (VPN) connections. Bandwidth optimization appliances such as caches, compression devices, and protocol-acceleration devices also interact with data flows. All of these affect application performance.

If the application is accessible from the public web, designers must also consider and handle the possibility of a flash load. Flash loads occur when overwhelming numbers of users try to connect to a web server simultaneously, possibly in response to a news event or a television advertisement. The flash load inevitably leads to widespread abandonment of slow transactions, causing resource recovery problems (transaction abandonment is difficult to detect in web environments) and probably further degrading performance. Public web applications must also consider the universe of possible browser types and the fact that a small percentage of users disable cookies on their browsers.

WAN emulation and load generation devices and services can reproduce all of these problems, and it is much less stressful for everybody if problems are discovered before the application is put into production. Surprising things happen to data flows on production WANs: Always test before deployment.

WAN Optimization Whitelists and Blacklists

Posted by: Eric Siegel

I've been working on a survey of best practices for acquiring WAN optimization solutions; it'll appear as a Burton Group paper and also be used as the basis for a talk at the upcoming Burton Group Catalyst Conference on July 27-31 in San Diego. It's been fun to talk to folks about what worked and what didn't work in their efforts to bring WAN optimization technology into their enterprises.

The papers and speech won't be out for another month or so, but meanwhile there's a very important point about this technology that I want to get out here: Yes, optimization is a fantastic way of saving money and creating really happy customers at the same time, but it doesn't work flawlessly for all applications.

The advanced dictionary-based compression component seems to be foolproof; it works and saves you, oh, 50% and up of your bandwidth at the same time that it usually makes a major improvement in performance that'll have the users loving you. So networking organizations that are primarily concerned with saving bandwidth expense and that dread having to work with users or application developers will usually choose optimization devices that are almost purely compression-based, and that don't fiddle much, or at all, with the protocols. They'll get a lot of performance improvement also -- especially if the same files are sent back-and-forth across the network with minor changes, or if they contain internal duplications, or if there's a lot of commonality among files -- and they'll never break an application, no matter how badly it's programmed.

But some enterprises really need the massive performance improvements that WAN optimization can provide for some common situations, such as access to remote fileshares using the Microsoft CIFS or Unix NFS protocols, or applications using SQL queries, or web browsers, or email. For those situations, fiddling with ("accelerating" or "spoofing") the underlying TCP, file, and application protocols can create simply unbelievable improvements in the performance seen by end users. But when the WAN optimization device starts interacting with protocols, some applications will break. It's almost never the optimization device's fault, because the optimizer engineers are experts in the protocols and carefully design their devices to squeeze every bit out of them without violating the protocol standards. But a surprising number of enterprise programmers don't know those standards very well.

The result is that protocol acceleration typically works flawlessly for broadly-used applications, such as access to remote fileshares by Excel, or use of web browsers, or SharePoint. But weird, custom-built client-server applications with embedded TCP socket calls that were built by overworked programmers under horrible deadline pressure who have no previous experience with TCP or other protocols? Well, that explains the "your accelerator broke my application" telephone calls. Truth is, the application was already broken, but its failures were usually rather rare and were attributed to "cosmic rays" or similar nonsense. When the optimizer starts really exercising the protocol, however, the breaks in the protocol become more obvious, and the irate telephone calls start.

And that brings me to the point of this blog entry: most enterprises quickly find that whitelists are wonderful tools for managing WAN optimization. They use whitelists to tell the optimizers which applications should be given protocol acceleration in addition to advanced compression. Applications that aren't on the whitelist are compressed, but their protocols are passed through the optimzers unchanged. At initial rollout, the network organization puts broadly-used applications, such as Excel, Internet Explorer and SharePoint, on the whitelist. Then, as time goes by, additional applications can be tested and added to the whitelist. By using whitelists, all of those weird applications that you didn't even know existed will not get into trouble. (By the way, organizations that try blacklists usually find that they have dozens, or hundreds, of applications that they didn't know about and that have mistakes in their use of TCP or other protocols. They find that out as a result of enraged telephone calls from the applications' users. Use whitelists instead.)

Some WAN optimization vendors have extensive deep packet inspection systems that can help in creating whitelists, but that's not truly necessary in all cases. Just filtering by IP address, hostname, or similar simple mechanisms may be all that you need.

So: don't fear optimization! You can save buckets of money and make your customers happy at the same time! But use whitelists when you do that rollout.