Network and Telecom Strategies Blog

Final Catalyst WAN Performance Optimization Notes

Posted by: Eric Siegel

OK, two final notes on WAN Performance Optimization from the Catalyst Conference.

(By the way, I'll be rewriting my 90-page, hyper-boring paper "Optimizing WAN Performance" starting in a week or so; it should be published in a couple of months. Ask me if you need a draft copy sooner than that. And post a reply to this blog entry if you have any suggestions for me or for anyone about WAN performance optimization!)

FIRST, be sure to test using realistic user scenarios. Don't just rely on vendor promises or magazine articles or your buddy's experiences or some simple "drag and drop across the Windows desktop" copies! I've been saying this for years, but it really bears repetition.

The classical example is assuming that all vendors handle Microsoft's remote file access protocol, CIFS (or its latest Vista version, SMB 2.0) in the same way. They don't. And there's a difference between string-based compression (e.g., Riverbed) and Wide Area File Services (WAFS)-based compression (e.g., Cisco WAAS and Expand Networks). If you just do a sequential copy (e.g., drag-and-drop across the Windows desktop from a remote file share to your local disk), all of the CIFS systems look good. BUT if you open a file from INSIDE Microsoft Word, Excel, or some other application (is that what your users do?), then you'll see major differences. An application doesn't usually do only sequential reads and writes; it'll do "random" reads and writes within the file, and it'll create temporary files. So the intelligence of the WAN performance optimization solution becomes relevant: it can greatly improve performance when using remote file shares, possibly using the inherent caching ability of CIFS (i.e., the "oplock" capabilities), but it'll also be more complex and vulnerable to possible failure if the programming is sloppy.

SECOND, don't underestimate the impact of Web caching. Even major vendors, such as Microsoft, don't set their cache tags for performance. You can make simply astounding improvements in performance by setting Web cache tags appropriately, and your application front end (AFE) or load balancer can even do that for you. Be sure to include an "Expires" header, or most caches will not cache your pages, or they'll have to check back with the server to check the expiration status of every single object (unless you're using a specialized Web accelerator). See my Burton Group paper "Optimizing WAN Performance" and http://www.mnot.net/cacheability/, http://www.squid-cache.org/, http://www.web-caching.com/, and http://webmaster.info.aol.com/caching.html for more information. (Also look at http://developer.yahoo.com/performance/rules.html, which Nick Fiekowsky recommends, for more caching hints and general performance suggestions for the Web.)

Building Applications for the WAN

Posted by: Eric Siegel

Our final session at Catalyst this year was a joint presentation between Network and Telecom Strategies and Richard Watson of Burton Group's Application Platform Strategies group. We talked about how programmers and network engineers can work together to improve application performance and problem diagnosis.

This session was based on the NTS report "Designing Applications for the WAN," which will be updated next week -- so go read it when the update comes out (it's out now!), but there are a two points that came out in the session that I think were rather surprising to us network folks and that should be emphasized here.

First, the applications developers are now using frameworks to enable them to develop applications more quickly. Those application frameworks completely hide ("abstract") the network. For example, an application developer will ask a "method" for a "data object." The developer doesn't even realize that the network is involved, or that there may be some SQL queries used by that method to create the data object. That's why when you have a crisis, and you put Wireshark or a Sniffer on the communications link and show the resulting trace to the developer, he or she has no idea what the heck is going on. I'll say it again: the developer didn't even realize an SQL query was generated by the method request, and has no idea whatsoever what that stuff in the query string, trapped by the trace, really means.

No wonder there's frustration and wasted time in diagnosis! The time has arrived for the network organization, the operations group, and the development organization to cooperate and purchase a diagnostic tool that will trace individual transactions through the production system for diagnosis. (An example is Quest Software's PerformaSure; there are others.) The programmers will understand what they're seeing, the network staff won't need to run around doing Wireshark or Sniffer traces for days on end, and the problem will be handled much more quickly.

And, as the second point, realize that performance is poor because of the "abstraction" of the network layer. The application framework has a lot of parameters that set timeouts, window sizes, buffer sizes, compression, whether to disable the Nagle Algorithm, etc. The programmers understand none of this, so they use the defaults. But the defaults aren't usually set by the vendor to ensure optimum performance; they're set to create the fewest trouble calls to the vendor regardless of the network characteristics. That may not be the optimum setting for your particular situation, and that's why the people who install and set up the framework should talk to the network staff to jointly decide how to set the networking parameters for the networks that the framework will be running over. That'll help quite a bit in creating good performance.

TCP stack Performance Tuning for Windows

Posted by: Eric Siegel

I held a Birds of a Feather (BoF) session on WAN performance optimization at Burton Group's Catalyst Conference this week in San Diego, and we had approximately 14 attendees, which was quite good. (Attendance at Catalyst is almost exactly the same as last year: approximately 1500 people! We're VERY happy.)

We started the BoF with a talk by Nick Fiekowsky of Wyeth; he also has a talk coming up on WAN optimization, but the BoF was a great opportunity for us to have a more unstructured discussion (it lasted over an hour) about some of the detailed technical issues.

Nick made some great new observations about TCP performance on Windows; they're very interesting and also seem to have a large impact on performance. So here, with his permission, are some of them:

First, as Nick says, "Latency doesn't end at the RJ-45." When you're computing the required buffer size (receive window, RWIN) for long fat networks (i.e., large bandwidth and large latency), we always think of just the latency on the path and the bandwidth. I hadn't previously thought about the time that the received packet spends inside the receiving station's TCP buffers, waiting to be processed and then waiting to be read by the application. But that's an important factor in the latency; after all, the RWIN can open until there's space in the receive buffer, and the receive buffer won't have space until the application has read something out of it!

So Nick looked at what actually happened, and he discovered that although the bandwidth delay product (and therefore the receive buffer size) was approximately half a megabyte for a particular path, there were still long pauses waiting for acknowledgements to be returned to the sender -- throughput was lower than it could be. He had to increase the receive buffer size to almost THREE MEGABYTES before the sniffer (actually, Wireshark) trace showed that the bandwidth was being fully utilized. Wow.

And that led to Nick's other major point: it's important to optimize Windows performance to keep it from stalling while processing those incoming packets:

• "Non-Paged Pool" (NPP), where the buffers are located, is a precious resource. Nick says that in a 32-bit Windows OS, the NPP = RAM/16. That's not very large, especially if you need large receive buffers. (Look at the Task Manager to see how much of the NPP is being used. Or you can just notice that the TCP stack is requesting a small RWIN when a new connection comes in, because it's under memory pressure.) The only way to increase that is to increase the total RAM in the processor. In the 64-bit Windows OS, the NPP allocation is handled much better, TCP auto-tuning in Vista, Windows 7 and Windows Server 2008 makes better use of NPP, and finally the 64-bit OS also makes the entire system run better because system processes are faster. So the 64-bit OS is a big improvement here.
• Defrag the hard drive to be sure that fragmentation isn't slowing down the processes and the stack.
• Use updated NIC drivers. Nick says that recently he's noticed that there's a new emphasis on performance in the drivers, possibly being driven by iSCSI and datacenter requirements. Also increase the number of transmit and receive descriptors in the driver configuration.

Nick had a lot of other recommendations, such as the big improvement made by ensuring that Web cache expiration tags are properly used, and I'll try to get to them in a later blog.

Meanwhile, thanks, Nick!

Backup for WAN performance optimizers at remote offices

Posted by: Eric Siegel

(This is the first of a few blog entries that I'm going to post to share some ideas that came out of the Birds of a Feather (BoF) session on WAN performance optimization at Burton Group's Catalyst Conference this week in San Diego.)

A primary advantage of WAN performance optimization appliances is that they cut down the bandwidth needs of a remote branch office -- usually by 50% or more. And they reduce response time to make remote file shares, etc., actually usable instead of being a miserable, irritating experience that destroys productivity.

But what happens when the optimizer breaks? Bandwidth requirements spike upwards, and response time instantly becomes lousy.

My suggestions in the past involved QoS in the branch office boundary router to ensure that critical traffic would still flow despite the worsened bandwidth restrictions, and recommendations that the enterprise make arrangements with vendors or resellers to have spares available in-country (to avoid delays in customs and in transit) and configuration information available on a USB flash drive or by downloading as an email attachment. There would be a day or so of worse-than-normal performance as the compression directories are rebuilt, and then things would be OK.

But Nick Fiekowsky, while sharing his slides at the BoF, had a great idea: use the new software client optimizers, such as Blue Coat's Proxy Client, Cisco's WAAS Mobile, Expand Network's Mobile Accelerator Client, and Riverbed's Steelhead Mobile to serve as emergency backups. These software optimizers run on the users' laptops and provide the function of the optimization appliance. They're not as good, because (except for the new Expand Networks Mobile Accelerator Client! More later...) they don't share their compression directories and caches. But they're a lot better than nothing, and they also provide the ability for the user to use acceleration from home or on the road. This is especially great if the software optimizer system licenses are inexpensive or can be turned on and off easily from the central server location, with billing only when they're used.

My updated recommendation is therefore to install the WAN accelerator vendor's mobile client in branch office desktops. If the office's WAN accelerator fails, the desktop clients will step in to reduce WAN traffic and provide an acceptable user experience. It is necessary to have enough license capacity to cover all the client devices that may become active when a single branch office's WAN accelerator fails.

Be a Hero! Cut costs to Zero!

Posted by: Eric Siegel

Is there anyone who hasn't felt the threat of layoffs this last few months? Everyone is digging foxholes, worrying about the future. That's the natural tendency, I know. (Hey, I'm typing this from the warm security of my desk's kneehole, and my cats have gathered around me, prepared to show their fangs to anyone who might endanger their food supply.)

But if you really want to avoid Trouble, the thing to do is to get out there and save your company money, or set it up to make money quickly! Winning enterprises are using this period to prepare to launch at rocket speed out of the recession, leaving their foxhole-dwelling competitors chewing dirt and breathing rocket exhaust. And there's an easy way to help position your company for that launch, at the exact same time that you save megabucks immediately! It's called WAN Performance Optimization.

Yes, the same drum I've been beating for years, this time with extra added incentives!

I'll make this short, so we can all get back to work. But there isn't much to say.

ONE: WAN Optimization really DOES pay for itself in under a year. (In these times, payback in more than a year doesn't impress management at all. You need instant payback, and then big savings.) Those charts showing 50%, 80% and larger bandwidth savings aren't fake. In a surprising number of cases, the bandwidth savings are immense. So you'll be a hero for saving costs, and your procurement department will be singing your praises!

TWO: Your users will be delirious with joy at the performance improvements, especially if they've been suffering with remote file shares or with the pain of sending huge files back and forth among team members. Because it now takes only minutes, not hours, to send project files back and forth, there will be closer teamwork and they'll win more deals and avoid more screwups. Server centralization will be enabled, too, which is more money saved. So you're a hero for improving productivity and morale, and your HR and operations departments and your branch office managers and project managers will be singing your praises!

Just remember to forward all of those love notes to your management; this is not the time to be shy.

The technology can be somewhat complex, so be sure to find some reference accounts with a similar architecture and network design to yours. The vendors will love to show you around the technology, and they'll even loan you equipment to try it out. Different vendors specialize in different optimization situations; it's a good idea to look around. Some leading vendors are:

• Blue Coat
• Certeon
• Cisco (WAAS and AVS)
• Citrix (WanScaler and NetScaler)
• Exinda
• Expand Networks
• F5 (WANJet and WebAccelerator)
• Juniper (WX)
• Riverbed
• Silver Peak
• Stampede

See my previous blog entry, "WAN Optimization Whitelists and Blacklists"  for more information on this topic. Burton Group clients should look at my 80-single-spaced-page hyper-boring "Optimizing WAN Performance" report. And in a few weeks a new report, "Evaluating and Deploying WAN Performance Optimization," will be out in print. (Ask me for a pre-production copy.) It talks about how to pitch WAN optimization to management, how to select and evaluate contenders, and how to go through a rollout without crises. Or go to Burton Group's Catalyst Conference, July 27-31 in San Diego, where we're going to have tutorials and sessions on this, and a Birds-of-a-Feather meeting. Or call me for a free "dialogue." Gee, there are lots of ways to get started; no excuses here!

WAN Optimization Whitelists and Blacklists

Posted by: Eric Siegel

I've been working on a survey of best practices for acquiring WAN optimization solutions; it'll appear as a Burton Group paper and also be used as the basis for a talk at the upcoming Burton Group Catalyst Conference on July 27-31 in San Diego. It's been fun to talk to folks about what worked and what didn't work in their efforts to bring WAN optimization technology into their enterprises.

The papers and speech won't be out for another month or so, but meanwhile there's a very important point about this technology that I want to get out here: Yes, optimization is a fantastic way of saving money and creating really happy customers at the same time, but it doesn't work flawlessly for all applications.

The advanced dictionary-based compression component seems to be foolproof; it works and saves you, oh, 50% and up of your bandwidth at the same time that it usually makes a major improvement in performance that'll have the users loving you. So networking organizations that are primarily concerned with saving bandwidth expense and that dread having to work with users or application developers will usually choose optimization devices that are almost purely compression-based, and that don't fiddle much, or at all, with the protocols. They'll get a lot of performance improvement also -- especially if the same files are sent back-and-forth across the network with minor changes, or if they contain internal duplications, or if there's a lot of commonality among files -- and they'll never break an application, no matter how badly it's programmed.

But some enterprises really need the massive performance improvements that WAN optimization can provide for some common situations, such as access to remote fileshares using the Microsoft CIFS or Unix NFS protocols, or applications using SQL queries, or web browsers, or email. For those situations, fiddling with ("accelerating" or "spoofing") the underlying TCP, file, and application protocols can create simply unbelievable improvements in the performance seen by end users. But when the WAN optimization device starts interacting with protocols, some applications will break. It's almost never the optimization device's fault, because the optimizer engineers are experts in the protocols and carefully design their devices to squeeze every bit out of them without violating the protocol standards. But a surprising number of enterprise programmers don't know those standards very well.

The result is that protocol acceleration typically works flawlessly for broadly-used applications, such as access to remote fileshares by Excel, or use of web browsers, or SharePoint. But weird, custom-built client-server applications with embedded TCP socket calls that were built by overworked programmers under horrible deadline pressure who have no previous experience with TCP or other protocols? Well, that explains the "your accelerator broke my application" telephone calls. Truth is, the application was already broken, but its failures were usually rather rare and were attributed to "cosmic rays" or similar nonsense. When the optimizer starts really exercising the protocol, however, the breaks in the protocol become more obvious, and the irate telephone calls start.

And that brings me to the point of this blog entry: most enterprises quickly find that whitelists are wonderful tools for managing WAN optimization. They use whitelists to tell the optimizers which applications should be given protocol acceleration in addition to advanced compression. Applications that aren't on the whitelist are compressed, but their protocols are passed through the optimzers unchanged. At initial rollout, the network organization puts broadly-used applications, such as Excel, Internet Explorer and SharePoint, on the whitelist. Then, as time goes by, additional applications can be tested and added to the whitelist. By using whitelists, all of those weird applications that you didn't even know existed will not get into trouble. (By the way, organizations that try blacklists usually find that they have dozens, or hundreds, of applications that they didn't know about and that have mistakes in their use of TCP or other protocols. They find that out as a result of enraged telephone calls from the applications' users. Use whitelists instead.)

Some WAN optimization vendors have extensive deep packet inspection systems that can help in creating whitelists, but that's not truly necessary in all cases. Just filtering by IP address, hostname, or similar simple mechanisms may be all that you need.

So: don't fear optimization! You can save buckets of money and make your customers happy at the same time! But use whitelists when you do that rollout.

UC and Catalyst 2009 – Part 1

posted by: Mark Cortner

The conversations that I’ve been having recently with our clients related to UC are increasingly focused on several topics. The dialogues have been requested as enterprise IT organizations have initiated the development of their future strategies relative to UC. The process and scope of defining an enterprise’s UC strategy can be a complex task given the inherent convergence and integration of numerous communication and collaboration applications, none of which is occurring in a vacuum as individual UC applications continue to evolve themselves.

The topics frequently include questions related to the business value of UC, the intersection of mobility and UC, and the renewed interest in video technology and its integration into an enterprise’s UC strategy. The strong interest in these topics will be reflected in our planned content related to UC at our upcoming Catalyst Conference in July. The conference sessions focused on UC will address many questions related to this list of topics including:

• How can UC show real value or mitigate costs?
  
• How should you position and take advantage of UC
• What is the relationship between mobility and UC?
• How can an enterprise avoid lock-in solutions with mobile UC?
• What benefits can an enterprise expect from visual collaboration?
• How is technology impacting the utility of videoconferencing?

I’m excited by the program we’ll be presenting at Catalyst … look for more details on our UC sessions over the coming weeks including additional topics that will be addressed.

Data Center Ethernet (DCE) - The One Wire Solution (1 of 3)

Part 1 of 3

Wires, lots of wires, have made connectivity difficult at data centers. The goal to reduce the number and amount of wiring at data centers, and even at desks, is a worthy goal. Wires under raised floors, in cable trays, in ceilings, between floors, passed through conduit between buildings – the stuff is everywhere.

Until the advent of Data Center Ethernet (DCE), data centers had separate wiring for: power; data networks - Ethernet, Token Ring (the other major LAN from days gone by); PBX for voice; Fiber Channel for San access; WAN (modem, 56k, T1, T3, OC-n); RS-232 (remember terminal servers?) just to name the major ones. With DCE we are facing the challenge of consolidating wiring to a unified fabric: 10 Gig Ethernet (10GbE). On the surface it sounds simple yet it does it pose architecture, design, process, support and ownership challenges.

Never bet against Ethernet is the same as years ago when no one got fired for using IBM. Token Ring tried and did cohabitate with Ethernet for many years, even residing in the same hub. Eventually Token Ring went the way of the Dodo. Consolidation to Ethernet was swift yet little changed as data networks were still just “computer data”. We survived.

When true VoIP installments replaced traditional PBX infrastructure, the telecom engineers who didn’t evolve to data networks slowly fell by the wayside. Migrating from phone wiring/PBX to VoIP was a more complex step than moving from Token Ring to Ethernet. This was tough as data network engineers now had to understand all the services accompanying voice: dial tone; call waiting; voice mail; CallerID; *72; etc. The voice world was truly a different bird than Token Ring making the transition to Ethernet more challenging. We survived.

Looking at the data center, wiring was still a mess. As applications evolved and as old equipment is replaced, services requiring dedicated X.25, modem and 56k wiring to the rack are being replaced. While a specific service may require specialized telecom connections, many of these services can be offered from equipment placed in a common communications rack with connectivity via Ethernet to the server. In other cases, this older garb has been replaced by services over the Internet and routed via Ethernet to the server or even end user. More wiring is reduced and replaced by Ethernet, yet this was still data. We are surviving this too.

With DCE, the goal of a single “wire” 10GbE connection from a server to a unified fabric that offers ubiquitous connectivity for access is the next hurdle which means adding SAN access via iSCSI or Fiber Channel over Ethernet (FCoE). Will we survive? We should.

Part 2 will cover the wiring fabric.

My upcoming Catalyst conference presentation

Posted by: Paul DeBeasi

Many enterprises are considering Wi-Fi CERTIFIED™ 802.11n draft 2.0 deployment because it has significant advantages over existing wireless technologies. However, these advantages present the enterprise network manager with important deployment considerations. At the upcoming Burton Group Catalyst Conference in July, I will examine the various deployment considerations for 802.11n in the enterprise. Some of the topics I will discuss are listed below.

Most existing 802.11 devices operate in a single frequency band, (e.g., 2.4 GHz or 5 GHz). 802.11n is different because it is specifically designed to operate in both the 5 GHz and the 2.4 GHz frequency bands. So 802.11n presents an opportunity for enterprises to reconsider which frequency band(s) to use. I will discuss some of the tradeoffs and issues enterprise managers need to consider.

Legacy 802.11 access points (APs) have a maximum power draw that is very close to the IEEE 802.3af Power over Ethernet (PoE) maximum of 15.4 watts. However, many 802.11n APs will consume more power than legacy APs. Enterprise AP vendors have addressed this problem in several ways and enterprise IT managers must consider which approach they will select.

802.11b/g/a APs typically use Fast Ethernet ports to forward traffic onto the wired network. Because the 802.11n data rate is designed to exceed the 100 Mbps capacity of Fast Ethernet (especially when using 40 MHz channels), most new APs will use Gigabit Ethernet for wired Ethernet communication. Therefore, some enterprises may choose to upgrade a portion of their wiring closet switches, and possibly their cabling, to provide gigabit Ethernet links to their 802.11n APs.

WLAN controllers that perform the data-forwarding function must backhaul all wireless traffic from hundreds of 802.11n APs, and so each controller will likely need to support several Gigabit Ethernet connections. WLAN controllers that distribute the data-forwarding function to the AP will need only a single Gigabit Ethernet connection.

802.11n presents a new security challenge because existing hardware sensors, without software updates, may not be able to recognize 802.11n APs. Therefore, it is important for enterprise managers to update wireless intrusion detection systems (IDSs), regardless of whether or not they plan to deploy 802.11n.

802.11n will impact network management tools. For example, spectrum analyzers must be able to recognize MIMO spatial streams, and visually communicate network behavior to the user. Therefore, enterprise managers will need to upgrade their network management tools in order to manage 802.11n networks.

Please join me in beautiful San Diego, July 27 – 31, for a discussion of these issues and many other interesting topics. Learn more about the Burton Group Catalyst Conference here: http://www.catalyst.burtongroup.com/Na09/.