Network and Telecom Strategies Blog

Congestion Going Into Network Branches = Packet Loss

Posted by: Eric Siegel

End of the month, and I've been going over the list of the "dialogues" I had with Burton Group clients to see if there's something that might be of general interest.

So I uncovered one that I participated in with my colleague Ken Agress of Burton Group's consulting organization, and it's interesting because I think that the topic of QoS congestion on links into remote branches isn't given enough consideration in QoS network design.

The basic symptom of the problem was that despite QoS on the routers, there was excessive packet loss on high-priority flows into the branches on a hub-and-spoke network design.

But that's because the server-room router was trying to push too much data into the network cloud for a specific branch office. The discards weren't occurring at the server room boundary router into the cloud, because the fat access link from the server room will easily accommodate lots of traffic to the branch, especially if the other branches aren't receiving much traffic at that time.

If you have a large bandwidth access link into a cloud, but just a small bandwidth access link out of the cloud to a branch office destination, the cloud will need to discard data packets on high-bandwidth flows or during bursts, because the cloud itself (i.e., the backbone routers) has minimal buffering.

True, a single TCP flow will self-adjust (by "self-clocking") to a bandwidth-restricted exit from the cloud because its acknowledgements will be delayed, but if there are a lot of parallel flows, or there are non-flow-controlled UDP flows, you'll be in trouble. TCP flows will get the "global synchronization problem," in which all TCP flows ramp up in bandwidth concurrently, lose packets concurrently, reset their flow rates downwards concurrently, and then repeat. RED or WRED or FRED mechanisms can help avoid global synchronization, but you'll still lose packets.

So, how should this be handled? If you own the cloud or can signal to the cloud's administrator (e.g., by setting DSCP fields or EXP tags on packets going into a MPLS cloud), then the cloud can pick which packets to discard if the exit bandwidth isn't sufficient at the branch office. And the tags will, we hope, tell the cloud to discard packets from low-importance flows. RED / WRED / FRED on those flows will then get those low-importance flows to cut their bandwidth use. Those low-importance flows will be losing packets and will be miserable, but the higher-importance flows will get through.

A better idea, we think, is to avoid letting the cloud do your packet discards -- especially if you don't actually own/administer the cloud directly. Don't give it more traffic than it can handle; use traffic shaping on subinterface definitions to ensure that you never feed more data into the cloud than can exit at the destination. You will probably do a better job of discriminating among flow types and of smoothing brief traffic spikes (e.g., your routers probably have larger buffers than the cloud's routers have).

For example, see the following Cisco documents:

href="http://www.cisco.com/en/US/docs/ios/qos/configuration/guide/reg_pkt_flow_shaping.html" target=_blank>Regulating Packet Flow Using Traffic Shaping

href="http://www.cisco.com/en/US/tech/tk543/tk545/technologies_tech_note09186a0080114326.shtml" target=_blank>Applying QoS Features to Ethernet Subinterfaces

Or you could install a Blue Coat PacketShaper in the server room's exit path to do the traffic shaping for both TCP and UDP. And the new PacketShapers can also do compression.

XML Compression

Posted by: Eric Siegel

XML compression came up recently in one of my conversations with a client. It's an interesting topic that a lot of us will need to deal with as XML replaces other forms of process-to-process messaging.

XML is inefficient in its use of bandwidth, a tradeoff that was made to gain ease of use and generality. However, that need for a lot of bandwidth can be a problem, especially on paths with restricted bandwidth or high error rates, such as mobile wireless communications.

Some vendors, such as Stampede and Accelenet (ViaSat/ICT), provide advanced compression solutions that can be extremely effective -- and they also improve the transport protocol, which can help on wireless links. These are quite different from classical "zip" compression. They're proprietary and there are no "standards" involved; you can't pair up compression from two different vendors and expect them to interoperate. However, in exchange for the proprietary nature of these things, you get great compression. These systems maintain compression dictionaries at both end of the path, substituting short pointers for data strings. You can also pre-load these dictionaries with XML phrases; e.g., Stampede's Web 2.0 Performance Series has special features for SOA, including pre-loading of XML compression dictionaries, persistent TCP connections despite the tendency of AJAX (asynchronous Java and XML) to disconnect and reconnect frequently, and extensive security that includes XML schema validation, message anomaly detection, and denial of service attack detection. Indeed, most WAN performance optimization vendors provide advanced compression solutions that can assist with XML.

An alternative is to use binary XML, such as the Efficient XML Interchange format (EXI), which is being specified and evaluated by the Efficient XML Interchange (EXI) Working Group of W3C and is available from vendors such as AgileDelta. AgileDelta says that they provide software that supports standard XML APIs and they can also provide HTTP and TCP Socket proxies to convert XML to Efficient XML (EXI) without any change whatsoever to the application. The software could be used for applications that you can get the vendor to write with EXI; the proxies could be used to convert EXI to standard XML for applications that you don't have control over.

A big question, of course, is whether EXI will become broadly used. Although the EXI standard hasn't been formally approved, it's just one step away from that approval (it's in "Last Call"). The wikipedia article on binary XML has some discussion of these standards -- both EXI and others.

Note, of course, that any encryption or non-standard compression (e.g., proprietary compression, or possibly, EXI) can interfere with network devices that are trying to do "deep packet inspection" of data flows to detect policy violations and attacks, or to route data flows depending on flow contents. And any flow conversion system (compression or protocol acceleration) must usually intercept both flow directions; that can be tricky if flows travel asymmetrically. So use of any encryption, interception, or compression device means you have to think about precisely where you're going to place that device.

My colleague Chris Haddad also says that edge CPU processing speed for XML is important; it can be a problem that binary XML can help. An article referenced by the wikipedia binary XML entry is interesting here: "i-Technology Viewpoint: The Performance Woe of Binary XML" by Jimmy Zhang in SOA World Magazine, August 5, 2008.

QoS Problems With VoIP Over Wireless

Posted by: Eric Siegel

A fascinating article, "Coexistence of VoIP and TCP in Wireless Multihop Networks," was just published in the June 2009 edition of IEEE Communications Magazine (vol 47, issue 6, pp 75-81; the authors are Kyungtae Kim and Dragos Niculescu [NEC Laboratories America] and Sangjin Hong [Stony Brook University - SUNY]).

Burton Group has been warning that using VoIP over wireless LAN (VoWLAN) is tricky because "QoS alone cannot completely regulate the bandwidth provided to flows on a wireless network... a single wireless station on a shared wireless channel can degrade performance for all other stations on the channel...." We've been saying that: "A reasonable alternative is to avoid voice over WLAN altogether, substituting cellular voice, possibly with femtocell or in-building repeater technology." (See the Burton Group reports "Quality of Service (QoS) Alternatives Demystifying Radio Management;along with the Technical Position "Quality of Service".)

This new article shows that the problems created by VoIP over wireless LAN are even worse than I'd thought. The authors did some experiments, and they found that TCP and VoIP do not coexist well on a wireless LAN. TCP's large frame sizes, large receive windows (RWIN), bursty traffic, and retransmission when a frame is lost make it interfere with short, evenly-spaced VoIP frames. The TCP frames will dominate radio transmissions because of the frame lengths and frequency during a burst. (Frames can't be interrupted after they start to transmit; therefore, long, frequent TCP frames will dominate over infrequent, short VoIP frames.)

Worse, the self-interference and half-duplex characteristics of a wireless LAN system increase the probability that the long TCP data bursts will need retransmission, further increasing their burstiness and their domination of the channel when compared to short VoIP frames. (Self-interference occurs when a single flow interferes with itself as it travels over the wireless system. When transmission and reception share the same half-duplex radio channel, a stream may compete with its own acknowledgements for radio time, delaying or destroying some data and acknowledgement packets. If transmission hops over a couple of radio links in sequence, then each hop's transmission may interfere with transmissions on other hops that are within reception range; a flow may interfere with itself.)

The authors of the new article show that no standard type of queuing algorithm in routers adjacent to the wireless network, even Priority Queuing (PQ), can handle these problems. And they also show that five standard TCP variants used in end stations (TCP Reno, TCP Vegas, TCP Westwood, TCP CUBIC [used in Linux], and C-TCP [used in Windows Vista and Server 2008]) do not handle these problems. In all cases, the TCP flows dominate and degrade the VoIP flows.

What should be done? Well, Burton Group has recommended that cellular should be considered instead of VoWLAN. The authors of the study make additional suggestions that sound interesting, but they also warn that none of them is a panacea - they all are only partial solutions, at best. The authors warn that "the methods examined here have straight-forward application only when the wireless capacity is fixed. In reality, the capacity is highly variable." They also warn that P2P, uploading traffic, and interactive streaming, such as Skype, that use TCP for voice will complicate the situation even more.

The paper's most interesting recommendations are:

• Use unusually-small TCP receive windows and eliminate TCP bursts on the wireless LAN. This may need to be done by gateways at the wired:wireless interface. The gateway would use smaller TCP packets and a small receive window; it would also use a buffer to ensure that any bursts were smoothed out and that TCP packets were spaced out to reduce self-interference and interference with VoIP frames.
• Use a traffic shaping system. The article says, "After a packet of size pkt_size goes out over the air, the next packet is scheduled no earlier than pkt_size/R. The gateway chooses R based on the network status to determine how much to send as well as when to send."

Other papers have recommended different methods, such as doing priority queuing at all intermediate wireless nodes and combining that with traffic shaping.

At this time, it seems to me that avoiding sharing the same channel between TCP and voice is still the best solution.

Optimizing video training downloads

Posted by: Eric Siegel

If you're using low-bandwidth links to connect to remote sites, traffic contention for bandwidth can be intense. Some enterprises are looking at QoS to handle the bandwidth requirements for steaming video, but that usually doesn't work very well. There' may not be enough bandwidth for the video, and the enterprise managers then start looking at some kind of distributed caching solution.

But even distributed caching doesn't fix all of the problem; in many cases the available bandwidth is still strained by the video downloads to the cache. It's therefore very important to try to decrease the size of the media files; don't assume that the network organization must push the entire insufficiently-compressed video out to the remote nodes as-is.

In most training videos, the images don't need to be displayed in very high quality; the audio is more important. Therefore, enterprises should set up a facility for providing a compression (and re-compression) service for all media that is to be distributed over the network. Existing media that are compressed with MPEG-1 or MPEG-2 should be recompressed before transmission. MPEG-4 Advanced Video Compression (AVC), which is defined by the MPEG-4 Part 10 video standard and is the same as ITU-T H.264, is much better than MPEG-1 for low-bandwidth paths. The new Scalable Video Coding (SVC) standard was approved in in mid-2007; it may be useful because it improves the ability of a transmission device to adjust the streaming bandwidth to compensate for poorly-performing or low-bandwidth transmission paths.

Blue Coat provides video caching; its ProxySG appliance can pre-load, filter, and track usage to audit who is watching what. Cisco's Application and Content Networking System (ACNS) is another alternative, as is Real Networks' Helix Proxy.

It's also important to use basic router-based bandwidth allocation, or flow-metering appliances (e.g., PacketShaper from Blue Coat's Packeteer acquisition, NetEnforcer from Allot Communications, or NE2000 from NetEqualizer) to ensure that the video flows do not overwhelm the paths to the end users and interfere with higher-priority traffic. If possible, videos can be scheduled to be pushed to remote distribution nodes or caches at times where WAN bandwidth requirements are otherwise low.

Comments? Just add them below!

WAN Performance Optimization for Storage Systems

Posted by: Eric Siegel

I'm getting a number of inquiries about using WAN optimization devices on the high-bandwidth (e.g., 155 Mbps and up) paths between large-scale storage systems. There are some interesting points about these situations:

• Transmission rate is so high that many WAN optimization devices either bypass deduplication and compression entirely, or they restrict the deduplication directory to a database that can be kept in memory instead of on the optimization device's disk. However, even if the resulting decrease in required bandwidth is only 3 or 4x reduction instead of, say, a 10x reduction, that's still a massive savings when you consider the monthly cost of high-bandwidth links. Ask the vendor what the proposed product does on high-bandwidth paths. 

• On high-bandwidth, high-latency WAN paths, the flow-control and error-control characteristics of the optimization device's transport protocol are more important to overall performance than they are on lower-bandwidth paths. Even if there are no lost packets, normal TCP with too-small windows will have difficulty keeping the path full of data. Bursts of errors -- or even a few individual errors -- can make the situation much worse. If you can't alter the TCP stack in the end-user devices (for example, in the storage systems) to optimize TCP flow, then the WAN optimization devices can make a notable improvement without any compression or deduplication at all. The WAN optimization device can use forward error correction (FEC) to conceal individual packet errors, or it can use a special protocol for WAN transport. FEC is good for errors that are evenly distributed; a special transport protocol (such as the SCPS-TP variant of TCP) or advanced versions of TCP (such as Microsoft Vista's Compound TCP) handle both evenly-spaced errors and error bursts.  

• Incorrectly-ordered packets can decrease throughput. The upper-level protocols, such as TCP and iSCSI, will re-order packets to ensure that even if a data flow is divided among a number of parallel paths, the final flow will be seamlessly reassembled. However, packets arriving out of order can greatly slow down this process and require special buffering inside the protocol stacks. A WAN optimization solution that does some efficient packet re-ordering of its own can save time and processing in the end-user stacks. Higher-bandwidth flows have a higher probability than do lower-bandwidth flows of including parallel sub-flows (someone may have inserted a set of parallel links somewhere to increase capacity, or the storage device may be using multiple TCP flows in parallel); it's therefore more likely that the higher-bandwidth flows will be more responsive to the advantages of packet re-ordering in optimization systems. 

• Data deduplication and compression facilities in large-scale storage systems are almost never as effective at detecting and removing redundancy as are optimization devices. For example, they may detect only duplicate data blocks within a file, not among separate files, and they may work in large, fixed length (e.g., 4 Kbyte) blocks instead of being able to detect duplicate strings of any length. Both of these restrictions greatly restrict the deduplication that can be done. In the first case, multiple files (such as different versions of a virtual machine OS) that are almost completely identical will not gain any improvement from their similarity; each file will be deduplicated separately. In the second case, inserting a single word into a long file will result in the entire file's appearing to be different from that point onwards. The single inserted word will push a word off the end of the containing file segment's boundary and into the next file segment, which will then lose a word into the file segment after that, etc. With fixed-length blocks, all those "different" file segments will need to be transmitted across the network. The WAN optimization devices can be useful in squeezing additional savings out of these situations, and they can also improve the performance of the underlying protocols and can deduplicate the application-level headers inserted by the storage systems' backup protocols. 

• If a file is already compressed, such as with the LZW or deflate/zip algorithms, then, depending on the file structures, it may be a good idea to decompress it before looking for duplicates. A single word change in the middle of a compressed file can alter almost all of the file. Files that are end-to-end duplicates before compression will still be duplicates after compression, but files that are different by only one word before compression may look almost completely different after compression! Some WAN optimization devices can decompress already-compressed files; others can't.

• Support for higher-level protocols such as CIFS, MAPI, and others may be less important for WAN optimization devices supporting only back-end storage systems that it is for WAN optimization devices supporting individual remote users.

Well, that's a brief set of thoughts about the special situation of WAN optimization and large-scale storage systems. If you have any additional comments or observations, write a note here!

Need volunteers to talk about WAN Performance Optimization projects!

Posted by: Eric Siegel, Senior Analyst, NTS

The WAN performance optimization market is as hot as ever, and the amount of confusing marketing is (sigh!) also as hot as ever.

Well, the technology IS complex, and, as I've always said, the main issue with WAN optimization devices isn't their effectiveness -- they are almost miraculous in the performance improvement and the bandwidth savings they can provide -- it's their integration with the rest of the network system and their implications for system management.

SO I'm putting out a call to all of you to help me in a WAN Optimization research project! I'm going to put together a "best practices" guide to acquiring a WAN optimization system, and I want to know your stories. So I'd love to have a conversation with you; just send me an email at esiegel at burtongroup.com and let's talk! (And don't worry, we can hide your identity! That will let us have fun exchanging horror stories.)

Among other things, I'm interested in how the optimization system fits into your existing network and application architecture, how you selected vendors for evaluation, how you performed that evaluation, how you rolled out the optimization system, and what you'd do differently if you had it to do over again.

So let's help each other out and enjoy telling stories! Send me an email, and we can get together for a chat over the next month or so.

Eric