Network and Telecom Strategies Blog

Be a Hero! Cut costs to Zero!

Posted by: Eric Siegel

Is there anyone who hasn't felt the threat of layoffs this last few months? Everyone is digging foxholes, worrying about the future. That's the natural tendency, I know. (Hey, I'm typing this from the warm security of my desk's kneehole, and my cats have gathered around me, prepared to show their fangs to anyone who might endanger their food supply.)

But if you really want to avoid Trouble, the thing to do is to get out there and save your company money, or set it up to make money quickly! Winning enterprises are using this period to prepare to launch at rocket speed out of the recession, leaving their foxhole-dwelling competitors chewing dirt and breathing rocket exhaust. And there's an easy way to help position your company for that launch, at the exact same time that you save megabucks immediately! It's called WAN Performance Optimization.

Yes, the same drum I've been beating for years, this time with extra added incentives!

I'll make this short, so we can all get back to work. But there isn't much to say.

ONE: WAN Optimization really DOES pay for itself in under a year. (In these times, payback in more than a year doesn't impress management at all. You need instant payback, and then big savings.) Those charts showing 50%, 80% and larger bandwidth savings aren't fake. In a surprising number of cases, the bandwidth savings are immense. So you'll be a hero for saving costs, and your procurement department will be singing your praises!

TWO: Your users will be delirious with joy at the performance improvements, especially if they've been suffering with remote file shares or with the pain of sending huge files back and forth among team members. Because it now takes only minutes, not hours, to send project files back and forth, there will be closer teamwork and they'll win more deals and avoid more screwups. Server centralization will be enabled, too, which is more money saved. So you're a hero for improving productivity and morale, and your HR and operations departments and your branch office managers and project managers will be singing your praises!

Just remember to forward all of those love notes to your management; this is not the time to be shy.

The technology can be somewhat complex, so be sure to find some reference accounts with a similar architecture and network design to yours. The vendors will love to show you around the technology, and they'll even loan you equipment to try it out. Different vendors specialize in different optimization situations; it's a good idea to look around. Some leading vendors are:

• Blue Coat
• Certeon
• Cisco (WAAS and AVS)
• Citrix (WanScaler and NetScaler)
• Exinda
• Expand Networks
• F5 (WANJet and WebAccelerator)
• Juniper (WX)
• Riverbed
• Silver Peak
• Stampede

See my previous blog entry, "WAN Optimization Whitelists and Blacklists"  for more information on this topic. Burton Group clients should look at my 80-single-spaced-page hyper-boring "Optimizing WAN Performance" report. And in a few weeks a new report, "Evaluating and Deploying WAN Performance Optimization," will be out in print. (Ask me for a pre-production copy.) It talks about how to pitch WAN optimization to management, how to select and evaluate contenders, and how to go through a rollout without crises. Or go to Burton Group's Catalyst Conference, July 27-31 in San Diego, where we're going to have tutorials and sessions on this, and a Birds-of-a-Feather meeting. Or call me for a free "dialogue." Gee, there are lots of ways to get started; no excuses here!

The Latest Communications Link Failure

Posted by: Eric Siegel

OK, yesterday there was a major disruption of the communications links in the San Francisco Bay Area. Ten fiber optic cables, belonging to AT&T and to Sprint, were cut at four separate locations by saboteurs. (See, for example, The San Francisco Chronicle, "Sabotage attacks knock out phone service" August 10, 2009,and The Mercury News, "Phone service fully restored; AT&T offers 100K reward" August 10, 2009)

All Hell broke loose; telephones were down, ATMs stopped working; 911 calls went nowhere.

There were the usual articles talking about how this should "remind" everyone about link failures, et cetera, and about how everyone was so surprised at this.

There really shouldn't be any surprise. THIS is a major reason for having a truly remote data center! In any geographic area, there are only a few paths for communications links. They're typically next to railroads or other utilities that have a long, narrow right-of-way. It's just too difficult to negotiate with hundreds of landowners for a new right-of-way for hundreds of miles. The result is that most communications links, even from different carriers, end up in the same manholes and tunnels. (Remember the July 2001 CSX train derailment in a Baltimore railroad tunnel? It started a fire and melted the fiber optic cables in that tunnel, causing major problems for at least five ISPs. Or how about the December 2008 cut of a major undersea cable in the Mediterranean, one of many cuts, often caused by boat anchors? [See WIRED.com "Undersea Cables Cut; 14 Countries Lose Web -- Updated" December 19, 2008 ] There are hundreds of examples.)

The rules are:

If you have "redundant" access links to the service provider, using the same or different carriers, you must inquire very carefully about the routing of those links. They will usually share facilities at some point! Be sure each access link arrives at a major service provider switching node, with many physically divergent routes from that point, before it encounters any shared facilities with any other access link. Physical diversity will guard against a single physical event killing both access links into the massive service provider meshes.

Strongly consider using alternative media, such as satellite or radio links (see the Burton Group reports "Fixed Broadband Wireless Access Technology: Unwiring the Last Mile" and "The Wild World of WISPs" ), to connect to a service provider who will be outside the radius of any probable disruptions.

Ensure that your "backup" link is actually working. Share traffic across it, or test it very frequently.

And remember that loss of communications is a major reason for having a redundant data center!

Please TEST before inflicting your app on the network!

Posted by: Eric Siegel

Here's another pet peeve of mine: lack of realistic pre-deployment testing.

How many development organizations use zero-latency infinite-bandwidth LANs and top-of-the-line switches in their development shop, then, when their tightly-coupled client-server application goes into production, they blame lousy performance on the network? As the saying goes, "If I had a nickel....." Well, I'd be writing this blog entry from a penthouse suite on Fifth Avenue.

A leading cause of friction between network operations staff and development staff could be removed if all applications were realistically tested before being placed into production. Real production network environments differ greatly from typical development environments; they include high-latency WAN and cellular links, bandwidth restrictions, congestion, link errors, packet duplication and misordering, and packet fragmentation caused by devices in the communications path, encryption, and virtual private network (VPN) connections. Bandwidth optimization appliances such as caches, compression devices, and protocol-acceleration devices also interact with data flows. All of these affect application performance.

If the application is accessible from the public web, designers must also consider and handle the possibility of a flash load. Flash loads occur when overwhelming numbers of users try to connect to a web server simultaneously, possibly in response to a news event or a television advertisement. The flash load inevitably leads to widespread abandonment of slow transactions, causing resource recovery problems (transaction abandonment is difficult to detect in web environments) and probably further degrading performance. Public web applications must also consider the universe of possible browser types and the fact that a small percentage of users disable cookies on their browsers.

WAN emulation and load generation devices and services can reproduce all of these problems, and it is much less stressful for everybody if problems are discovered before the application is put into production. Surprising things happen to data flows on production WANs: Always test before deployment.

WAN Optimization Whitelists and Blacklists

Posted by: Eric Siegel

I've been working on a survey of best practices for acquiring WAN optimization solutions; it'll appear as a Burton Group paper and also be used as the basis for a talk at the upcoming Burton Group Catalyst Conference on July 27-31 in San Diego. It's been fun to talk to folks about what worked and what didn't work in their efforts to bring WAN optimization technology into their enterprises.

The papers and speech won't be out for another month or so, but meanwhile there's a very important point about this technology that I want to get out here: Yes, optimization is a fantastic way of saving money and creating really happy customers at the same time, but it doesn't work flawlessly for all applications.

The advanced dictionary-based compression component seems to be foolproof; it works and saves you, oh, 50% and up of your bandwidth at the same time that it usually makes a major improvement in performance that'll have the users loving you. So networking organizations that are primarily concerned with saving bandwidth expense and that dread having to work with users or application developers will usually choose optimization devices that are almost purely compression-based, and that don't fiddle much, or at all, with the protocols. They'll get a lot of performance improvement also -- especially if the same files are sent back-and-forth across the network with minor changes, or if they contain internal duplications, or if there's a lot of commonality among files -- and they'll never break an application, no matter how badly it's programmed.

But some enterprises really need the massive performance improvements that WAN optimization can provide for some common situations, such as access to remote fileshares using the Microsoft CIFS or Unix NFS protocols, or applications using SQL queries, or web browsers, or email. For those situations, fiddling with ("accelerating" or "spoofing") the underlying TCP, file, and application protocols can create simply unbelievable improvements in the performance seen by end users. But when the WAN optimization device starts interacting with protocols, some applications will break. It's almost never the optimization device's fault, because the optimizer engineers are experts in the protocols and carefully design their devices to squeeze every bit out of them without violating the protocol standards. But a surprising number of enterprise programmers don't know those standards very well.

The result is that protocol acceleration typically works flawlessly for broadly-used applications, such as access to remote fileshares by Excel, or use of web browsers, or SharePoint. But weird, custom-built client-server applications with embedded TCP socket calls that were built by overworked programmers under horrible deadline pressure who have no previous experience with TCP or other protocols? Well, that explains the "your accelerator broke my application" telephone calls. Truth is, the application was already broken, but its failures were usually rather rare and were attributed to "cosmic rays" or similar nonsense. When the optimizer starts really exercising the protocol, however, the breaks in the protocol become more obvious, and the irate telephone calls start.

And that brings me to the point of this blog entry: most enterprises quickly find that whitelists are wonderful tools for managing WAN optimization. They use whitelists to tell the optimizers which applications should be given protocol acceleration in addition to advanced compression. Applications that aren't on the whitelist are compressed, but their protocols are passed through the optimzers unchanged. At initial rollout, the network organization puts broadly-used applications, such as Excel, Internet Explorer and SharePoint, on the whitelist. Then, as time goes by, additional applications can be tested and added to the whitelist. By using whitelists, all of those weird applications that you didn't even know existed will not get into trouble. (By the way, organizations that try blacklists usually find that they have dozens, or hundreds, of applications that they didn't know about and that have mistakes in their use of TCP or other protocols. They find that out as a result of enraged telephone calls from the applications' users. Use whitelists instead.)

Some WAN optimization vendors have extensive deep packet inspection systems that can help in creating whitelists, but that's not truly necessary in all cases. Just filtering by IP address, hostname, or similar simple mechanisms may be all that you need.

So: don't fear optimization! You can save buckets of money and make your customers happy at the same time! But use whitelists when you do that rollout.

Optimizing video training downloads

Posted by: Eric Siegel

If you're using low-bandwidth links to connect to remote sites, traffic contention for bandwidth can be intense. Some enterprises are looking at QoS to handle the bandwidth requirements for steaming video, but that usually doesn't work very well. There' may not be enough bandwidth for the video, and the enterprise managers then start looking at some kind of distributed caching solution.

But even distributed caching doesn't fix all of the problem; in many cases the available bandwidth is still strained by the video downloads to the cache. It's therefore very important to try to decrease the size of the media files; don't assume that the network organization must push the entire insufficiently-compressed video out to the remote nodes as-is.

In most training videos, the images don't need to be displayed in very high quality; the audio is more important. Therefore, enterprises should set up a facility for providing a compression (and re-compression) service for all media that is to be distributed over the network. Existing media that are compressed with MPEG-1 or MPEG-2 should be recompressed before transmission. MPEG-4 Advanced Video Compression (AVC), which is defined by the MPEG-4 Part 10 video standard and is the same as ITU-T H.264, is much better than MPEG-1 for low-bandwidth paths. The new Scalable Video Coding (SVC) standard was approved in in mid-2007; it may be useful because it improves the ability of a transmission device to adjust the streaming bandwidth to compensate for poorly-performing or low-bandwidth transmission paths.

Blue Coat provides video caching; its ProxySG appliance can pre-load, filter, and track usage to audit who is watching what. Cisco's Application and Content Networking System (ACNS) is another alternative, as is Real Networks' Helix Proxy.

It's also important to use basic router-based bandwidth allocation, or flow-metering appliances (e.g., PacketShaper from Blue Coat's Packeteer acquisition, NetEnforcer from Allot Communications, or NE2000 from NetEqualizer) to ensure that the video flows do not overwhelm the paths to the end users and interfere with higher-priority traffic. If possible, videos can be scheduled to be pushed to remote distribution nodes or caches at times where WAN bandwidth requirements are otherwise low.

Comments? Just add them below!

WAN Performance Optimization for Storage Systems

Posted by: Eric Siegel

I'm getting a number of inquiries about using WAN optimization devices on the high-bandwidth (e.g., 155 Mbps and up) paths between large-scale storage systems. There are some interesting points about these situations:

• Transmission rate is so high that many WAN optimization devices either bypass deduplication and compression entirely, or they restrict the deduplication directory to a database that can be kept in memory instead of on the optimization device's disk. However, even if the resulting decrease in required bandwidth is only 3 or 4x reduction instead of, say, a 10x reduction, that's still a massive savings when you consider the monthly cost of high-bandwidth links. Ask the vendor what the proposed product does on high-bandwidth paths. 

• On high-bandwidth, high-latency WAN paths, the flow-control and error-control characteristics of the optimization device's transport protocol are more important to overall performance than they are on lower-bandwidth paths. Even if there are no lost packets, normal TCP with too-small windows will have difficulty keeping the path full of data. Bursts of errors -- or even a few individual errors -- can make the situation much worse. If you can't alter the TCP stack in the end-user devices (for example, in the storage systems) to optimize TCP flow, then the WAN optimization devices can make a notable improvement without any compression or deduplication at all. The WAN optimization device can use forward error correction (FEC) to conceal individual packet errors, or it can use a special protocol for WAN transport. FEC is good for errors that are evenly distributed; a special transport protocol (such as the SCPS-TP variant of TCP) or advanced versions of TCP (such as Microsoft Vista's Compound TCP) handle both evenly-spaced errors and error bursts.  

• Incorrectly-ordered packets can decrease throughput. The upper-level protocols, such as TCP and iSCSI, will re-order packets to ensure that even if a data flow is divided among a number of parallel paths, the final flow will be seamlessly reassembled. However, packets arriving out of order can greatly slow down this process and require special buffering inside the protocol stacks. A WAN optimization solution that does some efficient packet re-ordering of its own can save time and processing in the end-user stacks. Higher-bandwidth flows have a higher probability than do lower-bandwidth flows of including parallel sub-flows (someone may have inserted a set of parallel links somewhere to increase capacity, or the storage device may be using multiple TCP flows in parallel); it's therefore more likely that the higher-bandwidth flows will be more responsive to the advantages of packet re-ordering in optimization systems. 

• Data deduplication and compression facilities in large-scale storage systems are almost never as effective at detecting and removing redundancy as are optimization devices. For example, they may detect only duplicate data blocks within a file, not among separate files, and they may work in large, fixed length (e.g., 4 Kbyte) blocks instead of being able to detect duplicate strings of any length. Both of these restrictions greatly restrict the deduplication that can be done. In the first case, multiple files (such as different versions of a virtual machine OS) that are almost completely identical will not gain any improvement from their similarity; each file will be deduplicated separately. In the second case, inserting a single word into a long file will result in the entire file's appearing to be different from that point onwards. The single inserted word will push a word off the end of the containing file segment's boundary and into the next file segment, which will then lose a word into the file segment after that, etc. With fixed-length blocks, all those "different" file segments will need to be transmitted across the network. The WAN optimization devices can be useful in squeezing additional savings out of these situations, and they can also improve the performance of the underlying protocols and can deduplicate the application-level headers inserted by the storage systems' backup protocols. 

• If a file is already compressed, such as with the LZW or deflate/zip algorithms, then, depending on the file structures, it may be a good idea to decompress it before looking for duplicates. A single word change in the middle of a compressed file can alter almost all of the file. Files that are end-to-end duplicates before compression will still be duplicates after compression, but files that are different by only one word before compression may look almost completely different after compression! Some WAN optimization devices can decompress already-compressed files; others can't.

• Support for higher-level protocols such as CIFS, MAPI, and others may be less important for WAN optimization devices supporting only back-end storage systems that it is for WAN optimization devices supporting individual remote users.

Well, that's a brief set of thoughts about the special situation of WAN optimization and large-scale storage systems. If you have any additional comments or observations, write a note here!

Need volunteers to talk about WAN Performance Optimization projects!

Posted by: Eric Siegel, Senior Analyst, NTS

The WAN performance optimization market is as hot as ever, and the amount of confusing marketing is (sigh!) also as hot as ever.

Well, the technology IS complex, and, as I've always said, the main issue with WAN optimization devices isn't their effectiveness -- they are almost miraculous in the performance improvement and the bandwidth savings they can provide -- it's their integration with the rest of the network system and their implications for system management.

SO I'm putting out a call to all of you to help me in a WAN Optimization research project! I'm going to put together a "best practices" guide to acquiring a WAN optimization system, and I want to know your stories. So I'd love to have a conversation with you; just send me an email at esiegel at burtongroup.com and let's talk! (And don't worry, we can hide your identity! That will let us have fun exchanging horror stories.)

Among other things, I'm interested in how the optimization system fits into your existing network and application architecture, how you selected vendors for evaluation, how you performed that evaluation, how you rolled out the optimization system, and what you'd do differently if you had it to do over again.

So let's help each other out and enjoy telling stories! Send me an email, and we can get together for a chat over the next month or so.

Eric

The Progress & Freedom Foundation and Broadband Policy

Posted by: Jeff Young

The Progress & Freedom Foundation.  With a name like this, how could anyone disagree with the Progress & Freedom Foundation?  I’m for progress!  I’m for freedom!  When you put them both together, I’m not sure how I feel about the PFF.  Lately the PFF has been doing a bit of cheerleading for the state of broadband in the US telecom industry and it’s contradictory to a recent report from Burton (you can read the PFF text here).  Prior to this report the PFF hadn’t really been on my radar but I’m glad they now are.  The site contains some very thoughtful material.

I picked up the report, read it, and honestly found places where I agreed with the author in principle.  I just couldn’t arrive at many of the conclusions he made. With regard to broadband in the US much of what is written in the PFF document flies in the face of popular thinking.  Reading the document left me to wonder “just who is the PFF?” 

Unfortunately foundation names don’t come with “truth in advertising” so it’s often a good idea to have a look at who is funding the work within a foundation; major sponsors are listed on the PFF web site.  I can save you the click, as you might have guessed the PFF supporters list reads as a who’s who of big telecom and media companies.  Of course, that doesn’t make what’s inside the PFF site wrong, it might just be prefaced with “in the opinion of large media companies around the world.”

Maybe the bit of exaggeration in this document goes along with the mood this political season.  Perhaps telecom lobbyists are getting a jump-start on the election outcome.  Comcast may have funded the document as a way to get us all to write about something other than their trials before the Federal Communications Commission.  No matter who funded the document or whatever the reason for writing it, the PFF document is, at least, thought provoking.  Read both documents and form your own conclusions.

The PFF document’s basic premise is this: 

“In this paper, I compare U.S. and (briefly) Canadian broadband policies and outcomes with the policies and outcomes in other advanced nations. The results show that the relatively deregulatory American approach to broadband policy has produced highly desirable results, including high levels of investment and innovation, nearly ubiquitous broadband availability, high and increasing levels of penetration, falling prices, and high levels of consumer satisfaction. Indeed, the U.S. model is producing better overall results than in countries, which continue to pursue mandatory unbundling and other highly regulatory approaches. Moreover, the advantages of the American model are likely to grow more pronounced over time. To avoid being left behind, other nations should abandon policies based on mandatory resale of incumbent networks and adopt the American approach.” 

As I stated, I can often agree with the author in principle, I just have trouble with some of the conclusions he reaches.  Few would argue that the early regulatory breakup of AT&T was a bad thing.  It certainly encouraged investment and it created an atmosphere in which technologies such as the Internet were able to grow and mature.  After 20-some years working in telecom companies I am proud to have worked in companies that were at the heart of the creation of the public Internet.

The PFF document (and in fact, the Burton report) might disagree with later regulatory policies, especially those that mandated unbundling (wholesale of telecom assets to competitors) and eventually led to the bandwidth glut and to the temporary collapse of much of the telecom industry.  One key difference in both documents is the question:  is unbundling flawed as an idea or was the implementation of unbundling by US regulators too far-reaching?

Looking back, in the US the regulators saw unbundling as a way to create competition in the last-mile of telecom.  The reasoning:  if unbundling can support strong competitors, sooner or later those competitors will grow to a size that requires they invest in their own physical infrastructure.  Thus unbundling will bring real competition to the last mile.  The problem, as many have pointed out, is that no company has ever made the jump from buying wholesale, unbundled assets to investing in physical plant.  Many countries in the world still mandate wholesale access to the telecom last mile (including Britain, France, Japan, and others).  Although the PFF document encourages other countries to abandon unbundling and follow the US, these countries all have a healthy competitive marketplace in broadband access.

Regulators in the US also allowed incumbents (the owners of the last mile) to create their own subsidiary companies and allowed those subsidiaries to compete with broadband startups.  This hastened the crash, and it lends credence to the notion that unbundling isn’t fundamentally flawed, regulatory policy set unbundling up to fail in the US.  For more on this, read the documents.

Unbundling seems a thing of the past, no longer necessary in the US.  This is because stepping forward to today, the US was fortunate to have something to fall back on, an industry that provided competition in the last mile without the aid of regulation.  That industry is the cable industry. 

As the telecom markets were crashing and competitive local exchange carriers (CLECs) were going out of business, the cable industry was facing it’s own crisis.  Revenues were flat, penetration of homes wasn’t increasing and increasing rates to achieve better average revenue per user (ARPU) could only do so much to compensate.  Cable turned to broadband, and eventually telephony to continue its growth.

The current state of consumer-based telecom in the US has cable gaining ground in the traditional market of telephony.  Consumers are finding and telephone companies are surprised to learn that the phone company just isn’t necessary.  In response, the traditional telephone providers are investing billions of dollars to become cable companies.  Penetration of broadband still rests at 57% of homes in the US leaving us ranked number 15 among the top 25 industrialized nations.  The PFF document finds that the broadband industry is healthy and further regulation isn’t warranted.  I’m certainly not a fan of regulation, but…

- I find it ironic that 47% of homes in the nation that invented the Internet do not have broadband,

- I find it ironic that telecom companies are investing billions to build a fourth way (in addition to CATV, Satellite, and over-air) to deliver television.

- I find it ironic that the future of media delivery (including television) will be over the Internet and that both telecom and cable are fighting neutrality laws that would accelerate this future rather than investing in it.

- Finally, I find it ironic that the telecom lobby has been able to pass legislation in many US states to prevent municipalities from building networks for their constituents. 

If competition is alive and well and provides consumer choice as the PFF document concludes, who is going to provide for the other 43% of us?  I have to throw my hat in with those who believe that broadband infrastructure has become the next generation’s roadway system.  How would the US economy fair if 43% of us couldn’t use the roads because roads were in private hands?

Aventail on the way out?

I always get nervous when a company I'm using gets acquired. Often, this means that development gets stalled and necessary upgrades take forever to come out. That's if they come out at all. Take Aventail for example. When Apple announced OS X 10. 5 (Leopard), Cisco and Juniper had already deployed VPN solutions that worked under this operating system. So why couldn't Aventail do the same? This isn't rocket science, folks. It's software. The most likely reason is that Aventail was in the middle of being acquired by SonicWall and was distracted from their core business of supporting their customers.

As a result, anyone using a Mac and connecting through an Aventail SSL device must wait for that company to release a new SSL VPN client in order to upgrade to Leopard. Those who have already upgraded are out of luck. Aventail support tells them to run 10.4 and that 10.5 is not supported. Ever try to back level an operating system? Yeah. Right. Backup, format, reinstall from scratch. Not fun.

Go ahead, tell me that the Mac population is insignificant. Compare to the Wintel world, sure it is. But it's growing. And that means that software and hardware vendors need to pay closer attention. I visited one major financial organization where most of the Director level and above uses Macs. Now put yourself in the shoes of that IT department. If they are using Aventail, and the C-level folks upgrade to 10.5 (Leopard), what's going to happen? Broken VPN connections, upset Directors, and pressure on IT to find a solution. And you know what that solution will be. "Dump Aventail and find something that works with my system!" Of course, IT could always tell the Directors and above they need to move to a Windows machine. Sure. That would work.

IT customer devices are become more and more personalized as the consumer tech world invades the enterprise. This makes IT's support job harder and places pressure on the vendors to make sure a myriad of devices will work when either they upgrade their products or the device manufacturer upgrades theirs. The customers don't care. They just want IT to work, and they will create the pressure necessary on their internal departments that will be translated to vendor buying decisions. The vendor community needs to keep this in mind.

Tough Decision

OK, it’s time for some help on a network decision: you’ve all heard of the “triple play” of voice/video/Internet access from a single provider.  Well, I’m racking my brains because while I’ve actually got the benefit of competition where I live, there don’t seem to be any good or obvious options to get the triple play.

Here’s what I have now:

• DirecTV satellite for television service (signed up back in 1998 because they were the first and only provider with high-def TV, long before over-the-air HDTV became available)
• Cox cable for broadband Internet access (selected because they were the ONLY provider years ago – I live too far from a telco central office to get DSL)
• Verizon for plain old telephone service (again, the only choice available when I first built the house)

What brought this issue to a head is a recent change in DirecTV. The good news is that they’ve just added a dozen more high-def channels, with the promise of over 100 HDTV channels by the end of the year (2007).  The bad news is that these new channels all use MPEG4 compression, and my high-def Tivo DVR, set-top boxes, and 3LNB rooftop dish only work with older MPEG2 channels.  I now need a new larger 5LNB dish and all-new hardware to get channels like National Geographic and Discovery in high definition.  A big downside is that I’ll have to get rid of my beloved high-def Tivo!  That’s right, DirecTV only now offers its own crappy DVRs and set-top boxes, and forces you to pay for them up-front even though they now only lease their hardware.  Subscribers don’t get a choice anymore.

I might have accepted this hardware “downgrade” in order to get the new high-def channels except for the way they’re treating me.  About a year ago due to a lightning strike, one of my set-top boxes stopped working.  Fortunately I had DirecTV’s extra-cost “protection plan,” so they could replace the failed box.  But they replaced my owned/purchased box with a leased/rental box, and then after the fact I discovered I was now under a 2-year contractual commitment to DirecTV!  (This wasn’t the case a few years earlier when a similar set-top box failure occurred – they changed their terms and conditions without bothering to tell me.)

This week, when I asked DirecTV for installation of a new dish and MPEG4-compatible hardware, they want me to pay $400, because I’m still within the 2 years.  (Other customers not under the 2-year lock-in have gotten their hardware changed out for free.) That’s a hell of a way to treat a customer who’s been paying nearly $100/month for 9 years!  Looks like I’ll have to pay a $100 termination fee just to leave DirecTV.

OK, so let’s consider my cable company as an alternative.  Cox Internet service has been pretty good, and because of competition from Verizon FIOS, they keep bumping my bandwidth at no additional cost:  I’m now getting more than 20 Mbps down and 2 Mbps uplink speeds.  And of course they now offer high-def channels and reasonable voice service.  And like all cable operators, by law they must offer CableCard, so that means I can avoid using Cox’s own crappy DVR and buy a CableCard-compatible High-Def Tivo DVR.  I also have a couple of CableCard-compatible HDTVs that would benefit from this technology.

So why not just go with Cox?  Like all cable operators, they’ve become RF bandwidth constrained.  They’re literally out of cable bandwidth and don’t have room on their system to launch all the same new HDTV channels as DirecTV.  So to compete, Cox is one of the first cable operators in the country to deploy “switched” cable service.  Their latest digital set-top boxes actually request which channel they want to watch from the cable head-end system, and then cable bandwidth is allocated dynamically for that channel.

While this is terrific for the cable operator, it means that CableCard (which lacks 2-way communications) is now obsolete, and any new Tivo I bought would not be able to receive any of the new HDTV channels.  This violates the “spirit” of the agreement that cable operators made with the FCC requiring CableCard support, so supposedly the cable industry is working on a USB “dongle” that will permit a Tivo or other CableCard set-top box to request a specific channel.  The problem with this solution is that it could be YEARS before it becomes available and is deployed.

OK, so that leaves Verizon FIOS as the remaining alternative.  Most people in the US would “kill” to have Verizon’s service available where they live, so why shouldn’t I just go with Verizon?  Let me count the ways:  First, as they install the fiber from the street to your house, Verizon always rips out your pre-existing twisted pair copper phone wiring.  This is done to keep you from ever going with a competitor (since Verizon is still legally obligated to resell its copper local loops to competitors).  Unfortunately, because fiber can’t carry electricity, this means that you lose voice dial-tone during power outages unless your battery backup is working.   And where I live, we’ve had multi-day power outages where a battery would certainly fail.

The 2nd problem is that Verizon was very short-sighted in deploying FIOS TV, and didn’t plan for enough “cable” channel capacity.  Their single optical wavelength that delivers all their TV channels actually uses old-fashioned cable TV technology, and Verizon needs to upgrade all of the equipment in its central offices to make room for more channels.  It reportedly will be at least a year before Verizon can begin adding the new HDTV channels being added now by DirecTV and Cox cable.

Assuming this upgrade occurs, Verizon’s current FIOS system does have the advantage of working with CableCard, thus allowing me to install a Tivo (instead of Verizon’s terrible DVR).  And unlike cable operators, Verizon’s optical network should be capable of delivering all the bandwidth needed, without resorting to use of switched channels.  So in a year or so, Verizon FIOS might be a pretty reasonable solution.

Except that in 2-3 years, Verizon has said it will be migrating the entire FIOS system to IPTV!  Yes, the same technology that AT&T and Microsoft have struggled to deploy in the former SBC territories of the US.  This will obsolete any Tivos I might buy, since IPTV is completely incompatible with CableCard technology.  But even worse, Verizon has said its future IPTV system will let it monitor EVERYTHING that its customers watch on TV, in order to tailor or target advertising to them.  Does this sound appealing to you?

On top of everything I’ve already covered, there’s another issue.  The good news is that Tivo has announced that next month it will permit its high-def Tivo DVRs to use customer premises LANs to share recorded programs – something never permitted with DirecTV’s Tivos.  For example, you could watch on your bedroom TV a show recorded on the Tivo down in your home theater.  Except that the cable operators have all started flagging their channels as “copy never” or “copy once” to enforce digital rights management.  This effectively cripples the Tivo networking except for sharing of over-the-air TV shows.

It’s all enough to make me want to give up watching any movies or TV anymore.  Got any suggestions?

- Dave Passmore