Network and Telecom Strategies Blog

Green Proxies for Green Networks

Posted by: Eric Siegel

There's an interesting article by Bruce Nordman and Ken Christensen in the latest (January 2010) issue of IEEE Computer (pp. 91-93); it's called "Proxying: The Next Step in Reducing IT Energy Use."

In it, the authors look at the reason that PCs and similar devices require power even when they're not being used, and they focus on the need for the device to maintain a "network presence." Many devices aren't put to sleep because they may need to respond to network probes (e.g., ARP messages) and to commands from a central management system or a P2P peer to receive, for example, system updates or downloaded files outside of business hours.

The central idea is to have a proxy, which could be a low-powered chip within the device, or even a separate appliance that's acting as a proxy for a whole group of devices. When the device wants to go to sleep, it informs the proxy, which then takes over the device's identity. The proxy listens to network traffic, responding appropriately to simple requests (e.g., ARP and topological queries) and waking up the device only if that's absolutely necessary (e.g., to receive a file).

According to the article, Apple introduced proxying within its PCs and access points in mid-2009, and Windows 7 includes the ability to handle network interface chips that can act as simple proxies.

There are also various standardization and research projects underway, such as Ecma's proxying committee standard TC38-TG4 and the joint Microsoft and UC San Diego project Somniloquy.

SNMP Security

Posted by: Eric Siegel

A question recently arose about best practices for SNMP security, so I feel that a quick blog entry about it would be useful.

Here is the simple rule: if you're running SNMP version 1 or SNMP version 2c (the current variant of SNMP version 2; the original version was abandoned because of a flawed security system), then there isn't any security. So don't allow anyone to write to your devices using SNMP, period. Completely disable all facilities on the devices that respond to SNMP write (SET) commands. If you have to write to the devices, then use a secure pathway, such as secure shell (SSH) or a secure XML flow, thereby avoiding SNMP entirely for writing.

For reading/GET functions in SNMPv1 and v2, I'd just give up and allow unencrypted community strings and data, but I still might secure the aggregated data in collectors or a central management platform. I'd use TCP and encryption for all communications between management platforms and between management platforms and SNMP collectors/pollers. I'd disallow any use of SNMP to sniff traffic, because that might reveal user passwords, etc. (And such sniffing is inefficient and awkward anyway!) I'd also see if I could restrict the data sent by sensitive devices (e.g., firewalls) in response to GET commands; e.g., I'd might not let them send their version numbers or configuration data.

An intruder could do a denial-of-service attack by issuing an overwhelming number of read/GET commands to an SNMPv1 or v2 device under these rules, so maybe you should require that devices have a way to throttle incoming SNMP requests to ensure that they aren't overwhelmed. (Many devices do this by ensuring that SNMP processes run at a relatively low priority.)

Having devices and the management agent check IP addresses and hostnames is not a guarantee against intruders, although it makes intrusion more difficult. Therefore, to decrease the probability of an attack on managed devices, the managed devices should be configured to accept commands only from sources in designated subnets (e.g., a management subnet) or from designated IP addresses or address ranges.

And, of course, you must ensure that devices do not include "hidden" community strings (i.e., strings that are embedded in the device, are typically used by the manufacturer, and are usually undocumented and not revealed to the enterprise purchaser of the equipment); these can create severe vulnerabilities.

You should have firewalls block all SNMP commands that are incoming to protected areas. Use remote collectors/pollers to gather SNMP data and push it through the firewall on a secure TCP flow if you have to go through a firewall.

SNMP version 3 is the most secure, robust solution. Unfortunately, its security features use a new security model that was designed to be independent of other security infrastructures. That helped ensure that SNMPv3 could function in a broken network, but the complexity has made many organizations reluctant to deploy SNMPv3. The IETF's isms Working Group is now trying to create an SNMPv3 security model based on SSH over TCP. When it becomes commercially available, the Integrated Security Model for SNMP will be the recommended security solution for SNMPv3.

Building Applications for the WAN

Posted by: Eric Siegel

Our final session at Catalyst this year was a joint presentation between Network and Telecom Strategies and Richard Watson of Burton Group's Application Platform Strategies group. We talked about how programmers and network engineers can work together to improve application performance and problem diagnosis.

This session was based on the NTS report "Designing Applications for the WAN," which will be updated next week -- so go read it when the update comes out (it's out now!), but there are a two points that came out in the session that I think were rather surprising to us network folks and that should be emphasized here.

First, the applications developers are now using frameworks to enable them to develop applications more quickly. Those application frameworks completely hide ("abstract") the network. For example, an application developer will ask a "method" for a "data object." The developer doesn't even realize that the network is involved, or that there may be some SQL queries used by that method to create the data object. That's why when you have a crisis, and you put Wireshark or a Sniffer on the communications link and show the resulting trace to the developer, he or she has no idea what the heck is going on. I'll say it again: the developer didn't even realize an SQL query was generated by the method request, and has no idea whatsoever what that stuff in the query string, trapped by the trace, really means.

No wonder there's frustration and wasted time in diagnosis! The time has arrived for the network organization, the operations group, and the development organization to cooperate and purchase a diagnostic tool that will trace individual transactions through the production system for diagnosis. (An example is Quest Software's PerformaSure; there are others.) The programmers will understand what they're seeing, the network staff won't need to run around doing Wireshark or Sniffer traces for days on end, and the problem will be handled much more quickly.

And, as the second point, realize that performance is poor because of the "abstraction" of the network layer. The application framework has a lot of parameters that set timeouts, window sizes, buffer sizes, compression, whether to disable the Nagle Algorithm, etc. The programmers understand none of this, so they use the defaults. But the defaults aren't usually set by the vendor to ensure optimum performance; they're set to create the fewest trouble calls to the vendor regardless of the network characteristics. That may not be the optimum setting for your particular situation, and that's why the people who install and set up the framework should talk to the network staff to jointly decide how to set the networking parameters for the networks that the framework will be running over. That'll help quite a bit in creating good performance.

Marketeers: Your Websites and Spec Sheets Stink!

Posted by: Eric Siegel

Today, my blog entry is simply a cry of anguish, a plea that someone, somewhere, in the dismal, sleep-walking depths of technology marketing departments will listen to my plea — the plea of an engineer, overworked and overwhelmed, who simply wants TO GET SOME HARD FACTS about your product when developing a "short list" of candidates.

Now, I've got money to spend, and you (theoretically) want some of that. Wouldn't it be nice to get my interest? (Yes, I thought so.) So why are you so incredibly clueless when you create your web sites and marketing literature? Do you ever think about your target audience? I'm not a marketing MBA. I'm not a salesrep. I'm an engineer! I tend to be interested in data, not marketroid foam.

But what do you give me in your data sheet? That's right. Stale marketing foam with lots of hot air and almost no little kernels of technical data. Ugh.

Let's pick on <name deleted to protect the guilty> today. Just what, exactly, does their "<whatever> system diagnostic product" do? I'm unable to figure it out from the datasheet. There are four glossy pages, and here are the translations and my comments: (The quotes are real, not reworded or exaggerated, unfortunately.)

>==== Page 1 ====

Point 1: We do it ALL, brother! We provide a "top-down, end-to-end, collaborative lifecycle approach for seamlessly managing, monitoring, triaging, diagnosing and resolving critical problems with applications — in both pre-production and production environments." Wow! AND we also fix cars, feed the cat, make coffee, put out fires, paint walls, manage group meetings, and clean the cat barf off the carpet when the cat food was too cold! [Look, this kind of yammering gets you nowhere; all it does is trigger cynicism. I've heard it all before, and this tells me absolutely nothing about what your product does, technically. Virtually all vendors describe their products using these words. Being the 99th to do so gets you nowhere. There's zero product differentiation, and I don't even know what market you're in; I have NO IDEA what your product does.]

Point 2: "the architectural complexity with composite applications is increasing and makes monitoring, triaging and diagnostics even more difficult if not impossible—sometimes with random outages and no resolution in sight." Your life is miserable and your systems are screwed up. [OK, happy people don't buy stuff; I know. But why waste your entire first page replaying tales of misery? End of page 1, and I still have NO IDEA what your product does! Other than "everything," of course, including that bit about the cat barf.]

==== Page 2 ====

Point 1: Wow! A cramped picture of four console screens, showing...what? Lots o' wiggly graphs and unreadable text! [Seems to be some sort of response time stuff, indistinguishable from 99 other products. I can't pull my usual stunt of reverse-engineering the product to figure out what it does by looking at the console because these graphs are too generic and unreadable.]

Point 2: "...starts with the end-user business process, then drills down into application components and system layers—helping you rapidly resolve the problems that have the greatest business impact." Starts at the top, drills to the bottom, and MAKES YOU HAPPY! [Whoopee! Again, this is how 99 other products are described. My toes are tingling with engineer joy and I'm digging for my boss's wallet — NOT.]

Point 3: This product is "fully integrated" (whatever that means) with our other indescribably delicious products! [Yay! I can buy a whole suite of stuff that does something, although I still don't know what! (Empty my wallet, maybe?) And excuse me if I don't quite believe that "fully integrated" bit. You probably mean that they both run on the same PC, which is why you're so reluctant to add some technical facts here. And now the second page is over, and I still have NO IDEA what your product does...cat barf....etc.]

==== Page 3 ====

Point 1: Hey, yet another unreadable picture of graphs with squiggly lines and unreadable type! [However, they're different from the second page's squiggly lines and unreadable type; I suppose we're making progress.]

Point 2: Remember that this works with our other products. We just told you that on Page 2, but maybe you forgot. [unprintable]

Point 3: "Key Features and Benefits" -- an assortment of technical-sounding terms we marketing copywriters found in the product's instruction manual. We didn't understand them, so we think they're impressive. [For the first time, there's some real information here! Too bad it's so random and disconnected. A lot of it is how the GUI works, but that doesn't tell me what the system itself does underneath the GUI. Where does the data I'm manipulating come from? That's a mystery that this spec sheet never does answer, and so I still don't know the basic type of product I'm dealing with here. Is it an aggregator of SNMP data? Does it have its own probe mechanism? Does it do "dye tracing" with software shims? I have no idea.]

Point 4: We provide services and support to assist you in figuring out what this product is, and we can also do whatever else needs doing in that smoking ruin of a system that you're responsible for. All you need is money, and lots of it. [Oh, boy.]

==== Page 4 ====

We provide product training. And here are some corporate logos. The End. [I still have no idea what this product does, and so I don't know if I should investigate further. I suppose I should call the sales staff, who will doubtless be just as informative as this datasheet. I'm not looking forwards to a few hours on hold, or talking to uninformed, confused, and confusing salesreps. I hope this isn't a preview of the support I'll get if I buy this pig in a poke (or cat in a sack). Maybe I'll just find another vendor...]

==== ====

Get it? ||: please, please, please :|| tell me what your product does, and how it does it, up front! "This product works by inserting software shims between the Java VM and your applications to track all calls and log their arguments, returned values, and response times. With only five percent overhead, it provides complete tracking of each transaction through your production system, which is an invaluable aid to diagnostic technicians...." You see? Not so difficult after all.

Don't Throw Out Those Network Error Messages!

posted by: Eric Siegel

While I'm out here blogging today, here's an idea from the other area I cover: network management.

Please, programmers, when you get an error message as a result of a communications call, don't just retry retryable errors and then not bother to log the problem! We really need to know what is going on; we really need you to log all errors and messages you get from the communications calls and network systems. Maybe the intermittent error didn't crash your program, but it's out there screwing around with someone else's program nevertheless. We'd really love to be able to see the history of the errors and almost-errors in a log somewhere when the production systems are down, people are climbing the walls, and management is making the usual threats.

And another idea: when you write that error message, please don't just issue some generic message: "Things are screwed up somehow, but I've no real idea what's going on. Some damn pointer somewhere is pointing to Mars, maybe. So I'm just going to toss this transaction into the crapper and reset, or panic, or whatever."

Instead, how about a message that contains two parts: first, a series of numbers that give a detailed description of the error, along with the application ID, the transaction ID, the step you'd just completed in that transaction, the specific identity of the servers, databases and other subsystems that were open, etc. And second, a number that includes the time of day and maybe a token that we can use to match to what's going on in the system log files and monitoring files at that time. The end user can read those numbers to the service desk from the screen when the error message shows up, and we can then sort on the first numbers to see if we're suddenly getting a lot of errors with the same database, or server, or whatever. We can use the last number to look at what was going on in our other log files and in measurement system reports. That'll really help cut down our mean time to repair incidents.

I know that your deadlines are insane and your management is foaming. But think of all the time you'll save the operations staff and yourself (when you get called in to consult on the crisis, as you will). That time savings will doubtless come when you're up to your eyeballs in a different project with different insane deadlines, and you'll be really happy with your nice error messages when they save you lots of time!

Well, end of entry. Thanks for listening!

Cisco the Computer Company, Act III (updated!)

Posted by: Eric Siegel

Hey, it looks as if my predictions may be correct! Today the papers are full of articles about how Cisco is "becoming a computer company." (See, for example, The New York Times and The San Francisco Chronicle)

I've blogged before about this before, ("Cisco the Computer Company" and "Cisco the Computer Company, Act II"), but I've always thought they were going to restrict themselves to becoming the intelligent, coordinating backplane for other vendors' computer systems. As I said, they would be "re-creating the Tandem Computers (HP) NonStop backplane in a heterogeneous server environment."

The articles indicate that Cisco may be aiming to take over the whole hairy, smoking boule of silicon 'n software. I suppose that's what happens when you have lots of hungry salesreps to feed.

But I'm not so sure that's what they'd want to do.

Yes, Cisco has a huge opportunity to pull together the world of virtual systems, ensuring that corresponding processes are located close to each other to decrease latency, that messages among processes are easily traceable and measurable, that even non-Cisco equipment can be integrated into an heterogeneous system bound together by communications paths and inter-process messages, that systems can be managed as a whole.

But Cisco has never shown major leadership in systems management; would they be able to rise to the occasion?

And Cisco's own development of its own principal operating system, IOS, has been, uh, difficult. Yes, they've made some real advances in the past few years, but they would be competing against companies who have been focused on OS development for customers who are programmers, not OS development as a partially-hidden piece of an infrastructure, where software tangles can be somewhat hidden.

So I'm going to stick to my original idea: that Cisco is really aiming at becoming the intelligent, coordinating backplane, where programmers could drop an inter-process message into that backplane (for example, with a SOAP header giving the name of the destination process) and be sure that it would be delivered to that process exactly once.

===== 2 February 2009 UPDATE =====

OK, I see that the press is full of news about how Cisco is going to go into the development of virtual servers, so maybe I should look at this again. Cisco's CTO, Padmasree Warrior, has just posted a video about this as a follow up to her her blog entry on 19 Jan 09.

She says that Cisco sees an opportunity in "breaking down the silos between compute, storage, virtualization and network platforms" to create an integrated system, with the network as the platform.  She wants to break down the "islands of virtulization."

Padmasree says that "phase two, or unified fabric, is the phase which optimizes and extends data center technologies through consolidation of virtualization across the network, storage, and server application. Phase three, or unified computing, virtualizes the entire data center through a pre-integrated architecture..."

How would this be done technically? Well, whenever a new process is created in a VM, it could be packaged inside a software shell that would automatically provide a higher-level set of session-level protocol headers giving the guaranteed identity of the sender, the precise time the message is sent, and a unique transaction and message ID. As I said, we're back to the Tandem Computers NonStop backplane, with Cisco as the unifying part of the ecosystem. As Padmasree says, "the network is the platform."

Network and Systems Management: VMess

Posted by: Eric Siegel

OK, now I'm worrying about how we're going to manage VMs without losing our minds.

You see, I've been hearing great stories about how there are going to be huge racks of servers containing zillions of VMs: virtual servers, virtual switches, virtual routers, virtual firewalls, virtual performance optimization appliances, virtual everything! And the system will instantly adapt to changing workloads, spawning new, fully-configured servers with their complete environment, along with switches, routers, whatever.

And if a newly-spawned virtual server is too many virtual-router hops away from closely-coupled applications or processes, well, then, that new virtual server will be automatically re-provisioned and re-configured closer to where it should be for minimum latency!

Yes, the entire mass of virtual whatevers is going to be in constant, thrilling motion! Everything is going to be moving around, minute by minute! Total efficiency in hardware utilization! Minimum transport latency! Automation! I just can't wait! It's going to be just totally wonderful!

Until it breaks.

And when it does break (and it will), I don't want to be the miserable technician trying to figure out where everything was when that transaction failed, trying to track down precisely what subtle interaction of data flow, firewall, optimization device, VLAN, switch... you get the picture... was the source of the difficulty.

It'll be like yelling "STOP!" six hours too late at a frenzied set of hyperactive squirrels loaded on caffeine and springing among the trees while chattering at each other. Now, where were they six hours ago, when the problem occurred? All the actors in the drama will have melted into thin air. I'll bet the guilty one won't even be there any more; he appeared, messed something up, and then vanished into the virtual fabric, leaving just the rack behind.

What will the vendors give us? Probably only the assurance that the virtual servers and switches and routers will have virtual MIBs, just like the real ones. But I know where the real ones are, and I can put my hand on them and monitor them. What happens if a virtual switch appears, switches a gigabyte or two, and then vanishes -- poof! -- inside of a minute? Where did it go? What did it do while it was alive? We'll be MIB-less in the fabric, without any vision.

I can already see a rule I'd like to impose on this mess, which will tie in with the "triage-based incident management" discussed in my post "Network and Systems Management: Over the Edge." It's this:

"Just because a production group can create a maze of unmeasured real and virtual servers, network components, and interconnections that change from one minute to the next is no reason it should. Edsger Dijkstra wrote a famous letter to the editor titled 'GO TO Considered Harmful' (Communications of the ACM, August 1968) in which he made the case for more program structure and less intertwined process flow. The same applies to IT infrastructure." [This is from my new Burton Group paper, "Using Network Management Platforms for Triage-Based Incident Management."]

I'll bet we network folks are going to be stuck in the middle of this VMess, and a miserable place it will be. We'd better start figuring out how we're going to handle it when incidents occur. And I think that triage orientation, with clearly-defined diagnostic domains and rules to ensure that the virtual servers providing a service are all grouped within the same domain, where we can evaluate them as a group with active measurement techniques, is a good place to start.

Cisco the Computer Company, Act II

Posted by: Eric Siegel

Yes! Cisco is indeed becoming a computer company, as I originally blogged a year ago in "Cisco the Computer Company." (Go look at it!) A little while ago (sorry, late blog entry), I went to the annual Cisco analyst's conference, and it was full of further progress towards making the network into the backplane for a distributed multi-server computer system -- in effect, they're re-creating the Tandem Computers (HP) Integrity NonStop backplane in a heterogeneous server environment.

Just look at what they're doing or planning to do:

• Lossless, extremely-low-latency transport ("lossless Ethernet" and cut-through switching, similar to NonStop's ServerNet low-latency inter-processor buses with "wormhole" switching)
• Positive identification of network users and of individual transactions, along with accurate timestamps. (Great for system audit and diagnosis in a message-based distributed server system)
• Standard API for talking to the network, asking for specific services and learning the identities, locations, and more about other network users.

With a secure, reliable "backplane" of the Cisco network, other manufacturers' servers and appliances could create a mesh of message-based services. And Cisco would provide the common, platform-independent transport API, integrating them all into one high-performance, secure fabric and, at the same time, handling data compression, transport performance optimization, and the measurement and diagnostic taps and tools needed to trace the performance of individual transactions or other interactions through the production system.

The big question, of course, is how this integrates with OS efforts from the computer system vendors, such as VMware and Microsoft, and with transaction monitors. Where is the line between the reliable, secure, "deliver the message exactly once" transport, and the higher-level OS functions? I'm very interested to see how this evolves!

Network and Systems Management: Over the Edge!

Posted by: Eric Siegel

This is the first of what will be a series of discussions on management of complex networks and systems, a problem that's confronting us all. Before I became an analyst, I spent years in freezing server rooms in the middle of crises, listening to lots of threats ("If you don't fix it fast we're going to ...") and gobbling lots of free pizza.

And those were the "good old days"; now, things are worse. Everything is more real-time; networks, appliances, servers, and distributed applications are more intertwined; network and data flow topologies are out of control; nothing is stable; it's almost impossible to reproduce the production system in a test environment. And yet we're expected to detect incidents and repair them as soon as they occur, if not sooner.

It's a nightmare. We have to get a solid grip on how to manage this mess, or we're all doomed to even more stress and stale pizza and late nights.

So let's see if we can come up with a scheme that makes our lives easier without being too expensive or impossible to implement.

I'd like to start by discussing triage-based incident management, which I've been working on and writing about for a few years now. It seems, at least to me, to be a decent alternative to the classical strategy of spending staggering amounts of money on system monitoring solutions that measure everything that's easily measured (and for which the vendor can bill licensing fees!), then hoping that an expensive "expert system" will somehow sort through that overwhelming flood of data to understand what is going on and assist the operations staff.

Triage-based incident management is based on rapid incident classification, and the basic idea is to find out where the problems are and which group is responsible for them—and then to turn the incident over to the responsible group, with sufficient credible evidence to quash any attempts at finger-pointing. The first goal is to determine the seriousness of the incident; the second is to decide which specialist group should handle it.

Incident classification is based on measuring the service delivered across the boundaries of clearly-defined subsystems, such as DNS, queries to back-end databases, transport across network backbones, and transit time through firewalls and load balancer subsystems. It looks primarily at services delivered, not at individual element measurements such as CPU time and queue length.

For incident classification to work well, clear demarcation points need to be identified at the boundaries between responsibility domains, and there should be easily-understood, credible, real-time measurements of the services being delivered across those boundaries. For example, the service desk can instantly see that DNS response time has suddenly jumped to 30 seconds, or that transit time across a critical VLAN has tripled (or the VLAN has vanished entirely!). They'll then know that they have to call the DNS people or the people responsible for the VLAN. They won't know why the service is failing, but they will be able to prove, in a few seconds, that there really are problems and they'll know the appropriate group to handle the issue.

In actual use, the service desk uses end-to-end performance metrics, such as total transaction response time or VoIP voice quality, to detect incidents, then uses other service measurements to instantly sort the incident into one of three groups:

• A well-known incident with a standard fix (e.g., reboot the server that runs some packaged software that hangs occasionally and that seems to restart without any further problems)

• A new incident that clearly is being caused by problems in a single "responsibility domain" (e.g., a DNS problem is slowing down all transactions, or a slow identity database response is slowing all new logons, so it's obvious which responsible organization gets the trouble ticket)

• A new incident that doesn't clearly fall into a responsibility domain (e.g., some subtle interaction among system elements is causing a confusing set of symptoms; diagnostic procedures should be started and higher-level diagnosticians should be notified)

Within each responsibility domain there can be additional, lower level "diagnostic domains" with the same types of clearly-defined demarcation boundaries and measurements to assist in rapid isolation of the incident to a particular subsystem.

In effect, all this is just what I call "pre-planning your next meltdown." After all, what do we do when we're confronted with a major crisis in the server room (after the obligatory freakout and power-cycling of every box we can find, of course)? We typically start putting in measurements or trying to find existing measurements to isolate the problem to a subsystem. Although we usually don't know what the performance is like when the thing is working, we can guess that response time is not some insanely-long value.

What we're doing with triage-based incident management is just pre-designing the measurements we'll use to diagnose system problems, but we're doing it early, so we can get the benefit of a baseline when the thing actually fails, and we don't have to do it while listening to threats from top management.

Well, that's a start. I'm going to keep writing on this topic, and I'd love to see your comments. What are YOU doing to manage the mess?  Does triage-based incident management seem reasonable?

(Burton Group subscribers, see: "A Framework for Network Incident Management" and the 2007 Catalyst conference presentation "Pre-Plan Your Next Network Meltdown" for much longer discussions of these ideas.)

Cisco the Computer Company

posted by: Eric Siegel

Let's take a few moments and think about what Cisco is doing with its AON (Application-Oriented Networking).

Cisco has already sold a router to every organization, enterprise, man, woman, child, and household pet in the entire known universe. Now its hungry army of salespeople need new markets, and "moving up the protocol stack" into the world of applications is an obvious direction.

AON has the potential to be much more than a quick move into a few hot applications markets, however. There's some fascinating technological history and concepts behind Cisco's recent moves. It seems to me that Cisco is beginning to re-create the Tandem Computers message-based operating system, Guardian, in a heterogeneous world.

The Tandem Computers system, which is still very successful and is sold by HP as the "NonStop" computer system, is a loosely-coupled, linearly-scalable, fault-tolerant highly-integrated set of hardware, OS, transaction monitor, and SQL database. It was originally designed for transaction processing but is now expanding into other areas that can use massive scalability and fault-tolerance with a UNIX and SQL programmer interface. One of the keys to its success is its message-based OS kernel, which guarantees to deliver any message exactly one time to its destination and to return a reply exactly one time to the message's originator. The message destination is a process name (which can be an application process or a systems process, such as a database or communications link), and there can be dozens, or hundreds, of identical processes in different processors sharing the same name and therefore sharing the workload. The OS distributes the workload over the processes, and it also handles recovery from failed processes. If a process hasn't successfully returned a reply to a message's originator, the OS automatically ensures that another process with the same name gets the work and that there isn't any database corruption. (I worked at Tandem Computers many years ago, before it became part of Compaq and then part of HP.)

Let's see what Cisco is doing, and compare it to the NonStop systems.

First, we need a message-based set of standards for applications to talk to one another. What a surprise! It seems that XML and SOAP are already standards, and they're already widely adopted. They can be used to send messages, and they have standardized internal metadata that can be used to find the information and context for load balancing. A load balancer (or Cisco device!) can look for a few key internal message fields for choosing a server; there's no need to understand everything in the message.

With XML and SOAP, it's easy build a general-purpose message switch that looks for a few internal fields. Note the contrast with raw TCP/IP, which is a stream of unformatted data. Doing the same thing with raw TCP/IP would require that the organization build its own custom internal format and program its own message switches to handle that format.

Second, we need to load-balance among multiple members of the same class. This capability is already provided by WCCP Version 2 and other load-balancing systems, especially for context-free messages, which is what we're talking about here.

Third, we need to provide the ability to return the reply to the originator of the message. Again, a Cisco device could easily do that.

And fourth, we need to guarantee delivery exactly once. This is more complex, but not insurmountable. Using SOAP header fields, the Cisco devices could embed a guaranteed-unique message identifier that Cisco devices would use to check that each message is actually delivered once, to suppress unwanted duplicates, and to re-deliver messages in a controlled way if the initial server fails before generating a reply.

There's more that should be done, such as the integration of a SQL database and transaction commit/rollback capabilities, which would provide fault-tolerant updates of a database despite server failure. But, clearly, it's possible for Cisco to start building the equivalent of the HP NonStop kernel on Cisco equipment.

And look at the result: Cisco would own the infrastructure underlying a massive distributed-processing system, which could use processors and operating systems from many different vendors. They'd need to use XML and SOAP, but they're already doing that! There would be a requirement that applications be context free and that replies repeat the transaction identifier, but that's minor. And many systems are already heading in that direction.

Cisco would be perfectly positioned to provide more than guaranteed-delivery and load balancing. It could also provide QoS prioritization and classification, security authentication and encryption, and -- because it would then know the security keys -- it could also provide detailed measurement, diagnostic transaction tracing (with synchronized clocks and records of the transaction IDs, of course!), logging, WAN performance optimization, and XML message and protocol transformation.

Cisco wouldn't need to get involved in the complexities of the operating systems and applications, although it would be sensible to build, or standardize on, a distributed-architecture SQL database and transaction monitor. Cisco could leave applications processors, operating systems, and applications to existing vendors.

And so, after enumerating all of these thoughts, I asked John Chambers at the Cisco Networkers conference if he was remaking Cisco as a distributed-computer company.

His answer? "Yes."