Network and Telecom Strategies Blog

Please Install Network Taps!

Posted by: Eric Siegel

(I see that today I'm just full of exclamation marks in my titles!)

Anyway...

I have a report coming out in a few weeks, "Network Tapping and Flow Tracing Tools" and a portion of it discusses network taps and "SPAN" (Switched Port Analyzer) ports, which are also called "mirror" or "monitor" ports.

A monitor port is a port on a switch, router, virtual machine processor, or other device that can be configured to copy (or “mirror”) the data from another of the device’s ports or from an otherwise inaccessible data path inside the device. A diagnostic or measurement tool can then be plugged into the monitor port to examine the data flow.

In contrast, a tap is not embedded in network or host equipment. Instead, it's a low-cost physical device that passively, and invisibly, intercepts traffic on a WAN or LAN link. Taps can be electrical, for copper WAN links and Ethernet, or they can be optical, splitting some of the light energy from an optical fiber.

Most enterprises don't install taps; instead, they use the existing monitor ports to connect monitoring equipment and intrusion detection and management equipment. This is a mistake.

While I was talking with tap vendors over the past few weeks, I was struck by how often they commented that enterprises just don't focus on the need for taps. So here are a few reasons that the small investment in a tap infrastructure pays BIG dividends, especially when you have a intrusion incident or diagnostic crisis: 

• Taps do not drop packets. Monitor ports may drop frames, especially under heavy load, because they handle frame monitoring at a lower priority than their primary function. This can be irritating, because it always seems to happen precisely when your network is being flooded by a problem or intrusion.

• Taps do not remove errored frames (e.g., short frames, jabbers); the diagnostic tool can see exactly what is on the communications link, errors included. Monitor ports don't see errored frames because they're removed as they enter the switch or router, long before the monitor port sees them. If errored frames are causing congestion, your diagnostic tools won't see them at the monitor port.  

• Most taps can multicast to multiple output ports, allowing more than one diagnostic tool to see the data flows. That's extremely useful! It's wildly irritating to race into a server room with screaming managers on your tail, and then discover that some mystery cable is plugged into the switch's sole monitor port. What does that cable do? If I unplug it, so that I can plug in my trace tool, what is going to break? Aaargh. Time wasted, tracing cables to figure out what is going on. It's much nicer to have an extra port on the tap! 

• Some taps can filter the traffic, decreasing the volume that the diagnostic tool must cope with. They can provide that function without using a network device’s limited processing capabilities.

• Some taps can combine traffic from multiple sources into one output port for the monitoring tool, possibly adding source identification and hardware time stamping to the frames.

Well, there's a lot more about taps and monitor ports -- and about matrix switches, storage, and trace tools, etc. But you'll have to wait for the report! (If you desperately need it now, and you're a Burton Group client, send me an email and I'll give you a pre-publication copy, of course.)

SaaS network diagnostics

Posted by: Eric Siegel

And while I'm on a SaaS chain-of-thought, one final discussion for the day: diagnostics.

What will you do when users at a branch office or out in the field call and complain about SaaS performance? Your operations center needs a structured checklist to follow to quickly manage the incident and isolate it to the responsible party -- a process known as "triage." (See, for example, the Burton Group report "A Framework for Network Incident Management.")

Here's where your SLA key performance indicators, or service level indicators (SLIs) can be useful. Your SaaS provider should be giving you on-line access to their performance as seen by users in their server room, and that's clearly the first checkpoint for the operations staff. If that's in trouble, well, the problem is easy to locate. But if that's OK, then the next step is to look at your service level indicators at your branch offices, especially the branch office that's reporting trouble. If that indicator is in trouble (and that indicator may be a full SaaS web page download time, or it may be only the TCP round-trip time and DNS lookup time for the SaaS provider), then, if you have a solid SLA, you should be able to send that evidence to the SaaS provider and insist on their opening a trouble ticket. It's not really your job to diagnose the problems between your branch office and the SaaS server room; it's the SaaS provider's job. And they probably have more leverage with their Internet service provider (ISP) than you do; they can even demand better peering with the ISP that your branch office is using.

Of course, it may turn out that it's your ISP having internal problems. In that case, you and the SaaS provider may need to explore new ways of connecting to the SaaS server site, through better peering at a point that doesn't encounter the congestion within your ISP, by a direct link to your major offices outside of the Internet, or by your moving to a different ISP for your own access (the threat of doing that might inspire your ISP to upgrade its systems).

Or it may be that it's your user who is having problems with his own browser or with the branch office's internal network; you'll know that because your service level indicators won't see any problems. In that case, it's an incident to be handled by your service desk; there's no need to file a trouble ticket with the SaaS provider. (And your relationship and credibility with your SaaS provider will improve when you're not filing minor reports that turn out to be inside your own system!)

Don't Throw Out Those Network Error Messages!

posted by: Eric Siegel

While I'm out here blogging today, here's an idea from the other area I cover: network management.

Please, programmers, when you get an error message as a result of a communications call, don't just retry retryable errors and then not bother to log the problem! We really need to know what is going on; we really need you to log all errors and messages you get from the communications calls and network systems. Maybe the intermittent error didn't crash your program, but it's out there screwing around with someone else's program nevertheless. We'd really love to be able to see the history of the errors and almost-errors in a log somewhere when the production systems are down, people are climbing the walls, and management is making the usual threats.

And another idea: when you write that error message, please don't just issue some generic message: "Things are screwed up somehow, but I've no real idea what's going on. Some damn pointer somewhere is pointing to Mars, maybe. So I'm just going to toss this transaction into the crapper and reset, or panic, or whatever."

Instead, how about a message that contains two parts: first, a series of numbers that give a detailed description of the error, along with the application ID, the transaction ID, the step you'd just completed in that transaction, the specific identity of the servers, databases and other subsystems that were open, etc. And second, a number that includes the time of day and maybe a token that we can use to match to what's going on in the system log files and monitoring files at that time. The end user can read those numbers to the service desk from the screen when the error message shows up, and we can then sort on the first numbers to see if we're suddenly getting a lot of errors with the same database, or server, or whatever. We can use the last number to look at what was going on in our other log files and in measurement system reports. That'll really help cut down our mean time to repair incidents.

I know that your deadlines are insane and your management is foaming. But think of all the time you'll save the operations staff and yourself (when you get called in to consult on the crisis, as you will). That time savings will doubtless come when you're up to your eyeballs in a different project with different insane deadlines, and you'll be really happy with your nice error messages when they save you lots of time!

Well, end of entry. Thanks for listening!

Network and Systems Management: VMess

Posted by: Eric Siegel

OK, now I'm worrying about how we're going to manage VMs without losing our minds.

You see, I've been hearing great stories about how there are going to be huge racks of servers containing zillions of VMs: virtual servers, virtual switches, virtual routers, virtual firewalls, virtual performance optimization appliances, virtual everything! And the system will instantly adapt to changing workloads, spawning new, fully-configured servers with their complete environment, along with switches, routers, whatever.

And if a newly-spawned virtual server is too many virtual-router hops away from closely-coupled applications or processes, well, then, that new virtual server will be automatically re-provisioned and re-configured closer to where it should be for minimum latency!

Yes, the entire mass of virtual whatevers is going to be in constant, thrilling motion! Everything is going to be moving around, minute by minute! Total efficiency in hardware utilization! Minimum transport latency! Automation! I just can't wait! It's going to be just totally wonderful!

Until it breaks.

And when it does break (and it will), I don't want to be the miserable technician trying to figure out where everything was when that transaction failed, trying to track down precisely what subtle interaction of data flow, firewall, optimization device, VLAN, switch... you get the picture... was the source of the difficulty.

It'll be like yelling "STOP!" six hours too late at a frenzied set of hyperactive squirrels loaded on caffeine and springing among the trees while chattering at each other. Now, where were they six hours ago, when the problem occurred? All the actors in the drama will have melted into thin air. I'll bet the guilty one won't even be there any more; he appeared, messed something up, and then vanished into the virtual fabric, leaving just the rack behind.

What will the vendors give us? Probably only the assurance that the virtual servers and switches and routers will have virtual MIBs, just like the real ones. But I know where the real ones are, and I can put my hand on them and monitor them. What happens if a virtual switch appears, switches a gigabyte or two, and then vanishes -- poof! -- inside of a minute? Where did it go? What did it do while it was alive? We'll be MIB-less in the fabric, without any vision.

I can already see a rule I'd like to impose on this mess, which will tie in with the "triage-based incident management" discussed in my post "Network and Systems Management: Over the Edge." It's this:

"Just because a production group can create a maze of unmeasured real and virtual servers, network components, and interconnections that change from one minute to the next is no reason it should. Edsger Dijkstra wrote a famous letter to the editor titled 'GO TO Considered Harmful' (Communications of the ACM, August 1968) in which he made the case for more program structure and less intertwined process flow. The same applies to IT infrastructure." [This is from my new Burton Group paper, "Using Network Management Platforms for Triage-Based Incident Management."]

I'll bet we network folks are going to be stuck in the middle of this VMess, and a miserable place it will be. We'd better start figuring out how we're going to handle it when incidents occur. And I think that triage orientation, with clearly-defined diagnostic domains and rules to ensure that the virtual servers providing a service are all grouped within the same domain, where we can evaluate them as a group with active measurement techniques, is a good place to start.

Cisco the Computer Company, Act II

Posted by: Eric Siegel

Yes! Cisco is indeed becoming a computer company, as I originally blogged a year ago in "Cisco the Computer Company." (Go look at it!) A little while ago (sorry, late blog entry), I went to the annual Cisco analyst's conference, and it was full of further progress towards making the network into the backplane for a distributed multi-server computer system -- in effect, they're re-creating the Tandem Computers (HP) Integrity NonStop backplane in a heterogeneous server environment.

Just look at what they're doing or planning to do:

• Lossless, extremely-low-latency transport ("lossless Ethernet" and cut-through switching, similar to NonStop's ServerNet low-latency inter-processor buses with "wormhole" switching)
• Positive identification of network users and of individual transactions, along with accurate timestamps. (Great for system audit and diagnosis in a message-based distributed server system)
• Standard API for talking to the network, asking for specific services and learning the identities, locations, and more about other network users.

With a secure, reliable "backplane" of the Cisco network, other manufacturers' servers and appliances could create a mesh of message-based services. And Cisco would provide the common, platform-independent transport API, integrating them all into one high-performance, secure fabric and, at the same time, handling data compression, transport performance optimization, and the measurement and diagnostic taps and tools needed to trace the performance of individual transactions or other interactions through the production system.

The big question, of course, is how this integrates with OS efforts from the computer system vendors, such as VMware and Microsoft, and with transaction monitors. Where is the line between the reliable, secure, "deliver the message exactly once" transport, and the higher-level OS functions? I'm very interested to see how this evolves!

Network and Systems Management: Over the Edge!

Posted by: Eric Siegel

This is the first of what will be a series of discussions on management of complex networks and systems, a problem that's confronting us all. Before I became an analyst, I spent years in freezing server rooms in the middle of crises, listening to lots of threats ("If you don't fix it fast we're going to ...") and gobbling lots of free pizza.

And those were the "good old days"; now, things are worse. Everything is more real-time; networks, appliances, servers, and distributed applications are more intertwined; network and data flow topologies are out of control; nothing is stable; it's almost impossible to reproduce the production system in a test environment. And yet we're expected to detect incidents and repair them as soon as they occur, if not sooner.

It's a nightmare. We have to get a solid grip on how to manage this mess, or we're all doomed to even more stress and stale pizza and late nights.

So let's see if we can come up with a scheme that makes our lives easier without being too expensive or impossible to implement.

I'd like to start by discussing triage-based incident management, which I've been working on and writing about for a few years now. It seems, at least to me, to be a decent alternative to the classical strategy of spending staggering amounts of money on system monitoring solutions that measure everything that's easily measured (and for which the vendor can bill licensing fees!), then hoping that an expensive "expert system" will somehow sort through that overwhelming flood of data to understand what is going on and assist the operations staff.

Triage-based incident management is based on rapid incident classification, and the basic idea is to find out where the problems are and which group is responsible for them—and then to turn the incident over to the responsible group, with sufficient credible evidence to quash any attempts at finger-pointing. The first goal is to determine the seriousness of the incident; the second is to decide which specialist group should handle it.

Incident classification is based on measuring the service delivered across the boundaries of clearly-defined subsystems, such as DNS, queries to back-end databases, transport across network backbones, and transit time through firewalls and load balancer subsystems. It looks primarily at services delivered, not at individual element measurements such as CPU time and queue length.

For incident classification to work well, clear demarcation points need to be identified at the boundaries between responsibility domains, and there should be easily-understood, credible, real-time measurements of the services being delivered across those boundaries. For example, the service desk can instantly see that DNS response time has suddenly jumped to 30 seconds, or that transit time across a critical VLAN has tripled (or the VLAN has vanished entirely!). They'll then know that they have to call the DNS people or the people responsible for the VLAN. They won't know why the service is failing, but they will be able to prove, in a few seconds, that there really are problems and they'll know the appropriate group to handle the issue.

In actual use, the service desk uses end-to-end performance metrics, such as total transaction response time or VoIP voice quality, to detect incidents, then uses other service measurements to instantly sort the incident into one of three groups:

• A well-known incident with a standard fix (e.g., reboot the server that runs some packaged software that hangs occasionally and that seems to restart without any further problems)

• A new incident that clearly is being caused by problems in a single "responsibility domain" (e.g., a DNS problem is slowing down all transactions, or a slow identity database response is slowing all new logons, so it's obvious which responsible organization gets the trouble ticket)

• A new incident that doesn't clearly fall into a responsibility domain (e.g., some subtle interaction among system elements is causing a confusing set of symptoms; diagnostic procedures should be started and higher-level diagnosticians should be notified)

Within each responsibility domain there can be additional, lower level "diagnostic domains" with the same types of clearly-defined demarcation boundaries and measurements to assist in rapid isolation of the incident to a particular subsystem.

In effect, all this is just what I call "pre-planning your next meltdown." After all, what do we do when we're confronted with a major crisis in the server room (after the obligatory freakout and power-cycling of every box we can find, of course)? We typically start putting in measurements or trying to find existing measurements to isolate the problem to a subsystem. Although we usually don't know what the performance is like when the thing is working, we can guess that response time is not some insanely-long value.

What we're doing with triage-based incident management is just pre-designing the measurements we'll use to diagnose system problems, but we're doing it early, so we can get the benefit of a baseline when the thing actually fails, and we don't have to do it while listening to threats from top management.

Well, that's a start. I'm going to keep writing on this topic, and I'd love to see your comments. What are YOU doing to manage the mess?  Does triage-based incident management seem reasonable?

(Burton Group subscribers, see: "A Framework for Network Incident Management" and the 2007 Catalyst conference presentation "Pre-Plan Your Next Network Meltdown" for much longer discussions of these ideas.)