Network and Telecom Strategies Blog

Network and Systems Management: VMess

Posted by: Eric Siegel

OK, now I'm worrying about how we're going to manage VMs without losing our minds.

You see, I've been hearing great stories about how there are going to be huge racks of servers containing zillions of VMs: virtual servers, virtual switches, virtual routers, virtual firewalls, virtual performance optimization appliances, virtual everything! And the system will instantly adapt to changing workloads, spawning new, fully-configured servers with their complete environment, along with switches, routers, whatever.

And if a newly-spawned virtual server is too many virtual-router hops away from closely-coupled applications or processes, well, then, that new virtual server will be automatically re-provisioned and re-configured closer to where it should be for minimum latency!

Yes, the entire mass of virtual whatevers is going to be in constant, thrilling motion! Everything is going to be moving around, minute by minute! Total efficiency in hardware utilization! Minimum transport latency! Automation! I just can't wait! It's going to be just totally wonderful!

Until it breaks.

And when it does break (and it will), I don't want to be the miserable technician trying to figure out where everything was when that transaction failed, trying to track down precisely what subtle interaction of data flow, firewall, optimization device, VLAN, switch... you get the picture... was the source of the difficulty.

It'll be like yelling "STOP!" six hours too late at a frenzied set of hyperactive squirrels loaded on caffeine and springing among the trees while chattering at each other. Now, where were they six hours ago, when the problem occurred? All the actors in the drama will have melted into thin air. I'll bet the guilty one won't even be there any more; he appeared, messed something up, and then vanished into the virtual fabric, leaving just the rack behind.

What will the vendors give us? Probably only the assurance that the virtual servers and switches and routers will have virtual MIBs, just like the real ones. But I know where the real ones are, and I can put my hand on them and monitor them. What happens if a virtual switch appears, switches a gigabyte or two, and then vanishes -- poof! -- inside of a minute? Where did it go? What did it do while it was alive? We'll be MIB-less in the fabric, without any vision.

I can already see a rule I'd like to impose on this mess, which will tie in with the "triage-based incident management" discussed in my post "Network and Systems Management: Over the Edge." It's this:

"Just because a production group can create a maze of unmeasured real and virtual servers, network components, and interconnections that change from one minute to the next is no reason it should. Edsger Dijkstra wrote a famous letter to the editor titled 'GO TO Considered Harmful' (Communications of the ACM, August 1968) in which he made the case for more program structure and less intertwined process flow. The same applies to IT infrastructure." [This is from my new Burton Group paper, "Using Network Management Platforms for Triage-Based Incident Management."]

I'll bet we network folks are going to be stuck in the middle of this VMess, and a miserable place it will be. We'd better start figuring out how we're going to handle it when incidents occur. And I think that triage orientation, with clearly-defined diagnostic domains and rules to ensure that the virtual servers providing a service are all grouped within the same domain, where we can evaluate them as a group with active measurement techniques, is a good place to start.

Flight Simulator - Boeing Style

Boeing's 787 Dreamliner just keeps making news lately, and not all of it is good. First it was production delays, and now the news that the entertainment and flight control networks are connected and there is the possibility, however remote that a passenger could take control of the aircraft. While this is distressing news in itself, it speaks to a deeper issue within our industry, mainly that security always seems to take a back seat in our work.

In my conversations with companies in the petroleum, pharma, power generation, and other industries where networks are used for process command and control, one common thread is "how do I keep critical networks isolated from the admin side?" The answer is simple: Don't connect them. Maintain a physical "air gap" between these networks and there will be no way for someone outside the company to penetrate and subvert them.

Likewise, limit the use of wireless. I was speaking with a couple companies and they wanted their process control to go wireless because it was cheaper to upgrade sensors and replace cabling. I told them they could do all of that they wished, but that the Big Red Button that shut the thing down must have a nice, thick, armored cable to ensure it could not be interfered with when the time came to do a rapid, assured cut off of the processes.

Security and convenience have battled each other since Samuel F. B. Morse tapped out "What God hath wrought" on the first telegraph. Ever since I was security architect at McDonald's Corp., I have chanted the mantra that security must be consistent and unobtrusive to be effective. We can now add in a factor for safety and cost.

Voice and data traffic are now commingled on the same wire in order to cut costs and gain flexibility. We are aware of the security implications, and there are steps we can take to protect that traffic against unauthorized interception, but those steps came two or three versions after the initial deployments and the risks were identified.

Wireless Equivalent Privacy (WEP) is another example. As time has passed, WEP has come to mean "won't ever protect" as a string of vulnerabilities and exploits have been found in that protocol. The IEEE has since addressed these vulnerabilities, but not until they were forced into it by lagging wireless LAN sales.

And it's not just network engineers. A 2002 survey found a notable lack of concern about privacy among the Disappearing Computer Project designers. Privacy was either an abstract problem, not a problem yet, not their problem, or a problem for “society.” This kind of attitude fosters the creation of services and devices that lack even the most rudimentary security or privacy controls.

There are a lot of really smart people working on these things, and I am constantly in awe of what my this industry has built and continues to build. If it is not a question of skill, the question in dire need of an answer is: Why weren't these things thought out more fully during the design?

It is incumbent on all of us as professionals to take a long, hard look at our network designs, indeed, at the overall design of our entire technical infrastructure from inception to retirement and ensure we are doing all we can to identify risks and mitigate those risks within the context of economic practicality. But economics must give way to safety, just as, occasionally, convenience must give way to security.

Network security has long been an afterthought or suborned to economics. Let us as network managers and engineers strive to end this trend and build in the mechanisms needed to ensure safe, secure communications in our designs.