Network Monitoring Nightmares

Network Monitoring Systems (NMS) are often cumbersome, ugly, hard to maintain, painful to install, and tedious to configure. In many cases, more energy is spent working around problems with the NMS than actually monitoring the network it's deployed on! And if you think this is a problem that can be solved by purchasing a commercial product, you're in for a real shock. So what do we do about the state of network monitoring?

Background 

First, some background. Not background of network monitoring software- there's far too many of them out there. Instead, this is  background on the problem faced by a typical IT or Operations department. First, there is a network. The network initially has only a few servers used by a few people, and everything hums along. When there's a problem, the person next to you notices and asks you to look into it. Because there are so few devices making this network run, it's easy to find where the problem is and typically easy to fix. But then the network grows. You need more storage, more servers, more switches. Next you get into complex networks designed for maximum stability, redundant connections, high-availability products, load balancers, routers, firewalls, and so on. Before you know it your tiny network has grown into a multi-datacenter goliath and you're spending most of your time trying to put out fires and find faults in your design. And like everything in life, the more complex you make things the harder they are to unravel when you need to find the source of an error.

Now you need a network monitoring system. Some piece of software that can watch each individual component on your networks to make sure they're operating properly. This system also needs to watch services running on servers to make sure they don't fail or return unexpected results. And of course, there are countless scripts and applications running in the background to make sure the plates keep spinning. The picture is simply too large for one person to watch manually, and you'll never catch problems before users are impacted. No matter how good you think you are.

Step 1. - We can do this ourselves!

The first step most administrators take is to write a collection of checks themselves. Most of the time they pick a language they know already and get to working. First they check they can get to the web servers. Childs play for all but the newest admin. Open a connection, retrieve a page, make sure the page downloaded, and we pass. Otherwise we fail. Chalk up a success for the admin team!

But then the web sites change, the file moves, and the test fails even though the server is up and working fine. So the admin checks to make sure any page is returned. Eventually, the site changes again and the page that's being successfully returned is actually an error! Now we think the site is working fine, but it turns out customers have been reporting errors to the support line for the past three hours! This game of cat-and-mouse goes on and on, and the collection of scripts gets larger and larger until finally the administrators decide it's becoming too complex to manage the network monitoring system themselves. Now they start step two- searching for a monitoring product.

Step 2. - Just pick a monitoring system!

Now that the administrators (there are a team of them now) have admitted that their time would be best spent on administration of their company's infrastructure and not on writing a monitoring system, the search for an NMS begins. Because the team is used to doing things manually or modifying the monitoring system they've cobbled together every time there's a change, they will almost certainly pick a product that offers a small feature set and requires lots of manual intervention. This isn't because they want a system that's difficult to use, but rahter it's what they're accustomed to and they don't know any better yet. So they suffer through the initial setup and maintenance.

Eventually, someone will notice a system that offers more features and more automatic functionality exists and it would make the job of monitoring much easier for everyone. Unfortunately, so much effort has been invested in the current solution, and it has collected so much historical data that it's deemed too difficult to switch NMS products. This typically happens several times, and every time the story is the same. But inevitably something so drastic happens- either a failure in the monitoring system, a loss of data, or a lack of expandability- that the team agrees the time has come to once again change monitoring systems.

Step 3. - Maybe we should pay for this?

Once it has been deemed that the monitoring system is critical to the business, a project can be created and actually assigned money. This can go one of two ways depending on what the administration group looks like- completely commercial or mostly commercial.

A completely commercial system would be an HP OpenView or an IBM Tivoli system. These are large packages that have tons and tons of functionality, professional development, lovely graphical views, fantastic charts and graphs, prediction models, event correlation engines, inventory modules, expensive support contracts, and serious system requirements. It's typically not enough to just buy the monitoring system- you need database software to manage all of the information generated, too! And that can add thousands to the price tag. But, as long as there is money in the budget, and the sales people do their jobs, these solutions seem like they have endless capabilities and it's a no-brainer to go with a completely commercial offering. But the price tag is often so high that the sticker shock is insurmountable. Even worse, if you do end up with a completely commercial offering, you quickly find out that you're essentially on your own to write the components that check your environment again! Now you're back to step one, but you're tens or hundreds of thousands of dollars poorer. And you've learned a very valuable and very expensive lesson.

By contrast, a mostly commercial system is typically a product that has a free or free/open source component that is expanded on by the commercial branch. These companies lure you in by offering extra value, product support, training courses, development resources, plugin packs, and things of that nature. Most of the mostly commercial offerings compare themselves directly to the completely commercial products, and sometimes they even offer truly better products. Sometimes they can be rougher around the edges than the highly polished products offered by HP and IBM, but for the most part they offer exactly the same functionality at a reduced cost. But like everything, the buyer needs to beware. The development teams at these companies are usually pretty busy building new features and fixing bugs. Getting new features developed for your organization can be difficult or take a very long time, and this can be a hard lesson to learn when you've spent tens of thousands of dollars on a product that you're beginning to realize does barely more than the product you've just replaced. What's worse is that you're beginning to realize that the features in the commercial offering are barely worth purchasing, and you probably could have used the free offering instead.

Wonderful.

Step 4. - Just settle.

So here we are. Your team has spent several man years trying to solve a problem that the success of your business has created. If you've made it to step three, you probably realized the network monitoring system space is a wasteland of half-baked, half functional solutions. There are countless solutions to choose from, but you now realize all of them have the same short-comings and none of them address the one killer feature you need.

What's worse is that if you've purchased a solution, you no doubt realize that the cost of developing extensions or plugins for the monitoring system is so high that it's effectively out of your reach. So, once again, instead of renewing your support contracts or paying a programmer to write the same things you wrote what feels like forever ago, you decide it's time to look at the market again.

I've been living this nightmare for the past decade. I've used every piece of monitoring software you can imagine, and I've used many of them more than once. Little has changed in the past 10 years, which is amazing! There are few software industries that have the same approach and concepts that they had 10 years ago, but somehow monitoring just seems to become stagnant the instant it's released. They all look the same, they feel the same, and for the most part they all have the same failures and successes. What's worse is that severan newcomers base their products on offerings that haven't seen development in so long that the projects have effectively been abandoned!

So what do you do now? Where do you go from here? Well, if you're like the majority of administration groups, you settle. You'll find a package that does most of what you want, you'll find someone skilled enough to add the functionality you need, and you'll be frustrated every single day you use it. That's the sad state of monitoring, and we're all in the same boat.

Conclusion 

I'd personally like to call on Google, Yahoo, Akamai, Facebook, and other massive networks to tell the rest of us what they use. They must be going through the same pains as startups, and with the skilled people they have onboard they must have found a solution. So what is it? What's their silver bullet? Are they willing to release the code for their tools, to host talks about monitoring, to teach us their ways? I certainly hope so, because almost every group of admins could benefit from their knowledge.

Writing PowerShell Scripts

After using PowerShell for about a year, I've actually come to love it. I'm not saying it's perfect, and I'm not saying it's the best it could be. PowerShell is not as powerful as Perl and it's not as native as BASH, but what it does very well is .Net.

When Microsoft created the .Net framework, they created a fantastically powerful platform that could be easily extended and shaped to fit any need. Like any good Object Oriented programming framework, there was a strong set of basic objects built in that allowed both Microsoft and third parties to build very robust solutions quickly and easily. Nearly all MS products now have .Net assemblies, which add capabilities into the framework for the product being used. This is true not only of products like Office and Visio, but of back-end products like Exchange, PowerPoint, Windows itself, and Active Directory! And PowerShell can access it all.

So, what's the best way to get started with PowerShell? I found that learning the PowerShell language was similar to learning any programming language- have an itch to scratch. What I mean is everyone can write the "Hello World" app. Even people with no programming experience at all can do it, especially in PowerShell. But to really get down to it and learn the ins and outs of a language, you need to be applying it to an actual problem you have.

So, let's see an example. What if you were in charge of a group of servers and you needed to check some statistics from them all. Well, we know that WMI already contains that data and more. And wouldn't you know it, one of the Commandlets in PowerShell gets us an Object containing WMI data! So, let's take a quick look:

$wmidata = get-wmiobject win32_computersystem

That's it! We now have an object that contains the WMI data from win32_computersystem. Sure, you can do this in VBScript so far, but it's not nearly as simple! And we're about to demolish what VBScript can do!

$wmidata | Get-Member

   TypeName: System.Management.ManagementObject#root\cimv2\Win32_ComputerSystem

Ok, so we see that this object is actually a .Net framework data type from the System.Management... assembly. We also see that there's a commandlet called Get-Member in PowerShell, which returns all of the methods, properties, and ScriptMethods an object exposes! At this point, Doc Brown will say "Great Scott"!

Alright, we want to actually do something here, so let's get to it. We want to collect system information for several computers on our network. Because we're admins, and we love being informed...right?

$wmidata = get-wmiobject -computername somecomputer win32_computersystem

"Machine name: " +$wmidata.Name

"Model: " + $wmidata.Model

"Manufacturer: " + $wmidata.Manufacturer

"Logged on user: " +$wmidata.UserName

"Total RAM: " +$wmidata.TotalPhysicalMemory

Our output:

Machine name: somecomputer

Model: OptiPlex 755

Manufacturer: Dell Inc.

Logged in user: MYDOMAIN\auser

Total RAM: 4158242816

Ok, so far this has saved me about 20 minutes of VBScript and it's got me all this data! And here's a neat trick...

"Total RAM: " + $wmidata.TotalPhysicalMemory / 1GB

Yes, you can abbreviate all that conversion math to go between KB, MB, and GB by using those letters.

So, we have a script that can connect to a remote computer (assuming you have permission) and gather WMI data. It then displays that data onto the screen for you. And it took us, what, 2 minutes of typing to get this? Using your imagination, you can probably see that you could wrap this all in a for loop and rip through a list in a matter of minutes. And the fun doesn't stop there!

If you're interested in learning PowerShell, I suggest you take a look at the Microsoft PowerShell site here . It's open to the public, and it's helpful. But you may find that Microsoft's information is a bit terse. Once you get your head around PowerShell, you'll actuall find this kind of information the most helpful. Also, check out the .Net framework documentation...this all applies to PowerShell objects! So, find an itch to scratch and get coding!

Something Funny

I just learned that in order to install the Exchange management tools on a Workstation with Active Directoy Users and Computers on it, you need to install the Windows SMTP service. This in itself is odd and makes me worry, but the oddities don't stop there. Nope. Not even close.

To satisfy the requirements for the Windows SMTP service installation, you also need to install the World Wide Web service (http server), and the Internet Information Services Snap-In. The latter is obvious- Installing the Web server without any way to manage it would be bizarre. But, the fact that the SMTP service REQUIRES a web server is plain stupid!

Good server or workstation management would dictate that the fewer unnecessary services you have installed, the fewer security holes you will have to watch for. And we all know full well that Microsoft's http server is lousy with security problems when it hasn't been installed and maintained properly.

Now, if I want to manage my users' email properties from my workstation, I have to install all of this extra software that is filled with vulnerabilities and potential honey pots for SPAM bots on a network. When you consider this is a management workstation, that's a pretty scary scenario! My PC has fewer checks against it by our security devices, simply by nature of my job! Now, I'm second guessing if just logging into a server every time I need to make a change isn't a bad idea.

Either way, smarten up Microsoft. Seriously. Installing a web server on a mail server is a retarded idea in the first place. And just to make it all worse, the fact that I need to do all of this on a system that shouldn't have EITHER on it...that's unforgivable.

Extents are better

More and more, modern filesystems are moving away from block-based filesystems and moving toward extent-based systems. What the heck does that mean? Well, on a very high level, this means data is stored on disk with less overhead, leading to better performance and more efficient use of disk space. Technically speaking, though, let's break it down...

Block-based storage

Block-based filesystem layouts are traditional, well tested, and old. Really old. The theory behind its operation is very simple- A chunk of data is used to describe a larger chunk of file data is stored. Information such as location, permissions, creation / modification / access time, and where on the disk the actual file data is store are stored in the filesystem block, telling your computer where and how to access file data. The files actual contents are stored elsewhere on disk, typically in chunks of 4KB. Each of these 4KB chunks of file data require a filesystem block, so for a 100MB file you need 25,600 filesystem blocks! Each of these filesystem blocks needs to be read to tell the computer how to read from one end of the file to another. The more your hard disk needs to search around for the location of filesystem and data blocks, the longer this whole process takes. Usually, this all happens very quickly, but there certainly are cases where it can take a very long time.

These filesystem and file data blocks also lead to a phenomenon known as file fragmentation. Simply put, fragmentation is caused by files being modified after they were initially created, or files being created on heavily fragmented disks. Fragmentation itself is simply the separation of file data blocks with regard to each other on disk. To best envision this, imagine going on a scavenger hunt across your town, collecting pages of a book before you could read it. On that scale, it could take you months to re-assemble something like Moby Dick! Don't worry, though. There's a better way!

Extend-based storage

An filesystem extent is much the same as a filesystem block, except that it describes a collection of data bytes instead of strictly sized blocks. In other words, an extent describes a section of a file. All of the same filesystem data is contained in an extent- disk location, file name, etc. But the largest difference is that it also contains the size of the segment of a file's data it describes. So, theoretically, if there is a section of your disk drive that contains 100MB of free space, a single extent could be used to describe a 100MB file! This is so much more efficient than block-based storage that there are very few filesystems not using extents today!

Visualizing it

So, some of you may not be able to envision all of this in your head. I'll be the first to admit the fact that it's weird that I can. For you, I've made graphical representations. In the image below, imagine the green blocks are filesystem blocks- the data that describes your file's contents. The red blocks are the actual data of your file.

Block-based file layout

Here we can see that there are several chunks of data used to describe your file's content, which has also been split up into multiple chunks. Remember that the disk needs to read each green block to know where and how to find a red block.

Now we look at extent-based storage. Again, the green square represents the filesystem data that describes your file's content, which is found in the red squares.

Extent-based file layout

Instantly, you see the stark contrast. There is less data wasted describing your files contents, which are laid out in a more contiguous manner. Since disks read contiguous data faster than data that is scattered around a disk, your benefit is two fold. You have to read less descriptor blocks, and you have to search around the disk fewer times for actual file data.

Wrapping it up

So, what does it all mean to you? Well, it means you can store larger and larger files on your disk drives with less and less overhead. It also means that data can be retrieved in a much quicker manner, and finally it means someone out there cares about how you spend the milliseconds in your life. After all, it's nice to know someone cares, right?

Upcoming Ubuntu 9.04 release

Canonical and the Ubuntu community will release the next version of Ubuntu Linux- codenamed Jaunty Jackalope. This new version will be coming with many new features, and quite a bit of polish on the surface. Some of these changes will cause issues for some people running in certain hardware situations, such as those that require the use of third-party drivers. Many vendors do work hard to get drivers available as soon as possible, and several are already packaged up with the testing releases, but there are still those that drag their heels.

New Features

• Package Updates - As with all distribution version updates, there are a host of packages that get version updates. These range from common utilities to linked libraries. For details about updated packages, take a look at the jaunty-changes mailing list at https://lists.ubuntu.com/mailman/listinfo/jaunty-changes.
• X.Org Server 1.6 - X.org is the underlying graphical system of Ubunbu Linux. Version 1.6 now ships with DRI2 to manage graphics rendering (including 3D), X Input 1.5 which allows automatic detection and configuration of input devices such as mice, keyboards, pens, touch screens, etc., A new pointer acceleration system, and RandR 1.3, which manages the resizing and rotation of screens. In addition to these updates, there are several smaller but no less critical changes. Many of these increase performance and stability across the board, and provide driver updates to more fully support new hardware versions.
• Optimized font sizes - Ubuntu 9.04 will detect your display settings and automatically set your font's dot-per-inch size. This will make text more easily readable and consistently sixed across an array of devices. If you prefer a customized side, you can also manually set this value.
• New notifications - I am personally excited about this feature. Ubuntu now offers a standardized notification framework to other applications, making notification messages appear and behave in a predictable way. To see a preview of these notifications, please check out this example on Mark Shuttleworth's blog.
• Updated kernel - Jaunty will ship with kernel version 2.6.28. This kernel version comes with a long list of newly supported devices, EXT4 filesystem which makes disk access in most cases faster and more reliable, and Intel's new Graphics Execution Manager (GEM), which provides a new system for managing the memory of graphics systems. Naturally, quite a bit of work went into fixing bugs, updating existing drivers, and optimizing performance.
• EXT4 filesystem - The EXT4 filesystem is an update to the EXT3 filesystem, but also so much more. EXT4 has been discussed at length on every Linux forum and benchmarking site, but the breakdown is this. Files are allocated in a new way, which makes creating and deleting significantly faster. This new allocation method also requires less data to describe files themselves. In addition, data is cataloged with 64bit addresses, which means the amount of usable storage is orders of magnitude larger than the number of devices you can attach to your system.

Unfortunately, a few features were cut. Most significant to me, personally, was the encrypted home directory. The basic rundown of this feature was that when you installed Ubuntu, it would ask you for a password and then create an encrypted directory in your home folder. This directory could be used to store sensitive documents, data, and things of the like. If your disk or computer were ever stolen, you didn't have to worry about that information getting into the wrong hands- without your password even the NSA wouldn't be able to access that data. Due to some "outstanding issues", the feature has been removed from the testing releases and will not be put into the final version for download.

In all, Ubuntu has been shaping up very nicely over the past few years that I've been using it. At this point, the system is completely ready for use by any computer user, and that is a claim I will stake my reputation on. There are very few situations where Ubuntu can not fully satisfy any need a computer user has, but just like switching from a PC to a Mac, you need to be ready to change the way you think about using computers.

If you haven't tried Ubuntu yet, I encourage you to head over to www.ubuntu.com, download a copy, and boot up a live-cd. This will allow you to try Ubuntu without installing it on your PC. If you like it, I say back up your data and take the plunge. If you don't, I'd love to hear what you feel is wrong with it.