federalcto.com The Blog of a Federal CTO


Your Directory Has Been Breached – landing page

This is a landing page for a project Bob Bobel and I have been working on recently. Today (May 30th, 2012), I am presenting the first draft of the slides which will be posted shortly. A white paper is due out shortly. All information will be posted on this page, so it will be revised in the coming weeks.

(2012-05-30 - 13:00 ET)
First up, here is Bob's initial post on the topic:

(2012-05-30 - 16:15 ET)
Here is the slide deck just presented at the Department of Energy NLIT Conference (National Laboratories IT Summit):

(2012-06-05 - 18:00 ET)
Here is the first (rough) draft of the document to correspond with the slides. Yes, whole sections are missing, and will be filled in during the coming weeks. Stayed tuned.



US Government Smartcards; CAC, PIV and PIV-I

Recently, I had the pleasure of trying to get some government certified smart cards for some of the technical people at Quest, and I can't believe how much of a headache and hassle it was. I actually don't work for Quest Software, but a subsidiary (Quest Software Public Sector) which is focused on the public sector space. And while most Quest employees don't have a need for government-issued or government approved smart cards, our company does. And while I knew that non-government employees can get a relatively new (a little over a year old) flavor of the PIV card called PIV-I, I was amazed at how difficult the process was to navigate. Thankfully, we have some pretty good and persistent purchasing folks, but the process is still pretty arduous for an organization that has been working with the government for years.

I won't get into all the details, but if you're interested, feel free to contact me offline.

In any case, I find myself constantly having to explain what the different is between a smart card, a CAC, a PIV card, and now a PIV-I card. A smart card is pretty straight forward - it's a generic term, and all the other cards fall into this category. You can find a lot more details on them here at Wikipedia. In any case, the thing that makes a smart card a CAC (which means Common Access Card, so please don't say, "CAC Card" as it is redundant) is that it is used by the US DoD (Department of Defense). If you want to do any work on government systems in the military, you will most likely need one of these.

Like their military counterparts, employees within civilian agencies also need smart cards. Of course, they opted for a similar, but not the same, standard. That standard is a PIV card (Personal Identification Verification). The cards are slightly different from CACs, and have varying information printed on them, depending on the issuing agency. Plus, they use a different set of CA (Certificate Authority) servers than the ones that CACs use, as the DoD have their own servers.

Finally, and this is the confusing thing, there are PIV-I cards. PIV-I stands for PIV-Interoperable. There are some great docs about PIV and PIV-I at www.idmanagment.gov which is a site run by GSA but I'll save you the trouble of wadding through a lot of documentation. The PIV-I FAQ here states:

2.2    What is the difference between a PIV-I Card and a PIV Card?
The term “PIV Card” may only be used to describe an identity card that is fully conformant with Federal PIV standards (i.e., FIPS 201 and related documentation). Only a Federal entity is capable of fully meeting these standards and issuing a PIV Card. A PIV-I Card meets the PIV technical specifications of NIST SP 800-73 and is issued in a manner that may be trusted by Federal Government Relying Parties, but does not meet all of the requirements of FIPS 201.

What does this mean? A Federal entity (aka employee) uses a PIV card, and a trusted, non-government entity has to use a PIV-I card.

So there you go. In summary:

  • CAC is for Department of Defense users
  • PIV is for civilian users working for the Federal government
  • PIV-I is for non-Federal entities that need to access government systems

Smartcards, certificates & OS X: Does Lion Roar or Meow?

I spent some time the week before last dealing with smartcards, certificates, Macs and OS X Lion (10.7), specifically. My initial take on Lion was this; I can't believe Apple has taken this direction for the enterprise user.

Before going any further, I should come clean and say I love Apple products. As a consumer, they do what I need, and work as expected. The fact that I've not had to rebuild my wife's MacBook in it's entire life (of more than 3 years) is remarkable. That all changed with OS X Lion. I downloaded it, and was not impressed from the get go. Just starting with the fact that I had to hunt down the DMG file so I would have a copy was irritating, and things continued from there. But I work with technology, need to keep up, and wanted to check out the buzz.

I also did some initial research, but articles like this one didn't help; http://www.macworld.com/article/161493/2011/08/lion_enterprise.html . This guy clearly doesn't run an "enterprise shop" and probably considers an office of 10-15 people to be representative of an enterprise. But I bit the bullet, upgraded my personal MacBook Pro, and have been wincing since. But that was as an end-user. On the enterprise side, things turned out to be worse. The AD Plug-in - broken. The Login Screen - completely screwed up. Apple broke everything when it came to Directory Services and the Login page. And to top it all off, smart card support has been deprecated! Really? This is enterprise ready?

Before we go any further, here's a quick review of how smart cards worked in OS X prior to 10.7:

  1. A card is inserted into the reader
  2. The reader uses the tokend file (ships with the OS) as a driver to read the certificate off the card
  3. The login screen is re-drawn, the username is pulled off the card (to display on the screen), and the screen is changed to only display the PIN request field
  4. The user enters the PIN tied to the card
  5. The certificate (issued by a Certificate Authority aka CA) is confirmed to be valid
  6. The CA's certificate is also checked, and confirmed to be valid, so it's certificate (and any up the chain) have to be in the OS X Keychain
  7. The user is allowed to log in

Apple no longer ships tokend files (step #2), and the default login window is no longer refreshed by a smartcard insert (though it does still flicker). We have since had a very helpful conversation with an Apple architect, and will be making changes to make it all work properly in Lion, but in the mean time, it's possible to get it all working with some elbow grease and lowered expectations.

So . . . how did we do it? Well, we started with the QAS Smartcard guide that you can find with the standard QAS documentation. And there's a whole lot of good stuff there about smartcards and certs, but here's what I boiled it down to:

1. Go to http://smartcardservices.macosforge.org/ and download the latest package to get the tokend needed to read the specific cards you are using. Apple has open sourced it all, and are relying on non-Apple folks to maintain this. If your card is not there, you will need to ask your smartcard vendor to give you a tokend file (basically, a little library to allow you to work against the specific card). We were lucky enough to be using CACs so the tokend we needed was in the beta2 batch from Aug 19, 2011.

2. Once the tokend is installed, and in place (and QAS is already up and running), run the following commands in Terminal to set up QAS to be able to use the smartcard:

sudo /opt/quest/bin/vastool smartcard configure \
pkcs11 lib /usr/libexec/SmartCardServices/pkcs11/tokendPKCS11.so

# just a test with a valid smartcard to make sure it works
sudo /opt/quest/bin/vastool smartcard test \
library /usr/libexec/SmartCardServices/pkcs11/tokendPKCS11.so

sudo /opt/quest/bin/vastool smartcard configure macos

(note that tokendPKCS11.so is the name of the file we used - your mileage may vary)

3. The client I was working with had 3 CAs in their certificate chain, so all 3 CAs had to be imported into Keychain. What that means is that Keychain had to have a copy of every CA cert that is in the chain of the user. We actually used a Windows host, and got those certificates in a cer format (crt would work as well) and used the QAS Files policy to get them over to the /etc/opt/quest/vas/ folder. We then imported them onto the Mac using the following commands:

sudo /usr/bin/security add-trusted-cert -k \
/Library/Keychains/System.keychain -r trustRoot -d /etc/opt/quest/vas/enterpriseCA.crt

sudo /usr/bin/security add-trusted-cert -k \
/Library/Keychains/System.keychain -r trustRoot -d /etc/opt/quest/vas/layer1.crt

sudo /usr/bin/security add-trusted-cert -k \
/Library/Keychains/System.keychain -r trustRoot -d /etc/opt/quest/vas/layer2.crt

We put these 3 commands into a script that deployed as a GPO Script policy that ships with QAS and set to 'run once' after the machine is joined.

After that, we were able to log in, but not without some trial and error. It turns out that if we had our card in the reader, and everything configured before the login window was drawn, we had the correct username read off the card, and a single field (labelled 'Password') that allowed us to enter a PIN and login. If we inserted the card after the login screen was already drawn with 2 fields (labelled 'Username' and 'Password'), we could not use the smart card to login, but could get in with the AD credentials, or a set of local credentials.

That was it. Most of it was what you'll find in the install guide, along with some additional configuration and troubleshooting sections. Also, don't think that smartcards can be bypassed with QAS. In the testing that we did, we left it open to allow username/password and local accounts to log in. But we didn't have to. QAS allows you to configure an entire host to require smart cards for login by editing vas.conf. Add the following directive:

require-smartcard = true

under the [vas_macos] section. You can set this option for multiple machines using group policy QAS Configuration policy extension. In addition, you can enforce this on a user-by-user basis by setting the 'SmartCard Required For Interactive Login' option on each user using Active Directory Users and Computers (ADUC).

(edited 2011-09-13 21:42 GMT to correct some formatting problems)


VDI – nothing like watching the lightbulb go on!

I was a conference for a Federal agency this week, and had a fantastic experience with a (prospective) customer. We were talking about this and that, but the conversation worked it's way around to mobile computing. And while agencies are announcing "mobile initiatives" left and right, the reality is that everyone is still really concerned about what BYOD (Bring Your Own Device) means.

My friend, Joe Baguley, has been going on about "the consumerisation of IT" for a while (and he's English, so it's an "s" and not a "z" in consumerisation) as have others, but the Feds work differently. It's not purely about savings, and things are a bit more measured. And that's one of the reasons that the Blackberry continues to hold on, and the iPad has made no movement yet. When the OS cannot be locked down through a central server, and policies cannot be set on applications and data, the device has a slim chance of making it in the Federal space.

But Quest have an interesting solution to the problem; one that took 5 minutes to demonstrate, and turn on the lightbulb for a senior-level manager at this agency. It was vWorkspace. He had his iPad with him (running a beta of iOS 5, in fact - good to know it looks like it works there, too!), and we started talking about how he can control data and access. Thankfully, I had a demo account on a set of servers that a colleague had up  and running (Rob Mallicoat - many thanks!), and he had a network connection on the iPad. Within 5 minutes, I was able to download the vWorkspace iPad app, configure the settings, and voila! I was showing him Visio running on an iPad in no time.

And the thing is, I tell people about it time and again, but until you actually see it working, you don't realize how cool it is. Even the sales guy in the booth lit up when he saw how easy it was, and what it did. That got this manager interested, and proved one of the core problems with VDI; users need to touch it and see it for themselves. It's not enough to give them docs and white papers, and even web demos don't cut it.

But that's not enough; what's unique is how Quest does it. We don't care if you have some apps and desktops on VMware, and others on Hyper-V. You could even be running Terminal Services! We combine all of that, and give you a single interface. As a user, you don't know (and shouldn't care) how an app or a desktop is delivered. We even have a slick, "local VDI" option, thanks to MokaFive.

That's what got the manager really excited - he realized he didn't have to be tied to a single VDI option or vendor, and he can even provide multiple environments all through 1 app, delivered to his users' desktops. Plus, add the fact that it can all be remote means he could even give out iPads, but keep all the data in his data centers.

So if you have an iPad or Android device, ping a Quest person, and see if they can get you access to a demo system. Or ping me if you're with a Federal agency. Because this is something you simply need to try, and not just watch or read about.


Federal CloudCamp in DC

Ken Cline and I just finished up a presentation  at FOSE for the Federal CloudCamp. The slides we used can be found here: http://www.federalcto.com/20110720-CloudCamp.pptx . I'm really pleased with how things have turned out so far. It's not exactly the CloudCamp I was expecting when I initially contacted Dave Nielsen to do one, but I definitely think it's definitely helped the Federal community, as well as advanced the notion of Cloud Computing within the Federal government.

If you attended, thank you for taking the time.

If you didn't attend, please consider going to the next one. You can find more info out about CloudCamp at http://www.cloudcamp.org and you can definitely contact me as well, as I'm sure to attend, if not present.


The battle for online taxes – the devil is in the details

Let me start by saying that I work for Quest, and we have no formal partnerships with Amazon, nor do we compete in any way that I can tell. Personally, I happen to be a satisfied (retail) customer of Amazon, but have done very little with their IT/Online services, beyond putting a bunch of MP3s in their CloudDrive.

The purpose of this post is to get everyone thinking about the complexity of online tax collection. Everyone sees it as a black and white issue, and it's not. It's not just the retailers "protecting their customers" and it's not just a single state or county that is having a revenue shortfall that needs this money. There are a lot of nuances that people do not consider when making their arguments in either direction. Personally, I'm on the fence as to whether online businesses ought to collect local taxes but I think my experience can help shed some additional light on the topic, as a whole.
I had a colleague on the State & Local Government side send me an article about how Amazon needs to just "man up" and help the states. The full title and link for the article is:
Why Amazon is winning online retail and should fold on this silly sales tax fight

One of the last paragraphs of the article states:

"Rather than fighting it out on a state-by-state basis, and yanking on the incomes of their affiliates while they’re doing it, Amazon needs to man-up and do what’s right."

And my response to the whole article was, "It's an interesting, if light, article." I went onto say that the author, whilst championing the States' case for collecting online taxes, completely overlooked the real problem with doing this. The problem is not just the fact that consumers wouldn't like the 5-10% hike in prices. The problem is that it would drive up prices considerably, and take any competitive edge Amazon has away. David Gewitz overlooks a key financial factor that is a huge obstacle for anyone collecting taxes for online sales. And that is the cost of manning and maintaining a "Sales and Use Tax" department. What's that, you say? Why would you need a "department?" Can't all this be automated?

Well, not quite. For a retailer of Amazon's size, you're talking about getting dozens of tax analysts and clerks to handle this because there are thousands of jurisdictions. And because you're talking about thousands of jurisdictions, with millions of different rules, codes and policies, often open to interpretation, you need humans. You would need to add more labor, (and supporting systems) to deliver the same products and services. I know, as I was one of these humans at one point.

In a previous life, I worked for a company called Tax Partners as a Data Development Manager. Tax Partners has since been bought by Thomson, and it's impossible to find a nice, clean link on their site for what Tax Partners did, but what they did was manage outsourced Sales & Use Tax filings. These would often be for companies that deal with consumers in retail, telecomm, automotive, travel and other sectors. These were Fortune 1000 companies, that had a presence in hundreds and thousands of locations. And my job was to manage the development team that would write systems to take the client's tax data, and normalize it to fit into our system, which was then used by our internal tax analysts.

Now, here's where the details come in, and I'll use a single client as an example. Imagine you're a telecommunications company, and you provide cell phone service nationwide. You have to collect a ton of taxes, and all those taxes are a liability. You have to pass them onto the respective jurisdictions, so you really don't want to hang onto this money. The short term interest sounds interesting (pun intended) but it's not nearly enough to justify having to deal with this money. So while you get this money, it is only a burden to you, and you really don't want anything to do with it. I can tell you that one large, US telecomm was paying $80-95 million per month in taxes. However, that money would have to get divided among 5,000-9,000 different tax returns! Just try to wrap your head around those numbers on a monthly basis for 1 customer.

In some cases, the returns would only total as little as $5 or $10, but they were required to submit them. And everything had to be accounted for, with the money divided properly among the various jurisdictions in a state, even though you collect a lump sum from your customers. Oh . . . and those returns . . . often, they were a specific form, written by some policy maker, that said the form, in that exact format, must be submitted on the 10th of the month after business was closed. And perhaps even in blue ink! Now spread that out over more than 9,000 jurisdictions. And if you don't file, you not only get hit with interest, but a potential penalty. And that penalty may be $500 for a $10 return.

So, while each jurisdiction wants it's money, they're not exactly the easiest group to work with. It's one thing when you're a business with a physical presence in a location, and know what's going on in that location, but when you sell to people in places you've never heard of, it's rather hard to keep up with the rules and regulations that jurisdiction puts out. There's a cost to that, and it's not as easy as a company in the UK, that simply adds the same amount of VAT to everything. My colleague added:

"The states believe there is a river of tax money flowing by and they aren’t dipping into the stream.  They’re very thirsty right now."

They may be thirsty, but they're not very accommodating. Would that every jurisdiction used a common form, accept electronic feeds and transfers, and provided their rate information in an easy to use manner, it might be feasible. But the way things stand now, the cost (to a business) for collecting online taxes is very high, and there's one more thing. I mentioned before that if you get the taxes wrong, you could be facing a penalty. That penalty is often arbitrary, set by a judge in that very jurisdiction. Which means the odds are stacked against you, even if you make a good faith effort to help the states, as Mr Gewitz suggests that Amazon do. At best, you can "meet expectations" at a considerable cost to you. At worst, you could get it wrong, and get hit with enough fines and penalties to put you under, all while trying to "help the states."

I do agree that the states and other jurisdictions need help, and that there is a revenue stream they could tap into. But it's not by applying 20th century rules and methods to a 21st century problem. I have my own ideas on what can help, but this post is long enough as it is. Thanks for taking the time to read it.


Wait! I thought the Feds only cared about smartcards . . .

Monday, I posted how Quest's tokens allow for users to program their own seeds. Which then prompted questions (internally) of, "Why do you even care? I thought you focused on the Federal space, and the Feds only cared about PAP, or CIV cards?"

Well, yes. For the most part, the US Federal government does focus on CAC (Common Access Card - used by the DOD), and PIV (Personal Identification Verification - mostly used by the civilian agencies) cards. And you'll also hear about PIV-I (PIV Interoperable) being involved. In the case of CAC and PIV, a user has to file an application, and some Federal agency would need to sponsor the individual to obtain such a card. This usually includes a background check, or some sort of formal review, before the card is issued. PIV-I tried to lower this barrier, by allowing non-Federal organizations (think government contractors, state governments, first responders, etc) to issue interoperable smartcards that are trusted by Federal systems.

However, PIV-I has a lower "assurance level," and often involves the same (or similar) sort of background check, just by a different organization. Assurance Levels are set by NIST and can be found detailed in their Special Publication (SP) 800-63. You'll see there are 4 assurance levels, and PIV-I only gets to level 2 (really, it gets to level 3, but with a less stringent background check, it can only be considered level 2.5 at best). CAC & PIV strive for level 4, if not level 3.

Ok. So we've established that smartcards are the main vehicle for 2-factor authentication in the Federal government, but I still haven't explained why tokens (RSA and other ones) crop up. And this is because a token is also an acceptable form of 2-factor authentication (read SP 800-63. and you'll see them mentioned as 'one-time passwords'). The "identity proofing" is still required, but tokens are actually a lot more flexible for several reasons.

1. Tokens can be assigned to any user (part 1). In the case of smartcards, they are usually assigned to people, while tokens can be shared. In fact, with our tokens, users can share tokens, but continue to use their distinct credentials (username and password) with the token. Which means that multiple admins can share a token to access a common system, but you can determine which admin logged in to do something with the particular token. This lets you continue to have 2 factor authentication, but also a "check-in/check-out" system for the particular token, allow more controls over the physical token (perhaps locked away in a safe or vault).

1a. Tokens can be assigned to any user (part 2). This is really a corrollary to 1, which is that the token can be assigned to service, or privileged accounts. Putting in the same sharing (check-in/check-out) model for a token, plus tie the password to a password vault product, and you have some pretty solid security around your privileged accounts such as oracle, root, Administrator, and other 'non-named' accounts.

2. Tokens are independent of environment. With smartcards, you have to have a reader attached to the user's console. No reader, or a malfunctioning reader, makes authentication a bit more difficult (read not possible). There are also situations where PKI simply isn't used. Older applications or platforms that do not make use of certificates cannot be changed quickly or easily. With a token, along with a username and password/PIN, you can continue to get 2FA, even in a scenario where a reader isn't available or not practical, or the app is expecting only a username and password. There is still some amount of work to be done, but it's often easier than tying an app into an entire PKI infrastructure.

2a. Tokens are independent of environment. There are some cases where the app or platform is capable of using PKI, but it is sitting in a location/area/network that simply cannot reach the PKI infrastructure to which the certificate on the smartcard is tied to. Not every system is actually on the internet (unbelievable, I know), which means tokens can provide access without requiring a centrally managed CA (certificate authority) to be present.

3. Tokens can be assigned much quicker and easier. And this is really the crux of why tokens come into play in the Federal space. Smartcards require a centrally issued certificate to be put onto the card. In some cases, there is no time for the requests to make their way through the system to get a certificate, publish it to a CA (certificate authority) and card, and get the card to the user. In other cases, the user simply will not get through a background check (or unwilling to get one), but has to be given access to certain systems. Yes, there are times when the Federal government has to deal with "questionable people" but they might as well make sure it's the right person, so 2-factor is still needed.

3a. Tokens can be revoked much quicker and easier. Because the token is usually some bit of plastic, it's easier to revoke and know that it won't be used in other ways. Most smartcards take a while to issue because they are also printed as a security badge. Meaning that even if the certificate on the card is revoked, the card may still be usable to get physical access to a building or location. However, it's unlikely that an agency will let you in because you have a black piece of plastic on your keychain.

So, with those all those reasons, tokens are not going away. Smartcards will continue to dominate, but there will continue to be a need for 2-factor authentication (2FA) using one-time passwords (OTP) in the Federal space.


The RSA Breach saga continues . . .

I've discussed the RSA breach before, but had a very interesting conversation last week with a customer that was in a dilemma as to what to do. RSA have said they would replace some of the tokens, depending on the situation (protecting Intellectual Property, Consumer-Oriented products, etc) and this customer was pretty certain they were to get new tokens from RSA, but didn't know if they wanted them. Nor do they want any other "standard" token that other vendors could provide, because that new vendor could be breached as well.

What this client really wanted was a token where he could program his own seed into it. I mentioned that our software tokens actually allow for this out of the box, and when you "program" a new soft token, the seed is automatically generated. Which means that Quest never knows what the seed record is for any software token that we provide. For an example of this, you can actually see a recording of it here where the seed is generated, and then here where the seed is put into our Desktop token (there is no audio on either recording).

However, this wasn't as interesting to him as programmable hardware tokens. He had concerns about keys getting compromised while being transferred to the end user, and wanted to send a pre-programmed hardware device. At first, I didn't think we had anything like that in our arsenal, however, it turns out that one of our latest token models (The Yubikey token) as well as some of other ones already allow for user programmability! In fact, if you look at the link for the Yubikey token, and then scroll down, you'll see that some are programmable. It took our Defender Product Manager (Stu Harrison, who blogs on and off here) to point this out to me, even though I should have remembered this from earlier on.

Obviously, each token does come with a default seed that Quest will know about, but if there was concern about a vendor having the "keys to the kingdom," a programmable token puts the onus back on the organization, and keeps Quest out of the limelight. I don't want to discount the fact that this will take more manpower, but if you don't want your vendors to have your seed records, reprogramming the tokens is the only secure way of doing it. It only makes sense, and as a security professional, you should never rely on RSA, Quest or any organization if you can minimize the number of people that have access to the token seed records.


The Federal CIO’s guide to partnering with Quest Software for Data Center Consolidation, Part III

Note 1: the bulk of this blog post was done on an Apple iPad - I point this out not because of a fascination with the iPad, but because of the fact that such long documents were not readily possible from a mobile platform only a few years ago. That still amazes me.

Note 2: this is a very rough, stream of consciousness blog entry. Grammar, spelling and other writing errors should be ignored. If you want a nice, clean "white paper" type of document, please contact me offline, and I'll get you something cold and clinical.

Preparation and migration
Preparing for the move, beyond the simple assessment seems a no-brainer. But a migration? Really? Right before a move? And I say "yes." First, let's be clear and qualify that "migration" (to me) means a cut over to another platform. This could be email, user directory, database, etc, and it may be to a new version of some current software, but it's definitely getting off the current version. The idea is to make the software and version of that software as current as possible before you actually move to a new location/environment. The only thing this excludes is the move from physical to virtual. That comes later, and I'll explain why then.

There are several reasons for you to consider doing a migration before consolidating environments. First, and foremost, the migration is going to happen sooner or later, and aren't you better off doing the migration in a comfortable and stable environment instead of your new one? Plus, the migration may actually shake some things out and make the consolidation easier. For example, if you use QMM to migrate mailboxes from an old version of Exchange to 2010, or use QMM for AD to restructure your Active Directory environments, you may actually find that there are many users, mailboxes, groups and other objects that could be deleted/abandoned.

The same goes for databases. If you're running an old version of Oracle for your databases, it's time to cut over, and see what features and benefits you get. And we even have a tool that let's you mix versions to replicate data while you make this sort of move, so it's not a jarring, "all at once" process.  That tool, BTW, is Shareplex.  And it also let's you replicate mixed versions of databases, which is pretty cool.

But why else should you do the migration before starting the actual move? Well, frankly, because support is easier. Sure, you can migrate a Windows 2003 server, or an Oracle 9i database into your new environment, but if there's a problem, what will the vendor tell your team? Most likely, they'll tell you to upgrade to the latest version.

It's not widely discussed, but the reality is that most software companies want you on the latest and greatest version of their software when you call support. It's usually the version their developers are currently using as the basis for the next release. It's the version that support is working with most, and one that they have set up locally to recreate your problems. One or two versions is often ok, but if you're running something more than 2-3 years old, I think you're asking for trouble. Get to the latest versions however you can, because you don't want to consolidate and move old software around.

Another reason is your staff's personal development. I've run IT groups in the past, and the most common question was always, "when are we going to get to the latest version of X?" where X was some database, operating system or programming language. If you are the CIO at a federal agency, your staff knows that data center consolidation is coming, in some form or fashion. You want them ready and energized for the task. Letting them get to the latest version of whatever software they work with will excite them, and you'll have a happy crew moving into the consolidation.

Now, I did mention that cutting over to a virtual environment should be last. The reason is that this is really the same as a hardware move. No matter what hardware your environment is using, something is sure to change. And your software may react adversely. Plus, if you couple that with a version upgrade, you are changing a lot out from under your user base as well as IT staff. The idea is to minimize risk, and that just doesn't cut it.  So do the "migration" first, get it settled, and then cut over hardware (which can be P2P or P2V).

And if you're contemplating a P2V move, you should definitely check out vConverter.  Not only does it work with P2V, but V2V (you may want to switch hypervisors, or try out multiple hypervisors with the same ), and even V2P in case you absolutely have to back out.  Or even want to switch hardware, using the move to virtual as a stepping stone.

Finally, if you upgrade, migrate and virtualize in a single move, how do you know where you got performance gains or losses? If you read my last post on this topic, you'll know I propose baselining before starting.  The only way to do that is to start with a known point, but then make incremental moves so you can collect more information on what impact each part of the upgrade, migration and consolidation has on your environment.


The Federal CIO’s guide to partnering with Quest Software for Data Center Consolidation, Part II

Note 1: the bulk of this blog post was done on an Apple iPad - I point this out not because of a fascination with the iPad, but because of the fact that such long documents were not readily possible from a mobile platform only a few years ago.  That still amazes me.

Note 2: this is a very rough, stream of consciousness blog entry.  Grammar, spelling and other writing errors should be ignored. If you want a nice, clean "white paper" type of document, please contact me offline, and I'll get you something cold and clinical.

Initial assessment and baselining
Let's dive right in and get started.

To begin, an assessment needs to be performed to determine all the things that will be part of the consolidation.  Presumably, this has already started as an initial plan is due to be submitted to the OMB for review and budgeting. However, everyone knows adjustments can and will be made. So I'd suggest you do an assessment assuming every item will be questioned.

There are lots of ways to survey what you have, but looking to Quest to help with that may not be something you thought to do.  Well, you should.  The reason is that we have a lot of tools, and lots of tools to help with your entire environment. From the operating system, to the databases and file servers, all the way to app and web servers as well as desktops.  And while we're not in the inventory management business, we can certainly hold our own if you need a list.

"What kind of lists can you provide," you ask? For starters, we can get you users and desktops. Nowadays, most users are in Active Directory. And most of their desktops and laptops are joined to AD. So you could use something as simple as Quest Reporter to pull a list of those objects out of AD.  Following the 80/20 rule, that should give you a good ballpark of your end-user environment. Need something s little more accurate? Then you'll need to do more work but get more results. You can either go with something like Desktop Authority to get you a good idea of what is actually out at the desktop level.  Or, you can fall back to your AD event logs, and monitor login events over some time period with Quest Change Auditor for AD. In both cases, the products are sure to give you a lot more benefits beyond the initial assessment. And both Change Auditor and Reporter give you a good feel for your Windows server environment as well.

But the assessment is more than just a 'survey.' You cannot just make a nice clean inventory of everything you are responsible for, and leave it at that. It is critical to know -how- those systems are performing. In other words, you need to set a baseline, and you probably need to do it in 2 ways. The first way is through some measurements and metrics. Quest's Foglight platform is fantastic for end to end monitoring, and it can serve double duty by providing those initial statistics up and down your entire app stack.

Foglight can also provide those initial lists I mention above.  You need RAM, CPU and disk numbers off your servers? We can get those to you, and help with some capacity planning as well. And if you run Foglight long enough, you'll have some very good trending and usage data to use beyond the consolidation effort.

The second baseline to check is subjective, and it's the user's perception of the current systems.  This wouldn't involve any Quest product, but to simply put together a quick, 5 minute survey of what the users think of the apps they use. There are many free and paid sites out there that can run such a survey for you but I'd really encourage you to get this initial feedback. And if it starts to look grim, and you're surprised by the results, check out Quest End User Monitor to walk through the apps, and see what the users are complaining about.

That's really it on the baseline side.  We can help with that initial assessment as well as providing initial metrics for how your environment is functioning.  Can we provide complete coverage of your environment?  Probably not, but the tools you'd use from us would continue to provide value beyond the assessment rather than being a throw away once the assessment is complete. And wouldn't it be nice to be in a new environment but with a familiar toolset? I think your IT staff would say, "yes."

Copyright (C) 2010-2011 Dmitry Kagansky – All opinions expressed are those of the respective author and do not reflect the views of any affiliate, partner, employer or associate.