federalcto.com The Blog of a Federal CTO


Your Directory Has Been Breached – landing page

This is a landing page for a project Bob Bobel and I have been working on recently. Today (May 30th, 2012), I am presenting the first draft of the slides which will be posted shortly. A white paper is due out shortly. All information will be posted on this page, so it will be revised in the coming weeks.

(2012-05-30 - 13:00 ET)
First up, here is Bob's initial post on the topic:

(2012-05-30 - 16:15 ET)
Here is the slide deck just presented at the Department of Energy NLIT Conference (National Laboratories IT Summit):

(2012-06-05 - 18:00 ET)
Here is the first (rough) draft of the document to correspond with the slides. Yes, whole sections are missing, and will be filled in during the coming weeks. Stayed tuned.



Smartcards, certificates & OS X: Does Lion Roar or Meow?

I spent some time the week before last dealing with smartcards, certificates, Macs and OS X Lion (10.7), specifically. My initial take on Lion was this; I can't believe Apple has taken this direction for the enterprise user.

Before going any further, I should come clean and say I love Apple products. As a consumer, they do what I need, and work as expected. The fact that I've not had to rebuild my wife's MacBook in it's entire life (of more than 3 years) is remarkable. That all changed with OS X Lion. I downloaded it, and was not impressed from the get go. Just starting with the fact that I had to hunt down the DMG file so I would have a copy was irritating, and things continued from there. But I work with technology, need to keep up, and wanted to check out the buzz.

I also did some initial research, but articles like this one didn't help; http://www.macworld.com/article/161493/2011/08/lion_enterprise.html . This guy clearly doesn't run an "enterprise shop" and probably considers an office of 10-15 people to be representative of an enterprise. But I bit the bullet, upgraded my personal MacBook Pro, and have been wincing since. But that was as an end-user. On the enterprise side, things turned out to be worse. The AD Plug-in - broken. The Login Screen - completely screwed up. Apple broke everything when it came to Directory Services and the Login page. And to top it all off, smart card support has been deprecated! Really? This is enterprise ready?

Before we go any further, here's a quick review of how smart cards worked in OS X prior to 10.7:

  1. A card is inserted into the reader
  2. The reader uses the tokend file (ships with the OS) as a driver to read the certificate off the card
  3. The login screen is re-drawn, the username is pulled off the card (to display on the screen), and the screen is changed to only display the PIN request field
  4. The user enters the PIN tied to the card
  5. The certificate (issued by a Certificate Authority aka CA) is confirmed to be valid
  6. The CA's certificate is also checked, and confirmed to be valid, so it's certificate (and any up the chain) have to be in the OS X Keychain
  7. The user is allowed to log in

Apple no longer ships tokend files (step #2), and the default login window is no longer refreshed by a smartcard insert (though it does still flicker). We have since had a very helpful conversation with an Apple architect, and will be making changes to make it all work properly in Lion, but in the mean time, it's possible to get it all working with some elbow grease and lowered expectations.

So . . . how did we do it? Well, we started with the QAS Smartcard guide that you can find with the standard QAS documentation. And there's a whole lot of good stuff there about smartcards and certs, but here's what I boiled it down to:

1. Go to http://smartcardservices.macosforge.org/ and download the latest package to get the tokend needed to read the specific cards you are using. Apple has open sourced it all, and are relying on non-Apple folks to maintain this. If your card is not there, you will need to ask your smartcard vendor to give you a tokend file (basically, a little library to allow you to work against the specific card). We were lucky enough to be using CACs so the tokend we needed was in the beta2 batch from Aug 19, 2011.

2. Once the tokend is installed, and in place (and QAS is already up and running), run the following commands in Terminal to set up QAS to be able to use the smartcard:

sudo /opt/quest/bin/vastool smartcard configure \
pkcs11 lib /usr/libexec/SmartCardServices/pkcs11/tokendPKCS11.so

# just a test with a valid smartcard to make sure it works
sudo /opt/quest/bin/vastool smartcard test \
library /usr/libexec/SmartCardServices/pkcs11/tokendPKCS11.so

sudo /opt/quest/bin/vastool smartcard configure macos

(note that tokendPKCS11.so is the name of the file we used - your mileage may vary)

3. The client I was working with had 3 CAs in their certificate chain, so all 3 CAs had to be imported into Keychain. What that means is that Keychain had to have a copy of every CA cert that is in the chain of the user. We actually used a Windows host, and got those certificates in a cer format (crt would work as well) and used the QAS Files policy to get them over to the /etc/opt/quest/vas/ folder. We then imported them onto the Mac using the following commands:

sudo /usr/bin/security add-trusted-cert -k \
/Library/Keychains/System.keychain -r trustRoot -d /etc/opt/quest/vas/enterpriseCA.crt

sudo /usr/bin/security add-trusted-cert -k \
/Library/Keychains/System.keychain -r trustRoot -d /etc/opt/quest/vas/layer1.crt

sudo /usr/bin/security add-trusted-cert -k \
/Library/Keychains/System.keychain -r trustRoot -d /etc/opt/quest/vas/layer2.crt

We put these 3 commands into a script that deployed as a GPO Script policy that ships with QAS and set to 'run once' after the machine is joined.

After that, we were able to log in, but not without some trial and error. It turns out that if we had our card in the reader, and everything configured before the login window was drawn, we had the correct username read off the card, and a single field (labelled 'Password') that allowed us to enter a PIN and login. If we inserted the card after the login screen was already drawn with 2 fields (labelled 'Username' and 'Password'), we could not use the smart card to login, but could get in with the AD credentials, or a set of local credentials.

That was it. Most of it was what you'll find in the install guide, along with some additional configuration and troubleshooting sections. Also, don't think that smartcards can be bypassed with QAS. In the testing that we did, we left it open to allow username/password and local accounts to log in. But we didn't have to. QAS allows you to configure an entire host to require smart cards for login by editing vas.conf. Add the following directive:

require-smartcard = true

under the [vas_macos] section. You can set this option for multiple machines using group policy QAS Configuration policy extension. In addition, you can enforce this on a user-by-user basis by setting the 'SmartCard Required For Interactive Login' option on each user using Active Directory Users and Computers (ADUC).

(edited 2011-09-13 21:42 GMT to correct some formatting problems)


The RSA Breach saga continues . . .

I've discussed the RSA breach before, but had a very interesting conversation last week with a customer that was in a dilemma as to what to do. RSA have said they would replace some of the tokens, depending on the situation (protecting Intellectual Property, Consumer-Oriented products, etc) and this customer was pretty certain they were to get new tokens from RSA, but didn't know if they wanted them. Nor do they want any other "standard" token that other vendors could provide, because that new vendor could be breached as well.

What this client really wanted was a token where he could program his own seed into it. I mentioned that our software tokens actually allow for this out of the box, and when you "program" a new soft token, the seed is automatically generated. Which means that Quest never knows what the seed record is for any software token that we provide. For an example of this, you can actually see a recording of it here where the seed is generated, and then here where the seed is put into our Desktop token (there is no audio on either recording).

However, this wasn't as interesting to him as programmable hardware tokens. He had concerns about keys getting compromised while being transferred to the end user, and wanted to send a pre-programmed hardware device. At first, I didn't think we had anything like that in our arsenal, however, it turns out that one of our latest token models (The Yubikey token) as well as some of other ones already allow for user programmability! In fact, if you look at the link for the Yubikey token, and then scroll down, you'll see that some are programmable. It took our Defender Product Manager (Stu Harrison, who blogs on and off here) to point this out to me, even though I should have remembered this from earlier on.

Obviously, each token does come with a default seed that Quest will know about, but if there was concern about a vendor having the "keys to the kingdom," a programmable token puts the onus back on the organization, and keeps Quest out of the limelight. I don't want to discount the fact that this will take more manpower, but if you don't want your vendors to have your seed records, reprogramming the tokens is the only secure way of doing it. It only makes sense, and as a security professional, you should never rely on RSA, Quest or any organization if you can minimize the number of people that have access to the token seed records.


Token Bloat, AD Bridge and Quest Privilege Manager

If you don't know what "token bloat" is, then I suggest you do some research.  Plenty has been written on it, and while I'll provide a brief description, others do a much better job of covering it.  The short version is that when you log in using your Active Directory credentials you get a Kerberos ticket (which is just your identity - who you are) plus a PAC (Privileged Attribute Certificate - which is a list of all your security group memberships (their UUIDs, actually) - what rights you have).  As a user gets more and more rights (through group membership), the PAC grows.  Eventually, it becomes big enough to affect logins and applications that use the AD Kerberos ticket (often called a token).  The effect is that things slow down, or just flat out break.

But how does this affect AD Bridge products, like Quest Authentication Services?  Unfortunately, in much the same way.  If you're a member of 10,000 security groups, it's going to take some amount of time to get through that list.  That doesn't change just because you've logged into a Unix, Linux or Mac machine.  At best, you can set a flag to ignore the PAC if all you want is authentication, but if you are a member of 10,000 groups, someone probably put you in those groups for a reason (note: 10,000 is hyperbole, and if you really are a member of 10,000 groups, you have some serious AD design problems).

And an increase in the number of groups is often a by-product of deploying an AD bridge solution.  We have other ways to alias out and re-use existing groups, and you can certainly use local groups, which are not in the PAC, but who wants that?  In some cases, you want those added groups.  You need those added groups.  A unix sysadmin is not the same as a Windows administrator.  And an Oracle DBA often has different rights, and requires different groups from your SQL Server admins, so simply generalizing your groups aren't enough, either.

But for customers that have QAS along with Quest Privilege Manager for Unix, there's a better way.  And one that provides a lot more flexibility and configuration options. Plus, you don't have to have QAS.  You can do this with local accounts, LDAP, NIS, and any other users you have on unix. I'll just stick with AD users because the topic of this post does focus on 'token bloat' which is an AD-specific problem.

What is Quest Privilege Manager (QPM4U)?  It is a tool for unix command control, and I/O logging.  There are lots of other descriptions people use - "Tivo for your unix machines," or "sudo on steroids" are often used, though most fall short in a 1 sentence pitch.  QPM4U allows very granular control over what rights users have on a particular machine, and that granularity includes options such as time of day, and day of week, environment variables, and multiple group memberships. In fact, it can use any variable/attribute about the user, the machine and environment.

And the last item I listed (multiple group memberships) is where things get interesting.  You actually write a rule that says, "if a user is a member of OhioUsers, and a member of SalesTeam, then the user can access files in the /SalesDocs/Ohio folder."  This is interesting, because in the Windows world, most people would be added to the SalesTeam group, and the OhioUsers group, and then a third group would get created called 'OhioSalesTeam' just to grant access to a similar folder.  And even with sudo, the same thing would happen.  Sudo can only check for 1 group membership, and you're either in the group, or not.

This is where QPM4U can let you use existing groups, and perhaps other attributes, to determine whether users can access resources or run certain commands.  In fact, you could simply query AD (check out the 'vastool search' command), find out the user's job title (Account Manager), their state (Ohio) and make the determination that way.  Which means that no groups are needed.  This is an extreme example, and not one I'd recommend, but hopefully you see how it could help prevent token bloat.  In addition, you can start adding in other conditions, such as time of day (if the user is a DBA, and it's after 7 PM, then they can run the 'backup database' command), allowing for even more flexibility, and control.

Now, I've been writing all this using pseudo code, and not showing you how simple this config would actually be to put in place.  So what would be needed in the Privilege Manager config file for that initial scenario? Well, thanks to my colleague, Paul Harper, it would look something like:

# check all the arguements to see if the filepath is the one allowed
c = 0;
protected_dir = '/SalesDocs/Ohio/';
prot_dir_len = strlen(protected_dir);

while (c < argc) { 
   if ( (strlen(argv[c]) >= prot_dir_len ) && ( strsub(argv[c], 0, prot_dir_len) == protected_dir )) {
      if (ingroup(user, "SalesTeam")) && (ingroup(user, “OhioUsers”)) {
      else {
  c = c + 1;
  # do other things here for other conditions.

To be fair, this is a poorly written script, and it has many faults, but it gives you an idea of the types of things that could be done with Privilege Manager, especially when it comes to multiple groups, and the fact that you can check for multiple group memberships, without having to create a brand new group just to manage access to a resource where other conditions already exist to check this.

There you go, Jeff. Hopefully this post helps you - give me a call if you've read this far down.  🙂


Upcoming posts

I have several posts queued up that are quite detailed about how Quest can fit a data center consolidation strategy.  But in the meantime, I have this post on a related site until those are published:


As the title suggests, I talk about how to block USB mass storage and CD-ROM drive on a desktop, and complementing it with a Quest product called ActiveRoles Server to add and remove the machine from groups on a temporary or ad hoc basis.

Copyright (C) 2010-2011 Dmitry Kagansky – All opinions expressed are those of the respective author and do not reflect the views of any affiliate, partner, employer or associate.