Windows Server


Password Change and its Replication    



Password Change and its Replication


Password Change and its Replication


Extract of one webcast by Mike Resnick, I hope this will make things clear about password change replication:




Microsoft Corporation

Microsoft Windows 2000 Server and Windows Server 2003: Password and Account Lockout Features

February 27, 2003

Note This document is based on the original spoken WebCast transcript. It has been edited for clarity.

Mike Resnick: Today we're going to discuss the password and account lockout features. This is derived from a white paper that we've been working on called "Account Policies, Best Practices, and Account Lockout Troubleshooting." It's kind of a long name to cover a pretty wide area. We have a lot of information to provide, so I'm going to dig in quite quickly here. And if you find that you need more depth, please refer to the account lockout best practices white paper. Right now it's still in tech review, but it should be on the Web site soon.

For our agenda today (slide 2) we're going to cover the security threats you face and the costs; the password and account policy settings that you can configure as well as the security password and account lockout recommendations; the current existing account authentication behavior; and the new features introduced in Service Pack 4, Service Pack 5, and Windows® Server 2003. Also, Joe will cover the procedures for troubleshooting account lockout events and the tools to use when troubleshooting this issue. We have some really great stuff, and I think you'll enjoy what we have to offer here for you.

Passwords are the keys to your kingdom (slide 3). There are multiple threats to passwords out there. There are dictionary attacks, as well as brute force attacks, and there is a lack of security practices by both users and administrators. What we mean by dictionary attack is an attacker will take a known list of passwords or a large dictionary and apply it against the user's password security. The brute force attack is more of a methodical method in which it will try every permutation available to guess a user's password.

What we mean by a user's lack of security practices — users find that passwords are more of a nuisance, but the security of your enterprise relies on the combination of password length, password uniqueness, and password life span. Your users are the key, especially your administrator or high-privileged users; they are the key to the security of your environment. If you don't take passwords seriously, your enterprise may not be owned by you.

Most users prefer easily remembered passwords, so protect yourself against those type passwords by enforcing a complex password policy, which we'll go into a little bit later. In addition to passwords, let's do a risk assessment here of what a complex password policy would do for you.

Long passwords (slide 4), or password lengths of six characters, have 689 billion possible combinations. With our password policies we throw out certain characters, like your user name and well-known passwords; but for the most part you still have 689 billion options. What does that mean for you? Your user has an option of between 26 lower case characters, 26 uppercase characters, 32 special characters, and 10 numbers. So that gives you 94 different characters available for each character of a six-character password, which comes out to the large number that you see there.

If we take a 60-day password expiry date and a password of seven characters, it would require 7,407,407 logon attempts per second to find that password. If you're a betting man or woman, the odds of playing the lottery are much better, to the extent that with the lottery you have only number combinations, versus password permutations. The combinations for a six-character lottery ticket are 15 million. So as you can tell, it's much easier to play the lottery to win than it is to play against six-character passwords or seven-character passwords.

The next slide is on password settings and password filters (slide 5). These are used to enforce what I've touched on in an earlier slide, the complexity in your domain. This is an easy attribute to set in the Active Directory® Users and Computers snap-in. You go into the domain policy and configure it there. We recommend that you configure this in only one location — all your account lockout policies, all configured on you domain policy. Even though it looks like you might be able to apply this on an organizational unit level, this should be applied only at the domain level.

The password filtering was introduced in Windows NT® 4.0, but it required a special .dll and registry change. There's a KB article there that provides you with steps for how to set that up in Windows NT. However, in Windows 2000, it is a simple attribute change in Group Policy.

Password settings (slide 6) also help us. There's password history. Password history prevents a user from using the same password over and over again. The user can't use a specified number of previously used passwords.

For example, the minimum is 0, so in this case the user can change his password back to the same exact password prior to the prompt to change his password. The maximum is 24. This is an easy setting, and has low cost for you. This ensures that your passwords stay unique in your environment. In other words, my user can't go in and automatically use this same password over and over again, every time he's prompted to change it.

The next selection is the minimum length. By default, we leave it at zero. However, I'd highly recommend that you set it to least 6, if not 7 or 8. This length dramatically reduces the odds of a threat in your environment by making the potential number of permutations available astronomical, and it makes an attack much more difficult. By default it's set to 0, and the maximum is 14. I've had customers who require password lengths of 10 characters and who have changed it every 10 days. That may be on the extreme side. However, your security needs may require that.

Maximum password age is when your user is going to have to change their password. This is the maximum time that password can be valid on a domain.

The maximum password age by default is set to 42 days, but it can be set up to 999. For a secure environment we recommend that you reduce that number and have your users change their password more often. The more often they change their password — and I wouldn't go to the one extreme — the more it becomes a moving target for a potential hacker, versus a stationary target, where we're using the same password over and over again.

Also, just to interject at this point, you want to change your local administrator account password on a regular basis. This would be a good practice, just due to the fact that those accounts are also susceptible to potential attacks on their passwords. If you change them on a regular basis, you allow the attacker less time to break that password.

Minimum password age, this setting is used to prevent users who have password history enforced from changing their password over and over and over again the same day, until they can use this same password again. If you set that to 1, that would normally suffice, because it would take 24 days for them to be able to get back to their original password, with just one change a day. You can set it to as high as 999, but a reasonable setting of 1 day is not too outrageous.

I wanted to discuss this next topic by itself, the account lockout threshold (slide 7), because this is a very important setting in our environment. The account lockout threshold is very commonly misconfigured. What I mean by that is a lot of people think that if they lower their threshold, they're increasing their security dramatically. But the difference between 3 and 20 is very slim when you look at the number of permutations available for a password. The statistical significance between 3 and 20 is absolutely miniscule.

Why then do many people go with 3? Somebody once decided 3 was a great number, and a lot of people have followed through with that. There are many corporations with policies to set that threshold to 3.

When you set this at 3, you're paying a price in administrative overhead. What I mean by administrator overhead is that your help desk is going to have to unlock unnecessary account lockouts, such as users making a mistake on their keyboard, or a user changing their password on Friday, then coming in on Monday and typing the old password three times, and then they're locked out.

A way to avoid this is by bumping it up, and I'd recommend to 10. Actually, that's Microsoft's recommendation as well. We'll go into that a little bit later, but 3 and 5 are way too low. Because in addition to user error, there are many applications out there that will try multiple attempts for a single sign on. I try to access a resource with the wrong password, it will go out there and try three times, or four times, or once every protocol. Every attempt there will go against your threshold and lock the user out. I'll explain a little bit later how the threshold works with the BadPWD count, which is on each PC. But I'll go into that a little bit later in our presentation.

The valid threshold settings: by default it's set to 0, but the valid non-zero settings are 1 to 999. Again, I highly recommend a setting of at least 10.

Account lockouts or other account lockout settings (slide 8): resetting the lockout counter, that's an ObservationWindow. The ObservationWindow is an interval after which an account or BadPWD count is reset. Every time a domain controller receives a bad logon attempt, it will mark down the bad attempt in its BadPWD count for that user. This value is not replicated between domain controllers (DCs). It will instantly go and talk to the primary domain controller (PDC) emulator, and it will ask if there is a new password. And if there is no new password, the PDC emulator will also increment the BadPWD count. We'll go into that in a little bit more detail later in the presentation.

The account lockout duration: When a user is locked out, instead of having the administrator automatically reset that user, you can automatically have that user be reset by the machine. Now is this a security problem, or is it not? In a highly secure environment, it is a good idea to have the administrators unlock the users. However, the statistical analysis tells us that including a lockout duration interrupts a brute force attack dramatically.

If a user is constantly being locked out, he's going to complain (and Joe will go into more detail on this), and you'll have additional events that will tell you that the bad password has occurred, that a user has been locked out. These events will also give you a clue as to what kind of a problem it is, whether it's an application or brute force attack. It will give you information that may assist you in determining the root cause of the problem.

The next slide (slide 9) covers the correct settings for invalid logon attempts. It's truly a balancing act. You have to weigh the potential administrative overhead, and if your users are being locked out on a regular basis, you may potentially miss, if your settings are incorrect, a real attack on your environment. Setting it too low can be very detrimental in your environments, not only financially, but on the security end as well. You may not determine or may not see a potential threat out there because of a high number of lockouts, because of some of the bugs that we'll go into, as well as some of the other potential problems.

Multiple applications will send credentials many times across to the DC, the Exchange client, the Windows 2000 Directory Services Client or Distributed File System (DFS) client, and you have Kerberos failures with NTLM, which will go back into the negative Kerberos caching.

This next slide (slide 10) has the official Microsoft-recommended settings for account lockout and password policies. We derived this information from a lot of testing, and through real-world testing with some of our enterprise customers, to determine what settings not only deal with the security needs of your environment, but also balance the administrative costs. We have a low setting, a medium setting, and a high setting.

I personally don't recommend the low setting, because I'm a true believer in complex passwords and maintaining a complex password policy. However, the medium setting is absolutely more than adequate in most environments. The settings are 10 at the threshold level and an observation window of 30 minutes. If a user is locked out, he's locked out for 30 minutes. We'll keep a password history setting of 24. That means that we'll have 24 passwords, which the user will not be able to use, before he can recycle his old passwords.

We'll have a minimum password age setting of 1 and a minimum password length setting of 7, plus complex passwords. These are very good settings in most environments, and will address most of the problems that we see on the product support side, as well as reduce your administrative costs in the real world.

For the high settings, it's only slightly different. The high settings increase the minimum password length to eight characters, and remember from the earlier slide, eight characters will get you into trillions of possible password permutations. Being able to guess a password in 42 days, with many trillions of permutations — you're looking at millions of attempts per second needed to try to break that password.

In addition to that, we have the lockout duration set to infinite, which means that if he is locked out, he's locked out until the administrator unlocks him. We have some good feedback from customers who have implemented this from settings, who had their lockout threshold set to 3 or 5; they're much happier with these settings. In addition to that, their security was enhanced by the complex password policy and the length of the passwords.

Recently we've seen an increase in volume of the external denial of service attacks against customer's user bases. We have some real-world recommendations here (slide 11) to assist you in fighting these attacks. One of the first recommendations is to enable a complex password on all accounts. No account should have a simple password.

Administrator blank passwords — an administrator account with a blank password should never occur. Change your administrator account passwords on a regular basis. These accounts are the keys to your kingdom. Lock your keys away, and your kingdom will be safe.

Rename your administrator account. Renaming your administrator account is a good practice to prevent the attacks against these accounts. Usually, by default, these accounts aren't able to be locked out. You can lock them out with some settings. However, preventing these accounts from being locked out, or renaming these accounts, saves you a lot of administrative problems. If these accounts are exposed, you've lost your keys to the kingdom.

The next item is to protect your environment with firewalls. I recommend blocking all ports, except the ones you actually need, through your firewall. Prevent access, especially to ports 135 through 139 on both UDP and TCP, as well as to port 445 on your routers and firewalls; block these ports from external access. These will help prevent the bad guys from enumerating your user account database.

Speaking of preventing them from enumerating your account databases, there's another setting called RestrictAnonymous. This setting, by default, is set to 0 in Windows. However, in a pure Windows 2000 environment or in a Windows 2000 and Windows XP environment, it's okay to set it to 2. However, this is with the proviso that some applications may rely on anonymous access to access resources. You may see that some applications are broken by this. I highly recommend that you test before you implement it. However, on the machines that are exposed to the Internet, you definitely should require RestrictAnonymous on there.

In addition to that, however, you also have to be aware of down-level trusts across your Internet, through a VPN; if you have RestrictAnonymous, you may have problems there as well.

The next item is protect site-to-site traffic with a VPN. In other words, if communication between a domain member in two sites is required, use a site-to-site VPN tunnel to securely connect site networks together. Don't open all the NetBIOS ports on the firewall. In many cases you'll notice that people will open their firewalls up, wide open, on the NetBIOS ports and RPC. Why even have a firewall at that point? Keep it closed, and use a VPN. We provide a VPN for you; it's free of charge. You can use routing and remote access for a site-to-site VPN tunnel. In the white paper I will also give you a link on how to set that up.

The next item is protect authentication and NetBIOS ports from Internet attack. On your edge firewalls and routers, block access to TCP and UDP ports 135 through 139 and 445; actually block them all, and only allow the ones you actually need. If you have no edge-filtering device, use the Internet Protocol security (IPSec) filters to block these ports. You can use the IPSec filtering to allow only the machines or IP addresses that you want to utilize these ports, greatly enhancing your security in this environment.

The next item is to protect authentication and NetBIOS ports from internal attack. Let's say you're in your environment and somebody is doing a denial of service attack internally on your network. You can protect these ports from an internal attack by restricting computers that are allowed to access these ports to just domain members. You can reduce the number of potential attackers from non-trusted domains inside your environment that might be attacking your servers.

The key thing here is to update your servers. Keep all your servers up to date with virus software and install Windows security patches to help prevent Trojan horses and viruses that would attack, if they get inside your firewall or security settings. One of the key lessons here is the Slammer virus — a lot of people learned from that they need be up to date as soon as possible.

On the next slide (slide 12), in the next part of this presentation we're going to cover authentication password verification, urgent replication triggers, Windows 2000 service packs and hotfixes, and forthcoming SP4, SP5, and .NET updates.

Password verification (slide 13). A non-PDC emulator will fail with the following status: WRONG_PASSWORD, PASSWORD_EXPIRED, PASSWORD_MUST_CHANGE, ACCOUNT_LOCKED_OUT. I'll give you a scenario here (slide 14). I'm a user and I try to present credentials to a resource in the domain. I send the bad user name and password. I make an NTLM request out to a resource, my file server. That file server will go out and make an NTLM request to an authenticating DC. The authenticating DC will look at its password for that user and say this is the incorrect password for this user. It will say "Before I increment my BadPWD account, I'm going to chain, and I'm going to go talk to my PDC emulator, because it should know the most current information about this user's password state." It chains it to the PDC emulator, in which case the PDC emulator also says "That's the wrong password; I'm going to mark my BadPWD count on this." Both the authenticating DC and the PDC emulator increment their BadPWD count.

What happens on Kerberos is slightly different (slide 15). The client is responsible, instead of the file server, for getting its Kerberos information. So the client says, "I want to talk to the file server. I'm going to go talk to my PDC, and I'm going to get a Kerberos session ticket for my file server." It gets a session ticket in the file server and it has access. But if it's a bad password the domain controller will tell it "That's a bad password, I need to talk to the PDC emulator," and it chains to the PDC emulator — very similar to the NTLM authentication. The key difference here is the client is responsible for going to get its ticket, versus the file server going to the domain controller to chain it to the PDC emulator.

Urgent replication triggers (slide 16). I'm going to cover three types of replication here. There's immediate replication, urgent replication, and we're coining to the phrase on-demand replication. When a password is changed, and this is immediate replication, it is pushed to the primary domain controller. Pushing means it is sent over the Netlogon secure channels to the PDC. My backup domain controller (BDC) makes a remote procedure call (RPC) to the PDC, which indicates the user name and new password. The PDC then sets this value locally. This is an immediate replication.

The following events will actually cause immediate replication to the PDC from the domain controller: replicating a newly locked out account; changing a local security authority (LSA) secret, which is changing the password; and RID manager state change — in other words we change the state of the RID master or move the role to a different box.

Immediate replication is not stopped by site boundaries. In other words, if I have replication scheduled at a certain time, and I hit an immediate replication event, I will immediately send that replication to the PDC, disregarding the site boundaries in time for site replication.

Urgent replication is a little different. Urgent replication occurs with a user password change, for example. On the DC that receives the change, it will make an urgent intrasite replication to its partners. In other words, it will urgently tell all of its colleagues that this is a new password for this user. These delay patterns are ignored, and an urgent replication is sent instead.

In other words, if I'm still waiting in my local site for the replication interval to talk to my partners, after an urgent replication event occurs, I will talk to my partners on the urgent change in the environment. There's a good article at the end of the presentation, on urgent replication triggers, that we can go into as well.

Kerberos negative caching (slide 17): in Windows 2000 Service Pack 2, they'll reduce the number of logon requests handled by the PDC. The Kerberos authentication pack implemented a negative caching mechanism to stop the forwarding of request to the PDC, if a bad password status is returned after one bad logon request for a period of five minutes. In other words, after the one attempt, it represses forwarding to the PDC emulator for five minutes.

This sounds like a really good behavior, except the only problem is that if the immediate replication to the PDC had not taken place, the Kerberos negative caching may cause an account lockout in this environment. So in SP3 we've kind of modified that behavior. There are a couple of articles: 306131 and 272065. These kind of explain why you would get potential lockouts to the Kerberos negative caching. For these environments I highly recommend, and we'll go into this a little bit later, installing SP3 and the latest QFEs.

Now most interesting features coming out for your environment are the changes that we have coming in SP4 and Windows 2000 Server (slide 18). The first one is RunAs Security Audit. You can audit the rogue client applications using the RunAs command with a special auditing event. In addition to that, we've made some auditing enhancements that are not going to be included in SP4, but will be included in SP5. The reason why we moved that back was because of the large amount of testing to accommodate this change of auditing.

What this does is determine what process on a computer is locking out an account. Joe will kind of iterate the painful process of trying to find the application that might be causing the account lockout, the bad password sent to the PDC and the DC. This will greatly enhance our ability to determine the process that's out there causing the issues.

We have some tools that Joe will go into later that will give us more detail. One is Alockout.dll. And Joe will go into greater depth on that as well. That's a tool that we're using until we have this auditing built into the OS. In Windows Server 2003 it is already there. But we're saving it for SP5 for Windows 2000 clients and PDCs.

Another tool that we have out is Acctinfo.dll. This has the ability to change a password on a user's computer account, on a DC in user's computer site. It also gives you additional information about the user, about his password, and about the current policies that are being applied.

The other replication that I mentioned earlier, and I didn't go into detail on it, was the on-demand replication (slide 19). This is represented in 812499. This is a recently released article, hot off the presses. The on-demand replication occurs when we reset a user's password, and we set the attribute so that the user must change the password on the next logon. It doesn't immediately replicate to the DC that may authenticate that user.

What this on-demand replication gives you — the scenario is I reset a user's account. He logs in on a remote domain. On that remote domain, his DC doesn't have the new password, so it goes and chains it to the PDC. The PDC says, "Yes, that new password is valid." However, at that point in the current behavior, we did not replicate that user's object. When that user tries to reset his password, he would fail, and he would get an error.

Now we do an on-demand replication. When that DC, in the remote site, goes to the PDC, it will check for the new password, see that there's a new password, and then the whole user object will be replicated to that DC. In that DC site, an urgent replication will take place. What happens is now we have an on-demand site where we don't have to wait for the replication topology to take place. That greatly reduces lockouts and administrative costs in your environment.

The next item is one that I'm really happy about, and that is the N-2. These are the last two passwords stored in the user's cache. What happens is actually an attribute in Active Directory. The authentication process will work like this. Just like in NTLM and Kerberos, my authenticating DC will go and check its BadPWD count; it will look at the password and say that's a bad password, and it will check the PDC to see if that password is good. However, before it increments the BadPWD count, it will determine if it is one of the last two passwords used by this user. If it is one of their last two passwords, it will not let the user in; however, it won't increment the BadPWD count.

What does that mean for you? That means that if I have a service account with an old password, he's not going to lock out my service account for my Exchange admin account, or my SMS account, or any third-party account. It also means that if I'm logged in on multiple workstations, and if I have the old password on one workstation and a new password on the other, that old password on the old workstation is not going to lock me out, because it's still in the password history. I have two changes of my password before that account may lock out that user. Hopefully in that time somebody logs on to that machine and resets the machine, reboots it in the meantime. But that will give you, if your password length is 42 days, 84+ days to hopefully get that user logged off or to realize that there are failures on that machine.

In addition, we have the new DS Client. It's going to be included with Windows 2003, but it's also described in KB article 323466. This is the Directory Services Client. It's not available on the Internet. However, if you call PSS and ask for that article, we'd be more than happy to send that to you. That article resolves a lot of client-side problems, which we'll go into a little bit later. Client-side bugs from Windows 9x clients — the Windows 95, Windows 98 clients that have caching — that send bad passwords to the DCs. That will relieve a lot of the client-side problems for your Windows 9x clients.

Before we let Joe give his section of this presentation, there are common causes for account lockouts (slide 20). We have the bad password threshold is set too low, which I've covered previously. We have multiple machines logged in with interactive session, so they're running processes that are throwing bad password accounts with the old logon. Outlook® Web Access, Outlook, Exchange, and IIS are using authenticating Web pages. It does a little caching there. There are also service accounts that are using a user name and password with the wrong password. We have disconnected Terminal Server sessions. We have scheduled tasks under the wrong credentials. We see persistent drive mappings with wrong credentials, as well as client-side and server-side bugs, which all would be addressed if you go to SP3, or preferably the KB article 812499. That has both the on-demand replication and the N-2 behavior, that is post-SP3. So for the N-2 behavior and the on-demand replication, get the KB article that I just mentioned —812499.

Joe, I'll let you go on from here with the troubleshooting.

Joe Vasil: Thanks, Mike. Before we get to troubleshooting, what I want to do is just try to set up a typical account lockout scenario that we see here at PSS, and that you folks have possibly seen in your own environments. That's the type of case where you or your help desk gets, say, 25 calls a day stating that user accounts are locked out for no apparent reason.

You have a multitude of accounts getting locked out every day. What are the best steps to determine the root cause of these lockouts and resolve the issue?

We have our scenario set up. Where do we even begin to troubleshoot these problems? As this "Troubleshooting Account Lockouts" slide (slide 21) suggests, the first step that I would like to recommend, even before looking into service pack and hotfix updates, would be to verify, as Mike has explained, the account lockout threshold setting. Obviously you have it set, or users wouldn't be locked out, so the first thing to do would be to verify that this setting is at the recommended threshold of 10-15.

Again, just some of the normal occurrences of account lockouts, processes, and authentication requests with, for example, Exchange clients and some other applications that are out there — if a user sends a bad password, it's possible that an application will try that password per binding. So if you have several bindings for a particular client-side process, say five or six bindings, that process could actually send out multiple authentication requests, even though the user only typed in the wrong password once.

Having an account lockout (AL) threshold of 3 or 5, it's possible just through normal process authentication that user accounts are going to be locked out. We started recommending the minimal AL threshold of 10-15. As Mike stated earlier, we've had a lot of positive feedback from customers that this setting has eliminated or at least decreased the total number of account lockouts that their help desk and that the administrator is receiving. That would be the number one thing. Just check the proper AL threshold setting, and if it's set too low, set that to the right threshold setting.

Number one on the list in the slide would be to take a look at the list of service packs and hotfix updates. There are an incredible number of account lockout-related updates in Service Pack 3. We'll go into a little bit more detail on the next slide.

The next step would be to enable auditing so that we could start tracking down the source of account lockouts. The whole intention when troubleshooting account lockouts is to try to narrow it down to the machines involved with the lockouts. There may be clients logged on to multiple machines, and you may not know it. The user getting locked out may not know it. "Yes, I logged into that machine a month ago and it locked at desktop," and they've since changed their password, and the application is running on that other desktop and is still using the old password, locking out the user account.

Auditing and even Netlogon logging will provide a means to pinpoint the exact machine from where the bad credentials are being sent. If the service pack and hotfix updates don't resolve the issues, then auditing, NetLogon logging, and even Kerberos logging may be helpful in some situations. We'll talk about that in a little bit more detail.

Then, of course, the last step would be to start gathering and analyzing the Netlogon logs and the Kerberos logs.

We'll just go into a little bit more detail in each of these troubleshooting steps. The Windows 2000 service pack and hotfix recommendations (slide 22), again, there were about 10 or so account lockout updates in Service Pack 3 that dealt with anything from how we handled locked and unlocking accounts and the urgent replication triggers that are incorporated with Service Pack 3, as well as some issues that we had dealing with BadPWD count resets on authenticating domain controllers, that setting being reset back to zero. Those are just to name a few.

There are several others that are incorporated into Service Pack 3. That's why it's highly recommended that you apply Service Pack 3 to any machines involved with the account lockouts. That, of course, would be the domain controllers, any resource servers, any Windows 2000 resource servers, or any Windows 2000 workstations involved in the lockouts as well.

Of course, it may not always be feasible to just roll out Service Pack 3 throughout your environment, so at the very least we recommend the fix per 327784, which is basically what we call the LSA/Samsrv bundle. Actually, this fix contains about 25 files. This particular QFE would be a rollup of all the account lockout-related fixes that are in Service Pack 3. If you can't roll out Service Pack 3, then 327784 would be the alternate there.

Follow-up information: Please refer to KB article 812499, which is the most recent LSA/Samsrv bundle.

The absolute best bet would be Service Pack 3 plus the QFE for 812499. That would probably eliminate most of the account lockouts, given the fact that KB article 812499 contains, as Mike mentioned, the N-2 password history and a single user object replication as well. That is the absolute best bet, and you'll probably see that that would eliminate a great deal of the account lockouts.

One other, back on that previous slide, would be the Windows 95/Windows 98 DS Client update for 323466, as Mike mentioned. That is the latest directory service client for Windows 95/Windows 98 machines. There were some issues with the earlier build, which was included on the Windows 2000 CD, where a user could type in one wrong password. But due to some of the DFS mechanisms in authenticating processes in that original DS Client, we would actually increment or send three logon attempts per request. So they would type in one wrong password, and if your account lockout threshold was 3, that account would be locked out. The new DS Client takes care of that as well. If you run a Windows 95/Windows 98 client with the DS Client, it's highly recommended that you update that to the latest build.

We have our hotfixes in place. We've updated the DCs, the client machines, but we're still getting a lockout. The next step, and you can do this right at the beginning in case the updates don't resolve the issues, is to have auditing in place (slide 23). The main thing that you want to set for auditing would be at the domain level, and that would be for account logon events for which you want to audit failures. You really don't want to flood your event logs for successful logon events. At least for account lockout purposes, there's no need to audit successful logon attempts for troubleshooting account lockouts. You're only interested in the failures. So for account logon events, you should only audit failures.

For account management it's recommended that you audit successes; that would show you the account actually being locked out. It would register an account lockout event and then the unlock event.

The third thing that you want to audit would be logon event failures. The differences between account logon events and logon events: account logon events will be registered on domain controllers, and logon events on the member servers and workstations in your environment. Again, these would be set at the domain level, so we have that covered domain-wide.

The next thing, and you would be surprised to know that even though we're in a Windows 2000 Kerberos environment, the majority of applications out there still make down-level calls to LogonUser. That's an API call that a lot of applications will call, LogonUser. That can trigger a down-level NTLM authentication. The greatest majority of our calls in PSS, for account lockouts, are related to NTLM authentication. It's highly recommended that you enable Netlogon logging (slide 24) on the domain controllers. Which domain controllers? If you have 200 domain controllers in your environment, I don't recommend that you enable this on every single domain controller. What you need to do is try to determine exactly which domain controllers are involved in the account lockouts.

Going back to that authentication request or the authentication chaining process that Mike was talking about with NTLM and Kerberos logon events, both of those authentication processes with Service Pack 3 will chain the request to the PDC. So you know in any account lockout domain, my account lockout case, that the PDC will always see those failed authentication requests. So the PDC would be the first domain controller to enable Netlogon logging.

For the other DCs, what I usually recommend is that if you have a particular site where you're experiencing most of the account lockouts, you should focus on just that site. If you have hundreds of users getting locked out throughout your entire environment, you don't want to troubleshoot a hundred account lockouts throughout the entire environment all at the same time. It's impossible, or it's going to require a lot of work.

Try to narrow down the scope of your focus when troubleshooting these. The PDC would be number one; then try to narrow it down to a particular site, if you have a site where a high number of lockouts are occurring, where you know those users log on and they're being authenticated, and then the PDC emulator, and then the DCs in that site.

To narrow it down even further, run the Set L command on some of the workstations, try to get an idea of exactly which domain controller they're being authenticated by, and then enable Netlogon logging on that DC or DCs. But always on the PDC emulator, and then try to pick a particular site or something of that nature to focus on, at that point.

In the enterprises with less than 10 DCs, my recommendation would be to enable it on all DCs. That way, wherever the lockout occurs, you can have the Netlogon logs that correlate to that lockout. The Netlogon process is not resource intensive. At the maximum it takes 40 MB on a disk drive. So for the logging processes we truncate the logs, so we'll make two logs maximum, by default, at 20 MB apiece.

To enable to logon logging, there's an nltest /dbflag, and then the flag you want to set is 2080ffff, which would be my recommendation for the most verbose. You can scale that back a little bit. It's a bitmap mask for this, so the higher this value the more verbose the logging. So 2000fff or 2080ffff would be my recommendation for that.

Kerberos logging (slide 25) would be necessary if you suspect specific Kerberos problems, problems with the KDC, or the Kerberos protocol itself. This is enabled. It's client-side logging, so you would have to enable this in the registry on each machine, whether you want to do it on the DCs or the client machines. There are ways to script this, so you can write a Visual Basic® script. You can create script to put into a Group Policy that applies to all the client machines as they are starting up or something of that nature, to enable the Kerberos logging on all the client machines. Again, it's only necessary if you suspect actual Kerberos problems.

We have our logging in place. We now basically sit back and wait for an account lockout to occur. Then after that lockout does occur, at that point we have some tools that I'll go over in more detail, on how to gather and analyze these logs (slide 26).

Actually, these tools, Mike is working on getting these wrapped up into a downloadable package that will be available on our Web site. We're working on assigning that package where all these tools will be included. You can go to our Web site when you download the account lockout white paper. We will have a collection of all the tools I'm talking about in this presentation available to you publicly as well.

The first one, Use EventCombMT, that's a tool that we use internally. And I'll go into it in a little bit more detail in some of the upcoming slides, but basically you launch it and it gathers event logs from one central location from all DCs.

Another one is the Lockoutstatus.exe, to get a little bit more information on the actual status of a particular user account and the BadPWD account on the domain controllers. This kind of helps you figure out exactly which domain controllers are involved in an account lockout as well.

Gathering Netlogon files. We have all the logging enabled, so let's go out and gather them from the PDC and the DCs involved in the lockout. Again, you could use the Lockoutstatus.exe while the account is still locked. You can determine exactly which DC is involved in the lockout and get the log files from those DCs.

Nlparse is a utility to parse through the Netlogon logs. It's very helpful. We have more detail on that in upcoming slides as well. What we're going into here are just some samples on what we are looking for in the Netlogon.log. Let's dive into this in a little bit more detail. We have lockout, we had Netlogon logging enabled on the PDC and on the authenticating DCs, and in this case we have it on a member server, because we were able to track that down (slide 27).

We have these logs, so the first log I'm going to look at is the log from the PDC emulator (slide 28). What does this log have? It shows us, obviously, that we should know exactly what user account was locked out, because help desk has called and said User1 was just locked out. You ran Lockoutstatus.exe against the User1 account to determine exactly which domain controllers were involved with that lockout, one being the PDC emulator and then another DC, DC003.

So you have the Netlogon logs from two DCs; you then parsed through those logs for any instances of the User1 account. In our domain, Tailspintoys\User1, you see it logging on to the actual client machine, Machine-006. That's the machine that user is logging on to, and you can see this as a network logon.

A transitive network logon means that this authentication request was sent, it's a transitive logon request, meaning that this was chained from somewhere. In this case I'm looking at the Netlogon log from the PDC emulator. You can see that this logon request was sent by DC003. So that means that this user was actually trying to authenticate, or that this logon attempt is actually being sent to the PDC emulator from the authenticating DC003.

Now the C000006A events are the key events that you want to look for. The 6As and the C00000234 down at the bottom; that 234 is an event indicating that the account is actually locked out. The 6A events are unknown user name or bad password. So you can see from this we have five C6A events, so guess what the account lockout threshold is set to in this domain? Five. So you see that you get five bad logon attempts, then the account is locked out. In this case, we should increase our lockout threshold to 10 or higher, which may eliminate this lockout.

Continuing on with the Netlogon logging flow (slide 29), in this situation I see this log; I see that they've been requested as coming by DC003. My next thing would be let's go get the Netlogon log from DC003. I go to DC0003 and get its Netlogon.log. I parse through User1 for any C6A and 234 events for User1, and I see that this logon request at DC003 actually came from MEMSERVER01.

What have we determined here? We know exactly what domain controllers are involved with this lockout, but we also know that for something running on Machine-006, it is using the Tailspintoys\User1 credentials to try to access a resource on MEMSERVER01. Remember how NTLM authentication works. The member server is responsible for authenticating that user's credentials to his authenticating DC.

If you also look at the time stamps, we know it's a process, because there are five authentication attempts within a one-second period. I don't think there are too many users out there who could actually type in a user name and password five times within a one-second period. So this is very important to understand that we know this is an application, more than likely authenticating on the users' behalf, given the fact that those authentication requests are occurring so quickly.

Moving on, what I would do at this point is look for patterns. You get another call coming in for User2, you go through the same process. You pull the Netlogon log from the PDC, you check it, and you track it down to the authenticating DC. You see that this MEMSERVER01 is always involved with your lockout. This machine seems to always show up in the Netlogon logs. It's always sending back credentials to domain controllers. Right there, you've accomplished quite a bit. You've narrowed it down to the MEMSERVER01. Something is running or clients are trying to access this machine, or a process is running on the client machines that need to have access to particular member servers, in this particular case.

At this point what I've done is go to MEMSERVER01, enabled Netlogon logging on that, and then this more or less just shows you that the request is coming from Machine-006 (slide 30). Again, we see that it's still several attempts within a one-second period, so we know it's some process running on these client machines that is hitting this particular member server. Now it's a little bit more tricky.

So now it's a little bit trickier, and we'll go into more details. After you've narrowed it down to a particular machine, we'll go into the tools you use. What I want to do is jump back to auditing, because not all account lockout events are going to be caught in a Netlogon log. A lot of them are going to show up in the audit logs. If it's not NTLM authentication, you're going to have to resort to auditing. It works basically the same way.

What you're going to look for in the audit logs is, again, you're going to go to the PDC. You're going to see event 675 (slide 31). So if you have auditing set up using the steps that we covered earlier, you're going to see event 675 preauthentication failures. The thing you're going to look for there will be an IP address. This is the IP address for the client computer, where their own credentials are being sent from.

Keep in mind that could be an authenticating domain controller, sending or chaining a request to the PDC. Just because you see an IP address, it's not always the client machine; it may be an authenticating domain controller.

What you do is go to that authenticating domain controller, get its audit logs, and look for the same 675 event. You can match up the time stamps, and that should show you the client machine that's sending credentials to the authenticating DC. Again, it's the same process as Netlogon logging. You're just tracking it down from the PDC to the authenticating domain controller to the member server, or from the authenticating domain controller to the client machine, to try to figure out exactly where these wrong credentials are coming from.

Again, the 644 events would come from account management or user account management success. That's just going to show the account being locked, and then if an administrator unlocks it, it will show the account unlocked events as well.

These slides (slide 32) just give you a sample of the specific 675 events. Again, down at the very bottom you can see the exact client address. That client address could be an authenticating DC or anything of that nature. What I would do at this point is go get the same audit, the security log from the machine that has the IP address Go get the audit log from that box and look at the same event. You would see the same exact date and time stamp. And then see what the source client address is on that box. And then use that address to track down the source of the bad credentials.

A couple other things. There are preauthentication types and failure codes that you can look for. We'll get into that in a little bit more detail. This again is just a sample of a successful (slide 33) account management audit of 644, showing that the account is actually locked. This would actually show a machine name as well, where that account is locked. After you get it down to the client side, you can look for 529 events, which would be the "Unknown user name and password" (slide 34). Again, we're looking for patterns here. We get a lot of these 529s and 675 events. Are several of them occurring within a one-second period? That will tell you whether it's a process or not.

If you're getting five or six bad logon attempts over a 10-second period, then you could say this user has mistyped the password several times. Again, you're just looking for patterns here to try to get a good indication on what's causing the lockout. Again, look at the logon type and the logon process in these events as well. The logon type here in the 529 (slide 35) you'll see as 2, the logon process is User32. In this case, User32 usually stems from typing in the wrong user name or password.

On the next slide (slide 36), the event 531 is just an account management type of event, indicating that the user account has been locked out or disabled. Then you would also get a related event, if you were to unlock the account.

Again, this is a client-side audit. This shows the actual audit (slide 37). Here again, I think the logon type and process are the important ones to look at here.

The logon types (slide 38) — if you take a look at this, logon type 2 is interactive. We also have a network logon. Was it a logon through a proxy? It has just the various types that may give you some idea, if you know the logon type, some indication of what's locking out the account. Again, in that logon type, down toward the bottom of the event audit, if you see a logon type 2, this means it is an interactive logon, based on this slide.

On the next slide we'll see the logon process (slide 39). We have the default authentication package, which is NTLM. Was it a Kerberos logon type process? User32, again, is basically just an interactive logon where the user has typed in the wrong password at the keyboard, going through WinLogon and MSGINA. As you noticed in that event 529 earlier, you saw the process was a User32.

More common ones are the advapi logon process, and that will give you an indication that there is a process or application making a call to LogonUser that has sent wrong credentials. Getting to this point with these client-side auditing event logs, this was what Mike was talking about, where we're working on some more enhanced logging to give you the actual process ID, application name, or executable that's sending the wrong credentials, making API calls to LogonUser. With these audits you get an exact PID or exact process name that's sending the wrong credentials. At this point we can only get it down to this level. We know it's a User32 process. We know that it's some application making the call to LogonUser. So it's still rather difficult to determine exactly what process or application it is that's sending the wrong credentials. Hopefully, we'll see some changes in the auditing forthcoming.

Again, this is just a sample of the Kerberos events, if you expect any problems with Kerberos itself (slide 40). For a typical Kerberos event, we see KDC error client revoke, just meaning that the user sent the wrong password. There's quite a bit of detail in Kerberos logging, and there's an article on it if you query on our Web site in the KB. Query on "Kerberos events," and you can get a little bit more detail on what the error codes for Kerberos events are, if you suspect a Kerberos problem. Most account lockouts that we see are just plain wrong user name or password (slide 41). Most of the lockouts that we see aren't specifically a Kerberos problem. Again, there are some articles out there that will give you more information, if you need it.

Some questions to keep in mind (slide 42). Do the logon attempts occur seconds apart? Or are there many 6A events within the same second? If they're spread out, then chances are the user is typing in the wrong password and you need to go have a talk with that user. If there are several 6A events, meaning the events in the Netlogon logs, the user name, bad password events, and Netlogon logs, if several of them are occurring within a one-second period, you know it's an application or a process. It's kind of important to understand exactly what's going on there.

What computers are the 6A events coming from? Start with the PDC emulator, look there, see what they're being sent in the via section of that log. Then go to that authenticating DC and do the same thing: parse through to see which machines are sending the actual request to that authenticating DC.

What client computers are appearing in the Netlogon logs? You may find that there's a rogue machine out there sending credentials on behalf of this user. The user may not know that they're even logged on to it. It could be a kiosk center, or some machine out on the network that they're no longer supposed to be using. You wouldn't believe the number of cases that we have where you say these credentials are coming from this machine. Then the administrator or the person that we're working with does some research and finds out that it's some box way back in the corner that they didn't even know existed, and the user had logged on a long time ago and has since changed their password. That's just one of the good things that the Netlogon logs can show you — exactly where those credentials are coming from.

What server is the client getting a bad password against? Again, is this an Exchange server that is showing up on the Netlogon logs, or it is an IIS server? Is there any particular file server or anything that all these lockouts seems to revolve around? So again, those are things to note and to keep an eye out for.

What accounts? Obviously, the account is being locked out. Then again, there are the patterns and things of that nature that we've already discussed.

Account lockout tools (slide 43). This is the good part here. I would say that within the past year or so our development team and some of the other internal groups here have created a lot of really nice tools to help us with our account lockout troubleshooting, monitoring, and user state gathering. In this list is LockoutStatus.exe, and this will be in the .NET resource kit, and we'll go into all of these tools in more detail. I just want to list them all. So there's LockoutStatus.exe, ALockout.dll, ALOinfo.exe, and Acctinfo.dll, which will be in the .NET res kit, and we'll talk about that shortly.

Again, all of these tools will also be available in an account lockout resource on the Web site. NLParse and EventComb we've already talked about a little bit. The FindStr command can be run against Netlogon logs. Of course, Replmon and Repadmin just make sure our domain and forest-wide replication are working successfully. And Network Monitor comes down to needing to gather a trace.

LockoutStatus.exe (slide 44) pulls up a GUI. You run this against an account, and the best thing to do is run this against an account that's locked out. On the next slide we have a screen shot of it. It displays multiple facets of the locked out account, the lockout threshold, and which domain controller actually did the lock out.

It assists in finding computers involved in the lockout, or the domain controller specifically involved in the lockout. Let me bring up a screen shot of this (slide 45). In this particular screen shot we see this was run against, in the blue title bar there, User1. User1's account was locked out, or it was still locked out at the time I ran tool this against User1. It's just a simple standalone executable. If you double-click it, it will pull up this GUI. You go to File, Select Target, type in the user name that's locked out, and then press F5 to refresh. It will go out and gather information from all the domain controllers. It's multithreaded, so it doesn't take an incredibly long time to run, under a minute or so in most cases. It will return. And here you can see that I have two domain controllers in this test domain — DC1 and DC2. The user DC1 is my PDC emulator, and it just so happens that with this particular user authentication it is authenticated with the PDC emulator.

But you see the BadPWD account is 10 and that the user state is locked. That means that I have an account lockout threshold of 10, and for some reason this user has reached that threshold and this account is locked. You see over on the far right where it says Orig Lock DC1? That's the DC that originally locked the account. If this had been a chain request, in most cases for users that don't authenticate against the PDC, you would see DC2 would have its BadPWD count at 10, and the PDC emulator would also have its BadPWD count at 10.

In most scenarios there would be two domain controllers showing a BadPWD count of whatever your account lockout threshold is set to. So you'd know those are the two DCs, the PDC and the authenticating DC, that are involved in the lockout. Those are the machines you need to get your Netlogon logs and the audit logs from. Then you can dive in and determine exactly which machines are involved with the lockout.

Alockout.dll (slide 46): If you remember a while back I stated that after you get this narrowed down to a client machine, our current auditing, unfortunately, doesn't give you the exact PID or process executable or anything of that nature that shows exactly the process that's making bad calls or sending bad credentials to LogonUser.

Alockout.dll is a file that you can put on the client machine. You restart the boxes. It will actually filter out those calls and give you a log file in the Winnt\Debug directory. It's called Alockout.txt, and it dumps it to a text file, and it will provide you some more information on applications that are sending the wrong credentials when making LogonUser requests. Again, it's a tool in the collection that you can use to try to narrow down exactly what process it is.

Aloinfo.exe (slide 47) is a pretty cool tool. Again, most of our account lockout cases occur after users change their passwords — 10 minutes later their account is locked. That's because there is some process or application running on the box that's still using the old credentials. Aloinfo can be run as a command-line utility; you can run it against your domain, and it will dump the list of user accounts and their password age. If you have a maximum password age, you can use this utility kind of as a proactive tool. You can determine that five users' passwords are going to expire in the next two days. You put Alockout.dll on those client machines. Hopefully you can get some information, if there are applications running on the box that are going to continue to use the old password.

Accountinfo.dll (slide 48) is a really nice GUI extension to the user properties in Active Directory. You place this in the System32 directory; again, this is going to be in the .NET res kit. It can also be used on Windows 2000 DCs. We place this in the System32 directory. You then register the .dll at a command prompt. And rather than talking about it, let me show you what this tool does (slide 49).

It creates an Additional Account Info tab on the user properties. So what does this show us? It shows us property set, last logon, last logoff, and last bad logon attempt. It shows us the user account control type. It shows us the logon count. And it shows us the bad password count on a particular domain controller for that user.

There are also three other buttons here. The one down on the bottom right is the Account Lockout Status button. And if you have the AlockoutStatus.exe in your System32 directory or path directory, you click this button and it will pull up that account status window automatically, and then you can see the DCs involved with the lockout there. It's a pretty nice tool.

The Domain PW Info will display all the domain password account lockout password policy settings. Set PWD On Site DC — let me just go through and show you, we have screenshots of these. This is the domain password information (slide 50). If you click that Domain PW Infobutton on that account information tab, and then if you click Change PWD on a site-specific DC, it will bring up this box (slide 51). And you can just type in the user's computer where that user is logged on to. You can just find the site where that user is logged on.

But the ideal here is that you always want to reset an account or change user's password on a DC in that user's site. That way we give urgent replications throughout that site, and all DCs on that site would then have the user's unlocked account or a new password. So this is really a good tool to find out exactly what site a user logs on to. Your help desk crew may not know. You may just say, "I logged on to this machine." Well, they don't know what site you logged on to, so that tool is pretty handy.

Again, EventCombMT (slide 52). You launch EventCombMT.exe. It has some built-in searches. You go to searches, and then there will be a drop-down window; you select Account Lockout Searches. These are the event IDs that it will go out and comb for. Select a search. There will be a list of domain controllers in the Select To Search box (slide 53). It will go out and it will automatically present a list of all the domain controllers in your domain. You can remove them if you want, or just search them all for account lockouts. The event IDs are already there; the 529s, the 644s, 675s, and so on are already there. You run that and it will automatically gather those events from all DCs to one central location and dump them into a text file in the Temp directory on drive C. It's a good way just to centrally gather logs.

With Nlparse (slide 54) you use the parse Netlogon logs. Let me just show you this one; instead of talking about it I'd rather show you (slide 55). Again, this is all used for Netlogon logs. A simple executable you pull up; it's going to bring up this GUI here. You click Open; you then point to the path of the Netlogon log that you want to parse. Then the events relative to an account lockout are the C6A and the C234 events, so we check those. We click Extract and it will extract all those events to a .csv file, in the same directory where the Netlogon log files exist. It's a really good way to parse through logs and find out exactly what you're looking for.

On to this slide (slide 56), the FindStr command is very useful in parsing through several Netlogon logs. If you have two DCs involved in account lockouts, or if you are not exactly sure, you just enable Netlogon logging on five DCs in your domain — and that's all the DCs you have. So you just gathered all the Netlogon logs from all those DCs after User1 was locked out. You could use the FindStr command against the User1 string. And the /I is to ignore case sensitivity.*netlogon.log*.log. You put all your Netlogon logs from all your DCs in one directory, run this against all the Netlogon logs, and it will output every instance of User1 to a text file, so then you can parse through Netlogon logs that way. It's much easier than going through five different Netlogon logs separately.

Again, Replmon or Repadmin (slide 57) are just used to verify and make sure that you don't have any replication problems, because obviously that could cause some problems with account lockouts as well.

Network monitoring (slide 58). If you want to be proactive and set up a trace and try to catch some of the resources that an application is using, you may see in that network trace a particular path to a network application. Then you'll see Access Denied or status=bad password come across the wire. If you know what exact directory we're trying to hit in the trace, then there's a good indication that you're going to know what application is trying to access that directory, if it's a networked application. Network monitoring can be used if you know exactly where the account lockout is coming from, and then you can be proactive in setting up that trace ahead of time.

This would be proactive, so the user account is not yet locked out, but you're expecting it to be locked out. Therefore you can set this up.

That's about it from the troubleshooting standpoint. We have several slides on the additional resources. I'm going to turn the floor back to Mike, and he'll go over these.

Mike: Thanks, Joe. Actually, because we're running tight on time, I'm going to let you guys look at the additional resources (slides 59-64). I've included most of the articles that you'd find on a lot of the bugs, and KB article as well. Remember 812499. That's the new SP4 behavior.

The things to keep in mind are: What are your settings and are they valid? Do we want to leave lockouts set at 3, or set it to 10? Then make sure you're on the correct service packs. Ideally, before you even call PSS, enable the auditing, which will help to go through your logs. It may speed up your time to resolution dramatically, if you do those things ahead of time. I highly recommend article 812499.

Thanks a lot. We've enjoyed presenting this portion of the presentation, and we'll open it up for questions.

Otto Cate: Before we jump into the Q&A, I'd like to share a couple of quick program notes with our listeners. If any of the details on the PowerPoint® slides were difficult to view in your browser today or you'd like to simply have a copy of the slides, they are available for download [click Past Support WebCasts] at support.microsoft.com/webcasts/ then go to today's WebCast. As Mike mentioned, at the end of the deck, there are several additional resource slides that cover the KB articles that were mentioned during the presentation.

The Q&A portion of the Support WebCast is intended to encourage further discussion of the topic that we addressed today. In addition, one-on-one product support issues are outside the scope of what we're able to address. If you do find that you need some more complex technical assistance, feel free to submit an incident on the Web, or contact Product Support Services directly and speak to a support professional.

Let's get into the first few question we have here: Do you know at this point what we're looking at for a release date of that white paper you mentioned?

Mike: Right now it's in tech review. It is based on the launch of Windows Server 2003. So we're looking at probably mid-next month, March 2003.

Otto: For Accinfo.dll, do you require Windows 2000 SP4, or will it run on SP2?

Mike: It's been tested on SP3. Joe, are you familiar with any errors with SP2?

Joe: No. I know it's not a requirement for Service Pack 4. Most of our testing has been for Service Pack 3. I'm not aware of any issues with Service Pack 2, though.

Mike: There are definitely some good reasons to go to SP3, in terms of account lockouts.

Otto: Can you kind of give us a brief overview of the problem that leads to the fix in 323466, Window 9x clients with the DS Client installed, and where the bad password threshold is set to a low value, like three attempts? The account lockout would be invoked because of a bug in DS Client.

Joe: Yes, there was an issue with the original DS Client that shipped on the Windows 2000 CD. If a user was running that DS in an Active Directory domain and the user typed in their password wrong one time, then based on some of the authenticating processes in the DFS — it was actually a DFS client piece that's built into the DS Client. We would see multiple logon attempts or authentication attempts go across the wire, even though the user only typed in the password once. If you had your threshold set to 3, and the user typed in their bad password one time, it would actually send that bad password three times, and the account would be immediately locked. That's with an account lockout threshold of 3. Again, that's been resolved in the new DS Client. That's in 323466.

Otto: Why would you apply account policy at the domain level? If it's undefined there, you could set different levels for users than critical service accounts.

Mike: The account policy is by default only actually viewed on the domain level. Processing a Group Policy — even though it's there for an organizational unit level, it's not processed. The settings inside the Group Policy engine and also the security mechanisms just ignore those settings in an organizational unit-level environment. It's just by design. Because the domain is the security boundary, that is why the account policies are applied on a domain level instead of an organizational unit level.

Joe: Another thing that I could think of is you don't want to move domain controllers to an organizational unit, so you would want to get some audit events on your domain controllers, as well as the client machines. I think you'd really want to do this at a domain level anyway, to make sure that we get the audit events on the machines that we usually gather them on.

Now they can set auditing on individual machines. So that may be helpful as well. Instead of doing that at a domain level, if they wanted to they could set auditing locally on a particular box. Mike, do you think that would work?

Mike: On account-specific properties, it's only on the domain level.

Joe: I see.

Mike: For auditing, you can set an organizational unit, or anywhere you want, at that point.

Otto: Next question: If Passfilt.dll is modified in Windows 2000, how will the upgrade to Windows Server 2003 be affected?

Mike: I personally haven't tested that, but actually your modified Passfilt.dll file should still be in effect. I don't see that it would be affected by that. We could probably do a test on that. Joe, do you have anything?

Joe: As far as the password filters that we would set in policies, because we wouldn't use Passfilt on Windows 2000 DCs, right?

Mike: He's talking about if he wrote his own custom Passfilt, to put in Windows 2000.

Joe: I see. That I'm not aware of.

Mike: I imagine the logic would remain the same on an upgrade between Windows Server 2003 and Windows 2000. I don't see why that would be affected. I personally haven't tested it, but I'd be willing to bet that it would remain the same.

Otto: How can you change the local admin password remotely?

Joe: I'm not sure I understand the question, how can you change the administrator password remotely?

Otto: Yes.

Mike: He wants to dynamically change, through Group Policy, a password for the local administrators.

Joe: For the local administrators.

Mike: There is a Group Policy setting to rename them, and I also think that you can set the password for them at that point as well.

Joe: That might be one that we want to take offline, because I'm not sure how to change that on a local administrator. I know we can change the administrator name, but I'm not sure about the password.

Follow-up answer: We recommend changing your privileged accounts passwords on a regular basis. The following resource kit utility can accomplish the change in local administrator and other accounts. This can be done through by using the Cusrmgr.exe resource kit utility.

Use the uppercase -P and specifically set a password. Do not use the lowercase -p switch.

Sets a random password to a user

usage: -u UserName [-m \\MachineName] \\ default LocalMachine

Resetting Password Function

-p Set to a random password

-P xxx Sets password to xxx

User Functions

-r xxx Renames user to xxx

-d xxx deletes user xxx

SetProperties Functions

-c xxx sets Comment to xxx

-f xxx sets Full Name to xxx

-U xxx sets UserProfile to xxx

-n xxx sets LogonScript to xxx

-h xxx sets HomeDir to xxx

-H x sets HomeDirDrive to x

+s xxxx sets property xxxx

-s xxxx resets property xxxx

where xxxx can be any of the following properties:






GetProperties Functions

-l 1 Lists Username, fullname, Comment, Profile and Scriptpathreturns 0 on success

Moving on here: Any comments about 7-character or 14-character passwords versus 8 characters, owing to how the password is stored in 7-byte increments?

Mike: I'm not sure on that one. I do know that Windows NT had a problem with the way it was split up, but for Windows 2000, I don't recall that it has the same issue. I'd probably have to research that; it's been a long time since I read that info.

Joe: I'm not aware of having any problems with an eight-character password. We haven't seen any problems at all with Windows 2000 having an eight-character password.

Follow-up answer: I believe you are referring to the legacy LMHASH that uses a 16-byte hash of passwords with the first seven characters creating the first 8 bytes of hash. The "NThash" (used in Windows 2000) is an MD4 hash of the whole password, instead of parceling it up into pieces. So the entire eight-character password is used for a single hash rather than having two separate pieces.

Otto: Is there a Microsoft recommendation for reset count?

Mike: The time they reset the password?

Joe: Or reset the lockout account?

Mike: Yes. We recommend 30 minutes. It's on slide 10.

Otto: Is there a list of all the special characters that are allowed in a password?

Mike: I think that's in the help file. If you go into help in Windows XP, look under "complex password".

Otto: Can you comment about the guest account being enabled?

Mike: I'd prefer not to enable it. As far as a comment, I would be very specific on what the guest account is allowed to do. I would lock down that account very tightly. Usually the first thing I do on a box is rename it and make sure that account is disabled. I also do the same thing with the administrator. I usually disable that account as well.

Otto: If the minimum password age is set to 1, the user is not allowed to change their password until the day after the admin resets it for them. Is there any way to log on, if the select user must change their password on the next logon attempt? Because they must wait one day to change their password. Have you noticed that?

Joe: I see. The administrator resets their password and then they set "user must change password at next logon." If they have the minimum password age set to 1, what's the result going to be?

Are they saying that it fails, or it won't let the user change their password? Because it's my understanding that the user is changing their password specifically. In that scenario the administrator is setting the original password, and then setting that account so that the user must change the password at next logon. In that case the user should be able to change their password at their first logon.

Otto: Can you explain restricting anonymous access and what it does? We have a Windows 2000 domain, but we also still have some Windows NT 4.0 machines.

Mike: If the value is set for 2, it prevents access. Really, with Windows NT 4.0 on the domains, I wouldn't set it at 2. Use a value of 1. What would happen with 2 is you would have to present explicit credentials when trying to access a resource. Applications that try a null session to access resources would fail. You would have to connect to that resource with credentials. Then that application should be able to use the resource correctly.

Instead of setting it to 0, at the very, very bare minimum, I'd set it to 1. In a Windows 2000 domain I'd set it to 2, if possible. In the additional resources slides there's an article on RestrictAnonymous (246261). That will go into detail on the effects on operations with RestrictAnonymous set to 2 or 1.

Otto: What about applications that authenticate through LDAP?

Mike: If they're authenticating through LDAP, they're still going to have to provide a user name and password to access resources. It's still the underlying security. If their passwords are incorrect, you're probably still going to end up with a Kerberos lockout, if it's throwing a bad password. Joe, do you have an opinion on that?

Joe: Yes. I think the process would be the same. The user would be making an LDAP request or a BIND request. You can make anonymous LDAP queries as well, but in this case I'm assuming you're talking about using a domain credential. I'm not sure what they mean by "What about LDAP requests." They could be coming over the Web, for that matter. Are they talking about troubleshooting that type of request? I think that you would have to just key in to find out again exactly where the lockout is occurring from. That LDAP server, whether it's a domain controller or not, would need to be analyzed to discover where that logon request was coming from.

I'm not sure I fully understand the question, but if it's just an LDAP request, and they're providing credentials, those credentials would still be validated by a domain controller; they normally would be, in a normal logon scenario.

Otto: What events cause immediate replication and what events cause urgent replication? The slides were a little unclear on this.

Mike: There's a good article on exactly that in the resources slides (232690). The immediate replication is going to be a password change, where I'm going to do an immediate replication to the PDC emulator. Anything that has to automatically contact the PDC emulator is going to be an immediate replication.

Let me get the article number for you here. I'll post it on the Web, unless, Joe, you have it off the top of your head?

Joe: Basically an urgent replication is the replication within a site where we ignore the defaults, which I believe is a 15-minute delay period for sending out a change notification. With urgent replication, we ignore that default delay, and we send out change notifications to all replicate DCs, even the PDC emulator. Then those DCs will pull that change from that DC. But we still are restricted to site boundaries. An immediate replication would occur over RPC. We would ignore site boundaries, and we make some pull changes or send changes immediately.

With urgent replication, we just ignore the delay for sending out a change notification from Windows 2000; I believe it's 15 minutes, and that can be adjusted in the registry. With immediate replication, it doesn't even consider site topology and replication topology. We just immediately replicate over RPC.

Follow-up information:

Immediate Replication:

When a password is changed, it is "pushed" to the PDC. Pushed means that the password is sent over the NETLOGON secure channel to the PDC. Specifically, the BDC makes an RPC to the PDC, which indicates the user and the user's new password. The PDC then sets this value locally. This push mechanism, also known as "immediate replication" is independent of Windows 2000 replication.

Immediate replication between Windows 2000 domain controllers consists of the following events:

·         Replicating a newly locked-out account

·         Changing an LSA secret

·         RID manager state changes

Urgent Replication

Urgent replication (used on password changes, for example) consists of the DC processing the originating update sending an urgent notification to its intrasite replication partners, indicating that there are changes available for them to pull. Normally, such a notification following the originating update would be sent after a certain interval governed by the two "Notification Delay" related per-DC registry key values. The default "Notification Delay" in Windows 2000 is five minutes (it is 15 seconds in Windows Server 2003). However, for urgent changes like a password change, these delay parameters are ignored and a urgent notification is sent instead.

However, even though the replication partner DC may request changes immediately in response to such a notification, note that the changes are still replicated in a single replication stream (i.e., there is no out-of-band stream by which the password changes are replicated ahead of other pending changes made prior to the password change). In other words, only the notification sent to partner DCs is urgent, but the subsequent pulling of changes by the partner DC will pull all changes from the source DC up to that point in time, and not selectively for the change for which the urgent notification was sent.

When an administrator (or a delegated user) unlocks an account, manually sets password expiration on a user account (by selecting the User Must Change Password At Next Logon check box), or resets the password on an account, these attributes are immediately replicated to the primary domain controller (PDC emulator) and then urgently replicated to other DCs in that site. By default, urgent replication does not occur across site boundaries, therefore it is highly recommended that administrators make manual password changes and account resets on a DC that resides in that user's local site.

Single User Object "On-Demand" Replication

In Windows 2000 pre-SP4 (812499) there is a situation where when an administrator resets and immediately expires a users password on a DC in site A (so user is given new password but forced to change it at first logon), and when the user logs on with that new password in site B, the logon will succeed (due to BDC/PDC chaining during authentication), but the subsequent password change fails because DCs in site B don't have the reset password (because of replication latency).

The change in SP4 (QFE 812499) for Windows 2000 helps the problem by implementing a on-demand replication scheme which works as follows: on a BDC, when an authentication succeeds due to PDC authentication chaining, an asynchronous request is made to the PDC to replicate down one single object (the user object whose authentication just succeeded due to PDC chaining). The idea is that the PDC has the most up-to-date password and the BDC should replicate it down when it has information that the PDC has a more up-to-date version.

Note: 812499 is not a published article at the publication of the white paper; however, it is available for free from Product Support Services.

The following events are not urgent replications in Windows 2000 domains:

·         Changing the account lockout policy

·         Changing the domain password policy

·         Changing the password on a machine account

Interdomain trust passwords (trusts between domain A and B).

For additional information on urgent and immediate replication, please look at the following article:

232690 "Urgent Replication Triggers in Windows 2000"

Otto: Regarding N-2. Without the lockout event, how will we be notified that the password is invalid or that an old one needs to be updated?

Joe: The application will fail. Whatever process is sending the wrong credential, whether it's a user logging on — of course, they're going to be denied access, and they'll either not be able to log on or they won't get access to the resource they're trying to hit. If it's an application sending wrong credentials, then that application will fail.

Mike: Prior to the N-2 behavior, the focus was on the account lockout itself. In other words, we're getting account lockout. It wasn't that their application was failing. Now you'll see that your application is failing, and so it takes the onus off the lockout itself, and then you can find your applications that are failing out your environment. Your users, when something fails, are not able to do something, or their failure will be something like a user may call up and say, "I keep trying my password, and it doesn't work." You may be trying the old password and not the new one.

Joe: It's going to be a matter of the application is failing, and then they need to work with the vendor of that application to find out why it's not refreshing or updating its password cache, and why it's not handling access denied error codes properly. Hopefully they will get some type of event or something to occur on screen to say this application has failed. And they would be able to track it down that way.

Otherwise, they will have to look for specific events or troubleshoot application failures in other ways. Then at that point, if there are no specific events, it may be necessary to enable auditing and Netlogon logging just to verify that yes, the application is failing because of bad passwords.

Otto: Can you provide some more information regarding account lockout from disconnected Terminal Server sessions?

Mike: Usually what happens is somebody is running a process or changes their password and they have a disconnected session. You have to try very hard to reproduce the problem. Joe and I sat for a good day trying to reproduce the behavior. It's usually that there's some session or some application running under that context that is trying to access a resource that is causing that lockout. We couldn't reproduce it, but we found several cases where that was the case, where there was a Terminal Server session out there that was a disconnected session, with an old password.

Otto: Aren't all scheduled tasks statically configured with user ID and password?

Joe: The answer is yes. That's one of the things that you would need to assess as well. Any services or scheduled tasks that have a static password, that account is set, or that account password has changed. If that scheduled task needs access to specific network resources so that it requires authentication, then yes, it would be static.

Otto: Fact or myth: enabling auditing eats up system resources and unacceptably degrades performance on the server. Is that a fact or a myth?

Mike: Maybe. It depends on what you set up as you audit policy, what you're auditing, and the types of access to it.

Obviously, if you're auditing system objects inside the OS, many of those are touched on a massive basis. Yes, you can bring a server to a crawl with auditing. For the most part, depending on what you're auditing, you can slow down a server, but with the type of auditing that we're discussing in this, I've never seen a perceptible difference. I've never had a customer complain to me about a perceptible difference in performance by enabling this auditing on the client and the server. I've never had anybody complain. Joe, have you had anybody complain?

Joe: No, I haven't. Actually auditing is enabled on Windows Server 2003, so I think it's just going to depend on the specific hardware that the machines are running. But in most cases it shouldn't be that big of a performance hit.

Mike: One key factor is in a good audit policy is you should know what you're looking for and know the events. I'd recommend Inside Windows NT, which is a good book, and it has a good section on auditing.

I'll always believe in planning your auditing out before you actually implement it live, so you don't get too much information, and so that is of value.

Otto: Do we know if there is any public information on when SP4 is coming out?

Mike: When it's ready. At this point it's in beta, so when it's ready and it has gone through all of its iterations and testing and things that we do to make sure it's ready for prime time, it will be released.

Otto: Great. Is there any easy way to change local administrator passwords on all clients in the domain?

Mike: We'll address that after the session.

Otto: I'll mark that one for offline follow up then.

Follow-up answer: See the follow-up answer above on Cusrmgr.exe in the resource kit.

We currently have our lockout account security settings implemented at the domain level, Windows NT 4.0. But we also have Windows 2000 servers as members in that Windows NT 4.0 domain. How can I set up local account policies and settings on all member servers by deployment in a mass task, instead of doing each individually?

Joe: I don't know if there's any way to do that, other than individually.

Mike: Either that or a security template. Apply a security template, because your Group Policy is not going to apply those. But your regular domain policy will apply — your domain settings. Overall, your local settings aren't going to apply like that. You'll probably have to do it in a Group Policy template or .inf file.

Otto: Does the enabling of Netlogon logging through the nltest /dbflag switch require a reboot or restart of Netlogon?

Joe: In my personal experience I've seen that it does not require a restart, but it does require a restart of the Netlogon service. It's important. In most situations, you would want to set that back to zero, after the troubleshooting has ended.

Otto: Great. With that, we have answered all the questions that we are able to get to in the live session. I want to thank Joe and Mike for coming out and giving us a great presentation, and I want to thank our audience for coming out and giving us a listen. I hope that the information was useful to you.

As always, you can e-mail us at the email address mention in the website Contact Us page or post back a comment below or post a topic in the support forum here; if you have any suggestions for future topics or comments on the WebCast program as a whole.

I hope everyone will have the opportunity to tune in again soon. Thank you, and have a great day.


No TrackBacks

TrackBack URL: http://www.skar.us/site/mt-tb.cgi/3208

Leave a comment


Author Bio          ★★★★★

Author Name:         ebhakt
Author Location:    India
Author Rank:          Writer
Author Status:        
The Green leave stands!!



  • eBooks
  • Games
  • Softwares
  • Tools
  • Tweaks
  • Wallpapers
  • Warez
  • Games
  • Tools
  • Wallpapers
    System Administration
  • dll Center
  • Scripts
  • Tools
  • .extensions database
  • Write-up
  • Download Database
  • Jobs
  • Lists
  • Polls
  • Glossary

01000011 01110010 01100001 01100011 01101011 01111010 01101000 01100001 01100011 01101011