Hello World…
Just want to share an interesting experience of mine on this event ID 16650, which get me laughing till now. This “experience” happened around 2 years ago… the story begins…
I got a call from some senior management which we work together before to assist on a case which has been unresolved for 1-2 months; I took the call and try to understand what is happening on the issue over the phone with his lead engineer (Mr. Macho). They are hitting error 16650 when trying to create any Active Directory object. With some basic information capture, I did some research on the net before going to meet their team for details discussion and action plan.
Surprise me, when getting into their meeting room, there is some architect, project manager senior manager, senior engineer is discussing the issue. Once there, the senior manager introduce me to the team and I did some clarification, detail understanding, like what happen, what has been done, what is the objective, etc… from the discussion understand from the team that before me, they already engaged several Wintel SME to assist but still failed to get it resolved. Basically, what they try is to create a test AD environment by backup and restore one of the domain controller, the restore is successful but failed to create object. After 30 mins of listening the “sound like very complex” technical story from Mr. Macho, I was requested to be on-site to the data center to look into the issue.
While packing up my stuff, a “Macho” question pop up from Mr. Macho to me like below:
Me: OK, I’m ready to go.
Mr. Macho: OK, just want to confirm, you are not struggling your brain, try to understand what I explain to you just now, right?
Me: (I was stunned by the question and speechless; don’t really expect such a question)
Me: No No.. I think it’s fine and I understand what you have explained + (smile while looking at the senior manager which earlier asked me to assist on this issue)
Senior Manager: (To make me feel better, he explain to Mr. Macho about my past experience ……)
Once in the datacenter, Mr. Macho showed me where the problem is.. and prove that he already remove all other domain controller except the being restored from the “Active Directory User and Computer” console. He is like show me the problem in 5 mins and left myself to troubleshoot with no hope of getting it resolved. Just like he want to fulfill his manager request to let me try to troubleshoot the issue, which I’m getting more frustrated, anyway keep cool… J
When looking at the ADUC.msc, indeed all DC has been removed except the restored DC, no more replication link on the “AD Site and Services” console, etc… to double confirm that he properly clean up the AD as claim by Mr. Macho , I did a detail check using NTDSutil tools.
For those who do not know what is NTDSutil, you can refer to http://technet.microsoft.com/en-us/library/cc753343%28WS.10%29.aspx for more details.
What I found from NTDSutil, the other domain controller information is still listed, which mean the other DC information is not cleanup from metadata yet. Great finding, upon verification on the DC details, I use metadata cleanup command from NTDSutil to get it cleaned.
Viola….. once cleaned up, I can create any Active Directory object that I want, user, group, etc…. Informed Mr.Macho that the problem seems resolved, with unbelievable eyesight, he try to create user object by himself and yes, it is resolved. He looked at me speechless and in return I gave my “Are you now struggling your brain on what I did to get it resolved?” eyesight… (That is the best moment I have after so many years in troubleshooting) Yeah….!
Lesson learned, when we restore RID Master from backup, we have to ensure the replication link between DC is working fine, which mean the RID Master must able to contact all other DC, this is to check that there is no other RID Master exists in the domain to avoid conflict. Of course if the domain only have single domain controller then this will not be an issue. In above scenario, because they are trying to create an isolated test environment, thus there is no network connection to all other existing domain controller, thus they must properly cleanup the Active Directory using NTDSutil metadata cleanup command and yes I meant properly clean up not just delete the computer object from ADUC like what Mr. Macho did. Please refer to http://support.microsoft.com/kb/216498 or http://www.petri.co.il/delete_failed_dcs_from_ad.htm on how to perform cleanup.
You can refer to http://support.microsoft.com/kb/839879 “Event ID 16650: The account-identifier allocator failed to initialize in Windows 2000 and in Windows Server 2003” on the issue mentioned here.
P/S: I wrote a summary report email to all relevant party including several senior managers with “Time taken to resolve = < 1 hour” (whereby Mr. Macho “struggling his brain out” on this issue for 1-2 months *Devil Devil* :P Mr. Macho, if you manage to read this blog, no hard feeling OK? Just want to register my experience on the internet. :)
No comments:
Post a Comment