One of the things I hope to do with this blog is post tidbits of information that I come up with after resolving an IT-related problem. Things that were not immediately obvious when searching around the internet to research the issue. Hopefully if other people run into the same problem, they will be able to benefit from these posts. Of course, all of the standard disclaimers apply – this worked in the environment I was in. Your environment might be different. Always backup first, your mileage may vary, etc 🙂
Over the weekend, I was asked to assist with a problem. About 500 user accounts had been accidentally deleted from an organization’s Active Directory. Their AD domain is a child domain of a larger forest, and the deletion had come from a user in the domain above them.
The network in question was running Windows 2008 R2 for some of their domain controllers, but several were still Windows 2003, so the functional level was still at 2003. This means that the Active Directory recycle bin feature was not available. Given the size of the deletion, it was necessary to perform a restore from backup.
Once we had done the tape-based restore, blocked incoming sync traffic, and marked the objects on the restored domain controller as authoritative, we attempted to use repadmin to push the changes out from the restored DC with:
repadmin /syncall (dc name) /e /A /P /d /q
and monitor the replication with
repadmin /showreps (neighbor dc name)
But we kept receiving an error message from /showreps stating that:
Last attempt @ YYYY-MM-DD HH:MM:SS failed, result 8418 (0x20e2): "The replication operation failed because of a schema mismatch between the servers involved."
Upon investigation, it turned out that the admins in the parent domain had issued a schema update related to a custom AD attribute on the same day as the deletion took place, meaning that the NTDS backup from the night before did not contain that schema update, but did contain the user accounts and group membership that needed to be restored.
The solution turned out to be ensuring that the objects in the restored DC were authoritative and then enabling incoming replication traffic on the DC:
repadmin /options (ServerName) -DISABLE_INBOUND_REPL
This allowed the schema update to complete, which enabled replication between the restored DC and the rest of the domain. After this was done, the /syncall replication performed normally. We then performed a second push to ensure group memberships were updated. When doing an authoritative restore it is possible that group membership lists will be pushed before user accounts are created, so when the group is replicated the users that don’t yet exist won’t be in the replicated group. Pushing the groups a second time (after all of the user accounts are replicated) resolves this problem.
All in all, it took a few hours, but the problem was resolved without any major disruption.