Nobody Has Tested Restoring Active Directory

Last October’s disaster recovery test restored a file server in two hours and a SQL database in three. The steering committee signed off on the recovery time objective. Nobody asked about the domain controllers, because the domain controllers stayed up. That is how a disaster recovery test passes without testing the foundation.

When the primary site actually goes down, the first failure is rarely the storage array. The backup administrator opens the console to initiate a restore and discovers their privileged account depends on the unavailable directory. They try the local break-glass account, but the password is in the enterprise vault, which requires SAML authentication, which requires Active Directory. The runbook detailing how to restore the environment is a PDF stored in SharePoint, which requires a valid session token generated by the dead domain. The instructions for the recovery are locked behind the door they are supposed to open.

Infrastructure teams treat domain controllers like standard virtual machines until the exact moment a restore triggers a USN rollback. Restoring a single VM from a snapshot is an automated task. Restoring an Active Directory forest is a manual orchestration of system state recoveries, metadata cleanup, and command-line utilities. If you roll one node back to last night, its update sequence numbers are out of sync with the rest of the forest. The directory isolates the restored node and stops replication.

A forest recovery requires seizing FSMO roles, cleaning up metadata for dead nodes, and performing authoritative restores of SYSVOL. If SYSVOL comes back empty because the authoritative restore step was skipped, Group Policy Objects do not apply. Users can log in and do nothing. The forest is up, but the environment is useless.

The recovery time objective in the slide deck is usually a fiction, and not because the engineers are slow. Part of a forest recovery is waiting. Resetting the krbtgt account requires two resets with a gap of at least the maximum ticket lifetime, which is ten hours by default. The first reset invalidates tickets signed with the old key. The second reset invalidates anything issued in the gap, catching a golden ticket an attacker might have minted. You cannot staff your way out of that wait. You can add people to the bridge, but you cannot add hours to the clock. Someone wrote a four-hour RTO before anyone opened the procedure.

Then there is the window problem. A usable backup has to be recent enough to be restorable and old enough to be clean. The tombstone lifetime sets the outer bound, typically 180 days. Backups older than that are not a recovery source; restoring them reintroduces deleted objects as lingering objects, re-granting access to terminated employees. The compromise date sets the inner bound. Machine account password age adds another constraint, as they rotate every 30 days. Sometimes the intersection of a clean backup and a valid restore window is empty.

Teams often point to Entra ID as the backup. It is a sync target holding a subset of attributes for a subset of objects. It will not give you SYSVOL, Group Policy Objects, forest trusts, the schema, or service account passwords. If the tenant is federated through AD FS and the domain is the incident, a cloud-only emergency access account is the only reliable way in. Most organizations have one. It is often a shared account whose password sits in the vault whose console is domain-joined.

To survive this, the artifacts must exist outside the domain. The minimum set is small. A printed runbook with the bridge dial-in on the first page, stored somewhere that does not require electricity from the building. A Directory Services Restore Mode password stored offline in a sealed envelope, knowing it was likely set at promotion time a decade ago. A local administrator account on the backup server, the hypervisor, and the out-of-band management network. Those passwords cannot be in the vault, cannot be in AD, and must be known to more than one person.

Week two of the recovery brings the appliance inventory. Badge readers, the SAN management interface, the UPS network card, the HVAC controller, the conference room booking panels. Every one of them has an LDAP bind account or a domain-joined service account. Every one generates a support ticket when its machine account password fails to authenticate against the newly restored domain.

Most organizations refuse to rehearse the full forest recovery because it requires dedicated, isolated hardware. The test lab they would use is a member of the domain they are pretending is gone. If a restored controller connects to the wrong network, a surviving or rogue domain controller will replicate the corrupted state right back into it.

A proper rehearsal requires an isolated segment. A hypervisor host that is not on the domain. A virtual switch with no uplink. A jump box with local accounts and the Active Directory tools copied locally. The test should not start by opening the backup console. It should start by retrieving the physical runbook, obtaining the offline emergency access, and building the isolated environment.

Restoring a domain controller is a Tuesday. Restoring a forest is a quarter. The incident ends when the first domain controller authenticates a user, but the recovery runs for weeks after that, rejoining member servers, resetting machine accounts, and rebuilding trusts. A disaster recovery plan that assumes the underlying identity infrastructure is already functioning in the secondary site is a business continuity plan for the applications. Until a team restores a single domain controller into an isolated network and authenticates a test user from a machine that knows only that controller, the plan is just a document describing a recovery that has not happened.