Hi there,

Need some help with a domain controller that appears not to be replicating. We have three DCs in our main office and one in our satellite office. The remote DC does not appear to be talking to our site and even though I can ping, access via RDP and other methods, the system appears to be unable to replicate.  Here is the main info from a DCdiag report run on the box:

                  

Doing initial required tests


   Testing server: Burlingame\LEDC01

      Starting test: Connectivity

         * Active Directory LDAP Services Check
         Determining IP4 connectivity 
         Failure Analysis: LEDC01 ... OK.
         * Active Directory RPC Services Check
         ......................... LEDC01 passed test Connectivity


   Testing server: Burlingame\LEDC03

      Starting test: Connectivity

         * Active Directory LDAP Services Check
         Got error while checking LDAP and RPC connectivity. Please check your

         firewall settings.

         ......................... LEDC03 failed test Connectivity


   Testing server: Bellevue\LESEADC01

      Starting test: Connectivity

         * Active Directory LDAP Services Check
         Determining IP4 connectivity 
         Failure Analysis: LESEADC01 ... OK.
         * Active Directory RPC Services Check
         ......................... LESEADC01 passed test Connectivity


   Testing server: Burlingame\LE-DC-01

      Starting test: Connectivity

         * Active Directory LDAP Services Check
         Determining IP4 connectivity 
         Failure Analysis: LE-DC-01 ... OK.
         * Active Directory RPC Services Check
         ......................... LE-DC-01 passed test Connectivity


   Testing server: Burlingame\LE-DC-02

      Starting test: Connectivity

         * Active Directory LDAP Services Check
         Determining IP4 connectivity 
         Failure Analysis: LE-DC-02 ... OK.
         * Active Directory RPC Services Check
         ......................... LE-DC-02 passed test Connectivity



Doing primary tests


   Testing server: Burlingame\LEDC01

      Starting test: Advertising

         The DC LEDC01 is advertising itself as a DC and having a DS.
         The DC LEDC01 is advertising as an LDAP server
         The DC LEDC01 is advertising as having a writeable directory
         The DC LEDC01 is advertising as a Key Distribution Center
         The DC LEDC01 is advertising as a time server
         The DS LEDC01 is advertising as a GC.
         ......................... LEDC01 passed test Advertising

      Starting test: CheckSecurityError

         * Dr Auth:  Beginning security errors check!
         Found KDC LEDC01 for domain ourdomain.com in site Burlingame
         Checking machine account for DC LEDC01 on DC LEDC01.
         * SPN found :LDAP/ledc01.ourdomain.com/ourdomain.com
         * SPN found :LDAP/ledc01.ourdomain.com
         * SPN found :LDAP/LEDC01
         * SPN found :LDAP/ledc01.ourdomain.com/ourdomain
         * SPN found :LDAP/ff5f9247-1d4f-4d36-aace-21bf25e5ec10._msdcs.ourdomain.com
         * SPN found :E3514235-4B06-11D1-AB04-00C04FC2DCD2/ff5f9247-1d4f-4d36-aace-21bf25e5ec10/ourdomain.com
         * SPN found :HOST/ledc01.ourdomain.com/ourdomain.com
         * SPN found :HOST/ledc01.ourdomain.com
         * SPN found :HOST/LEDC01
         * SPN found :HOST/ledc01.ourdomain.com/ourdomain
         * SPN found :GC/ledc01.ourdomain.com/ourdomain.com
         [LEDC01] No security related replication errors were found on this DC!

          To target the connection to a specific source DC use

         /ReplSource:<DC>.

         ......................... LEDC01 passed test CheckSecurityError

      Starting test: CutoffServers

         * Configuration Topology Aliveness Check
         * Analyzing the alive system replication topology for DC=DomainDnsZones,DC=ourdomain,DC=com.
         * Performing upstream (of target) analysis.
         * Performing downstream (of target) analysis.
         * Analyzing the alive system replication topology for DC=ForestDnsZones,DC=ourdomain,DC=com.
         * Performing upstream (of target) analysis.
         * Performing downstream (of target) analysis.
         * Analyzing the alive system replication topology for CN=Schema,CN=Configuration,DC=ourdomain,DC=com.
         * Performing upstream (of target) analysis.
         * Performing downstream (of target) analysis.
         * Analyzing the alive system replication topology for CN=Configuration,DC=ourdomain,DC=com.
         * Performing upstream (of target) analysis.
         * Performing downstream (of target) analysis.
         * Analyzing the alive system replication topology for DC=ourdomain,DC=com.
         * Performing upstream (of target) analysis.
         * Performing downstream (of target) analysis.
         ......................... LEDC01 passed test CutoffServers

      Starting test: FrsEvent

         * The File Replication Service Event log test 
         ......................... LEDC01 passed test FrsEvent

      Starting test: DFSREvent

         The DFS Replication Event Log. 
         Skip the test because the server is running FRS.

         ......................... LEDC01 passed test DFSREvent

      Starting test: SysVolCheck

         * The File Replication Service SYSVOL ready test 
         File Replication Service's SYSVOL is ready 
         ......................... LEDC01 passed test SysVolCheck

      Starting test: FrsSysVol

         * The File Replication Service SYSVOL ready test 
         File Replication Service's SYSVOL is ready 
         ......................... LEDC01 passed test FrsSysVol

      Starting test: KccEvent

         * The KCC Event log test
         Found no KCC errors in "Directory Service" Event log in the last 15 minutes.
         ......................... LEDC01 passed test KccEvent

      Starting test: KnowsOfRoleHolders

         Role Schema Owner = CN=NTDS Settings,CN=LEDC01,CN=Servers,CN=Burlingame,CN=Sites,CN=Configuration,DC=ourdomain,DC=com
         Role Domain Owner = CN=NTDS Settings,CN=LEDC01,CN=Servers,CN=Burlingame,CN=Sites,CN=Configuration,DC=ourdomain,DC=com
         Role PDC Owner = CN=NTDS Settings,CN=LEDC01,CN=Servers,CN=Burlingame,CN=Sites,CN=Configuration,DC=ourdomain,DC=com
         Role Rid Owner = CN=NTDS Settings,CN=LEDC01,CN=Servers,CN=Burlingame,CN=Sites,CN=Configuration,DC=ourdomain,DC=com
         Role Infrastructure Update Owner = CN=NTDS Settings,CN=LE-DC-02,CN=Servers,CN=Burlingame,CN=Sites,CN=Configuration,DC=ourdomain,DC=com
         ......................... LEDC01 passed test KnowsOfRoleHolders

      Starting test: MachineAccount

         Checking machine account for DC LEDC01 on DC LEDC01.
         * SPN found :LDAP/ledc01.ourdomain.com/ourdomain.com
         * SPN found :LDAP/ledc01.ourdomain.com
         * SPN found :LDAP/LEDC01
         * SPN found :LDAP/ledc01.ourdomain.com/ourdomain
         * SPN found :LDAP/ff5f9247-1d4f-4d36-aace-21bf25e5ec10._msdcs.ourdomain.com
         * SPN found :E3514235-4B06-11D1-AB04-00C04FC2DCD2/ff5f9247-1d4f-4d36-aace-21bf25e5ec10/ourdomain.com
         * SPN found :HOST/ledc01.ourdomain.com/ourdomain.com
         * SPN found :HOST/ledc01.ourdomain.com
         * SPN found :HOST/LEDC01
         * SPN found :HOST/ledc01.ourdomain.com/ourdomain
         * SPN found :GC/ledc01.ourdomain.com/ourdomain.com
         ......................... LEDC01 passed test MachineAccount

      Starting test: NCSecDesc

         * Security Permissions check for all NC's on DC LEDC01.
         The forest is not ready for RODC. Will skip checking ERODC ACEs.
         * Security Permissions Check for

           DC=DomainDnsZones,DC=ourdomain,DC=com
            (NDNC,Version 3)
         * Security Permissions Check for

           DC=ForestDnsZones,DC=ourdomain,DC=com
            (NDNC,Version 3)
         * Security Permissions Check for

           CN=Schema,CN=Configuration,DC=ourdomain,DC=com
            (Schema,Version 3)
         * Security Permissions Check for

           CN=Configuration,DC=ourdomain,DC=com
            (Configuration,Version 3)
         * Security Permissions Check for

           DC=ourdomain,DC=com
            (Domain,Version 3)
         ......................... LEDC01 passed test NCSecDesc

      Starting test: NetLogons

         * Network Logons Privileges Check
         Verified share \\LEDC01\netlogon
         Verified share \\LEDC01\sysvol
         ......................... LEDC01 passed test NetLogons

      Starting test: ObjectsReplicated

         LEDC01 is in domain DC=ourdomain,DC=com
         Checking for CN=LEDC01,OU=Domain Controllers,DC=ourdomain,DC=com in domain DC=ourdomain,DC=com on 4 servers
            Authoritative attribute lastLogonTimestamp on LE-DC-02 (writeable)
               usnLocalChange = 644484
               LastOriginatingDsa = LEDC01
               usnOriginatingChange = 31585308
               timeLastOriginatingChange = 2015-06-03 06:49:59
               VersionLastOriginatingChange = 255
            Out-of-date attribute lastLogonTimestamp on LESEADC01 (writeable)
               usnLocalChange = 724086
               LastOriginatingDsa = LEDC01
               usnOriginatingChange = 30900188
               timeLastOriginatingChange = 2015-04-08 07:29:09
               VersionLastOriginatingChange = 251
            Authoritative attribute pwdLastSet on LEDC01 (writeable)
               usnLocalChange = 31359243
               LastOriginatingDsa = LE-DC-01
               usnOriginatingChange = 238582
               timeLastOriginatingChange = 2015-05-11 10:18:51
               VersionLastOriginatingChange = 127
            Out-of-date attribute pwdLastSet on LESEADC01 (writeable)
               usnLocalChange = 726052
               LastOriginatingDsa = LEDC01
               usnOriginatingChange = 30949551
               timeLastOriginatingChange = 2015-04-10 20:50:09
               VersionLastOriginatingChange = 126
         Checking for CN=NTDS Settings,CN=LEDC01,CN=Servers,CN=Burlingame,CN=Sites,CN=Configuration,DC=ourdomain,DC=com in domain CN=Configuration,DC=ourdomain,DC=com on 4 servers
            Object is up-to-date on all servers.
         ......................... LEDC01 failed test ObjectsReplicated

      Starting test: OutboundSecureChannels

         * The Outbound Secure Channels test
         ** Did not run Outbound Secure Channels test because /testdomain: was

         not entered

         ......................... LEDC01 passed test OutboundSecureChannels

      Starting test: Replications

         * Replications Check
         DC=DomainDnsZones,DC=ourdomain,DC=com has 10 cursors.
         DC=ForestDnsZones,DC=ourdomain,DC=com has 10 cursors.
         CN=Schema,CN=Configuration,DC=ourdomain,DC=com has 12 cursors.
         CN=Configuration,DC=ourdomain,DC=com has 12 cursors.
         DC=ourdomain,DC=com has 12 cursors.
         * Replication Latency Check
            DC=DomainDnsZones,DC=ourdomain,DC=com
               Latency information for 5 entries in the vector were ignored.
                  5 were retired Invocations.  0 were either: read-only replicas and are not verifiably latent, or dc's no longer replicating this nc.  0 had no latency information (Win2K DC).  
            DC=ForestDnsZones,DC=ourdomain,DC=com
               Latency information for 5 entries in the vector were ignored.
                  5 were retired Invocations.  0 were either: read-only replicas and are not verifiably latent, or dc's no longer replicating this nc.  0 had no latency information (Win2K DC).  
            CN=Schema,CN=Configuration,DC=ourdomain,DC=com
               Latency information for 7 entries in the vector were ignored.
                  7 were retired Invocations.  0 were either: read-only replicas and are not verifiably latent, or dc's no longer replicating this nc.  0 had no latency information (Win2K DC).  
            CN=Configuration,DC=ourdomain,DC=com
               Latency information for 7 entries in the vector were ignored.
                  7 were retired Invocations.  0 were either: read-only replicas and are not verifiably latent, or dc's no longer replicating this nc.  0 had no latency information (Win2K DC).  
            DC=ourdomain,DC=com
               Latency information for 7 entries in the vector were ignored.
                  7 were retired Invocations.  0 were either: read-only replicas and are not verifiably latent, or dc's no longer replicating this nc.  0 had no latency information (Win2K DC).  
         ......................... LEDC01 passed test Replications

      Starting test: RidManager

         ridManagerReference = CN=RID Manager$,CN=System,DC=ourdomain,DC=com
         * Available RID Pool for the Domain is 8109 to 1073741823
         fSMORoleOwner = CN=NTDS Settings,CN=LEDC01,CN=Servers,CN=Burlingame,CN=Sites,CN=Configuration,DC=ourdomain,DC=com
         * ledc01.ourdomain.com is the RID Master
         * DsBind with RID Master was successful
         rIDSetReferences = CN=RID Set,CN=LEDC01,OU=Domain Controllers,DC=ourdomain,DC=com
         * rIDAllocationPool is 6109 to 6608
         * rIDPreviousAllocationPool is 4609 to 5108
         * rIDNextRID: 5017
         * Warning :There is less than 19% available RIDs in the current pool
         ......................... LEDC01 passed test RidManager

      Starting test: Services

         * Checking Service: EventSystem
         * Checking Service: RpcSs
            Invalid service type: RpcSs on LEDC01, current value

            WIN32_OWN_PROCESS, expected value WIN32_SHARE_PROCESS

         * Checking Service: DnsCache
         * Checking Service: NtFrs
         * Checking Service: IsmServ
         * Checking Service: kdc
         * Checking Service: SamSs
         * Checking Service: LanmanServer
         * Checking Service: LanmanWorkstation
         * Checking Service: w32time
         * Checking Service: NETLOGON
         ......................... LEDC01 failed test Services

 Starting test: SystemLog

         * The System Event log test
         An error event occurred.  EventID: 0xC000001B

            Time Generated: 06/04/2015   07:53:13

            Event String:

            While processing a TGS request for the target

            server krbtgt/ourdomain.com, the account

            J8CPNS1$@ourdomain.com did not have a suitable

            key for generating a Kerberos ticket (the missing

            key has an ID of 8). The requested etypes were

            18. The accounts available etypes were

            23  -133  -128  3  1. 

         An error event occurred.  EventID: 0x00000457

            Time Generated: 06/04/2015   07:59:47

            (Event String (event log = System) could not be retrieved, error

            0x3afc)

         An error event occurred.  EventID: 0x00000457

            Time Generated: 06/04/2015   07:59:48

            (Event String (event log = System) could not be retrieved, error

            0x3afc)

         An error event occurred.  EventID: 0x00000457

            Time Generated: 06/04/2015   07:59:48

            (Event String (event log = System) could not be retrieved, error

            0x3afc)

         An error event occurred.  EventID: 0x00000457

            Time Generated: 06/04/2015   07:59:49

            (Event String (event log = System) could not be retrieved, error

            0x3afc)

Starting test: CheckSecurityError

         * Dr Auth:  Beginning security errors check!
         Found KDC LESEADC01 for domain ourdomain.com in site Bellevue
         Checking machine account for DC LESEADC01 on DC LESEADC01.
         * SPN found :LDAP/LESEADC01.ourdomain.com/ourdomain.com
         * SPN found :LDAP/LESEADC01.ourdomain.com
         * SPN found :LDAP/LESEADC01
         * SPN found :LDAP/LESEADC01.ourdomain.com/ourdomain
         * SPN found :LDAP/171945ed-8fd5-4104-a9d5-9c109e11d3af._msdcs.ourdomain.com
         * SPN found :E3514235-4B06-11D1-AB04-00C04FC2DCD2/171945ed-8fd5-4104-a9d5-9c109e11d3af/ourdomain.com
         * SPN found :HOST/LESEADC01.ourdomain.com/ourdomain.com
         * SPN found :HOST/LESEADC01.ourdomain.com
         * SPN found :HOST/LESEADC01
         * SPN found :HOST/LESEADC01.ourdomain.com/ourdomain
         * SPN found :GC/LESEADC01.ourdomain.com/ourdomain.com
         Source DC LE-DC-01 has possible security error (1396).  Diagnosing...

               Found KDC LEDC01 for domain ourdomain.com in site Burlingame
               Checking time skew between servers:
                LE-DC-01
                LEDC01
                LESEADC01
               Getting time for \\LE-DC-01.ourdomain.com
               Getting time for \\ledc01.ourdomain.com
               Time is 1433432666 on \\ledc01.ourdomain.com
               Getting time for \\LESEADC01.ourdomain.com
               Time is 1433432667 on \\LESEADC01.ourdomain.com
               Time skew error between client and 1 DCs!  ERROR_ACCESS_DENIED

               or down machine received by:

                LE-DC-01
         Ignoring DC LEDC03 in the convergence test of object

         CN=LESEADC01,OU=Domain Controllers,DC=ourdomain,DC=com, because we

         cannot connect!

         Checking for CN=LESEADC01,OU=Domain Controllers,DC=ourdomain,DC=com in domain DC=ourdomain,DC=com on 4 servers
            Authoritative attribute pwdLastSet on LE-DC-01 (writeable)
               usnLocalChange = 570785
               LastOriginatingDsa = LEDC01
               usnOriginatingChange = 31598213
               timeLastOriginatingChange = 2015-06-04 08:16:45
               VersionLastOriginatingChange = 27
            Out-of-date attribute pwdLastSet on LESEADC01 (writeable)
               usnLocalChange = 714378
               LastOriginatingDsa = LEDC01
               usnOriginatingChange = 30520642
               timeLastOriginatingChange = 2015-03-24 06:29:17
               VersionLastOriginatingChange = 26
         Unable to verify the convergence of this machine account

         (CN=LESEADC01,OU=Domain Controllers,DC=ourdomain,DC=com) on these

         DC's (DC=ourdomain,DC=com,).  Does the machine account password

         need resetting?

         ......................... LESEADC01 failed test CheckSecurityError

      Starting test: CutoffServers

         * Configuration Topology Aliveness Check
         * Analyzing the alive system replication topology for DC=DomainDnsZones,DC=ourdomain,DC=com.
         * Performing upstream (of target) analysis.
         * Performing downstream (of target) analysis.
         * Analyzing the alive system replication topology for DC=ForestDnsZones,DC=ourdomain,DC=com.
         * Performing upstream (of target) analysis.
         * Performing downstream (of target) analysis.
         * Analyzing the alive system replication topology for CN=Schema,CN=Configuration,DC=ourdomain,DC=com.
         * Performing upstream (of target) analysis.
         * Performing downstream (of target) analysis.
         * Analyzing the alive system replication topology for CN=Configuration,DC=ourdomain,DC=com.
         * Performing upstream (of target) analysis.
         * Performing downstream (of target) analysis.
         * Analyzing the alive system replication topology for DC=ourdomain,DC=com.
         * Performing upstream (of target) analysis.
         * Performing downstream (of target) analysis.
         ......................... LESEADC01 passed test CutoffServers

      Starting test: FrsEvent

         * The File Replication Service Event log test 
         ......................... LESEADC01 passed test FrsEvent

      Starting test: DFSREvent

         The DFS Replication Event Log. 
         Skip the test because the server is running FRS.

         ......................... LESEADC01 passed test DFSREvent

      Starting test: SysVolCheck

         * The File Replication Service SYSVOL ready test 
         File Replication Service's SYSVOL is ready 
         ......................... LESEADC01 passed test SysVolCheck

      Starting test: FrsSysVol

         * The File Replication Service SYSVOL ready test 
         File Replication Service's SYSVOL is ready 
         ......................... LESEADC01 passed test FrsSysVol

      Starting test: KccEvent

         * The KCC Event log test
         An error event occurred.  EventID: 0xC000066D

            Time Generated: 06/04/2015   08:38:27

            Event String:

            Active Directory Domain Services did not perform an authenticated remote procedure call (RPC) to another directory server because the desired service principal name (SPN) for the destination directory server is not registered on the Key Distribution Center (KDC) domain controller that resolves the SPN. 

  Starting test: FrsEvent

         * The File Replication Service Event log test 
         The event log File Replication Service on server

         LE-DC-01.ourdomain.com could not be queried, error 0x721

         "A security package specific error occurred."

         ......................... LE-DC-01 failed test FrsEvent

      Starting test: DFSREvent

         The DFS Replication Event Log. 
         Skip the test because the server is running FRS.

         ......................... LE-DC-01 passed test DFSREvent

      Starting test: SysVolCheck

         * The File Replication Service SYSVOL ready test 
         [LE-DC-01] An net use or LsaPolicy operation failed with error 1396,

         The target account name is incorrect..

         The registry lookup failed to determine the state of the SYSVOL.  The

         error returned  was 0x574 "The target account name is incorrect.".

         Check the FRS event log to see if the SYSVOL has successfully been

         shared. 
         ......................... LE-DC-01 failed test SysVolCheck

      Starting test: FrsSysVol

         * The File Replication Service SYSVOL ready test 
         The registry lookup failed to determine the state of the SYSVOL.  The

         error returned  was 0x574 "The target account name is incorrect.".

         Check the FRS event log to see if the SYSVOL has successfully been

         shared. 
         ......................... LE-DC-01 failed test FrsSysVol

      Starting test: KccEvent

         * The KCC Event log test
         The event log Directory Service on server LE-DC-01.ourdomain.com

         could not be queried, error 0x721

         "A security package specific error occurred."

         ......................... LE-DC-01 failed test KccEvent

      Starting test: KnowsOfRoleHolders

         Role Schema Owner = CN=NTDS Settings,CN=LEDC01,CN=Servers,CN=Burlingame,CN=Sites,CN=Configuration,DC=ourdomain,DC=com
         Role Domain Owner = CN=NTDS Settings,CN=LEDC01,CN=Servers,CN=Burlingame,CN=Sites,CN=Configuration,DC=ourdomain,DC=com
         Role PDC Owner = CN=NTDS Settings,CN=LEDC01,CN=Servers,CN=Burlingame,CN=Sites,CN=Configuration,DC=ourdomain,DC=com
         Role Rid Owner = CN=NTDS Settings,CN=LEDC01,CN=Servers,CN=Burlingame,CN=Sites,CN=Configuration,DC=ourdomain,DC=com
         Role Infrastructure Update Owner = CN=NTDS Settings,CN=LE-DC-02,CN=Servers,CN=Burlingame,CN=Sites,CN=Configuration,DC=ourdomain,DC=com
         ......................... LE-DC-01 passed test KnowsOfRoleHolders

      Starting test: MachineAccount

         Checking machine account for DC LE-DC-01 on DC LE-DC-01.
         * LE-DC-01 Server Reference is incorrect!  Should be

         CN=LE-DC-01,CN=Computers,DC=ourdomain,DC=com, and is

         CN=LE-DC-01,OU=Domain Controllers,DC=ourdomain,DC=com.

         * LE-DC-01 Server Reference is incorrect

         Could not open pipe with [LE-DC-01]:failed with 1396:

         The target account name is incorrect.

         Could not get NetBIOSDomainName

         Failed can not test for HOST SPN

         Failed can not test for HOST SPN

         ldap_search_sW failed with 2:

         The system cannot find the file specified.

         ......................... LE-DC-01 failed test MachineAccount

      Starting test: NCSecDesc

         * Security Permissions check for all NC's on DC LE-DC-01.
         The forest is not ready for RODC. Will skip checking ERODC ACEs.
         * Security Permissions Check for

           DC=DomainDnsZones,DC=ourdomain,DC=com
            (NDNC,Version 3)
         * Security Permissions Check for

           DC=ForestDnsZones,DC=ourdomain,DC=com
            (NDNC,Version 3)
         * Security Permissions Check for

           CN=Schema,CN=Configuration,DC=ourdomain,DC=com
            (Schema,Version 3)
         * Security Permissions Check for

           CN=Configuration,DC=ourdomain,DC=com
            (Configuration,Version 3)
         * Security Permissions Check for

           DC=ourdomain,DC=com
            (Domain,Version 3)
         ......................... LE-DC-01 passed test NCSecDesc

      Starting test: NetLogons

         * Network Logons Privileges Check
         [LE-DC-01] An net use or LsaPolicy operation failed with error 1396,

         The target account name is incorrect..

         ......................... LE-DC-01 failed test NetLogons

      Starting test: ObjectsReplicated

         LE-DC-01 is in domain DC=ourdomain,DC=com
         Checking for CN=LE-DC-01,CN=Computers,DC=ourdomain,DC=com in domain DC=ourdomain,DC=com on 4 servers
         Failed to read object metadata on LEDC01, error

         Directory object not found.

         Failed to read object metadata on LE-DC-01, error

         Directory object not found.

         Failed to read object metadata on LE-DC-02, error

         Directory object not found.

            Object is up-to-date on all servers.
         Checking for CN=NTDS Settings,CN=LE-DC-01,CN=Servers,CN=Burlingame,CN=Sites,CN=Configuration,DC=ourdomain,DC=com in domain CN=Configuration,DC=ourdomain,DC=com on 4 servers
            Object is up-to-date on all servers.
         ......................... LE-DC-01 passed test ObjectsReplicated

      Starting test: OutboundSecureChannels

         * The Outbound Secure Channels test
         ** Did not run Outbound Secure Channels test because /testdomain: was

         not entered

         ......................... LE-DC-01 passed test OutboundSecureChannels

      Starting test: Replications

         * Replications Check
         DC=DomainDnsZones,DC=ourdomain,DC=com has 10 cursors.
         DC=ForestDnsZones,DC=ourdomain,DC=com has 10 cursors.
         CN=Schema,CN=Configuration,DC=ourdomain,DC=com has 12 cursors.
         CN=Configuration,DC=ourdomain,DC=com has 12 cursors.
         DC=ourdomain,DC=com has 12 cursors.
         * Replication Latency Check
            DC=DomainDnsZones,DC=ourdomain,DC=com
               Latency information for 5 entries in the vector were ignored.
                  5 were retired Invocations.  0 were either: read-only replicas and are not verifiably latent, or dc's no longer replicating this nc.  0 had no latency information (Win2K DC).  
            DC=ForestDnsZones,DC=ourdomain,DC=com
               Latency information for 5 entries in the vector were ignored.
                  5 were retired Invocations.  0 were either: read-only replicas and are not verifiably latent, or dc's no longer replicating this nc.  0 had no latency information (Win2K DC).  
            CN=Schema,CN=Configuration,DC=ourdomain,DC=com
               Latency information for 7 entries in the vector were ignored.
                  7 were retired Invocations.  0 were either: read-only replicas and are not verifiably latent, or dc's no longer replicating this nc.  0 had no latency information (Win2K DC).  
            CN=Configuration,DC=ourdomain,DC=com
               Latency information for 7 entries in the vector were ignored.
                  7 were retired Invocations.  0 were either: read-only replicas and are not verifiably latent, or dc's no longer replicating this nc.  0 had no latency information (Win2K DC).  
            DC=ourdomain,DC=com
               Latency information for 7 entries in the vector were ignored.
                  7 were retired Invocations.  0 were either: read-only replicas and are not verifiably latent, or dc's no longer replicating this nc.  0 had no latency information (Win2K DC).  
         ......................... LE-DC-01 passed test Replications

      Starting test: RidManager

         ridManagerReference = CN=RID Manager$,CN=System,DC=ourdomain,DC=com
         * Available RID Pool for the Domain is 8109 to 1073741823
         fSMORoleOwner = CN=NTDS Settings,CN=LEDC01,CN=Servers,CN=Burlingame,CN=Sites,CN=Configuration,DC=ourdomain,DC=com
         * ledc01.ourdomain.com is the RID Master
         * DsBind with RID Master was successful
         rIDSetReferences = CN=RID Set,CN=LE-DC-01,OU=Domain Controllers,DC=ourdomain,DC=com
         * rIDAllocationPool is 7109 to 7608
         * rIDPreviousAllocationPool is 7109 to 7608
         * rIDNextRID: 7123
         ......................... LE-DC-01 passed test RidManager

      Starting test: Services

         Could not open Remote ipc to [LE-DC-01.ourdomain.com]: error 0x574

         "The target account name is incorrect."

         ......................... LE-DC-01 failed test Services

      Starting test: SystemLog

         * The System Event log test
         The event log System on server LE-DC-01.ourdomain.com could not be

         queried, error 0x721 "A security package specific error occurred."

         ......................... LE-DC-01 failed test SystemLog

      Starting test: Topology

         * Configuration Topology Integrity Check
         * Analyzing the connection topology for DC=DomainDnsZones,DC=ourdomain,DC=com.
         * Performing upstream (of target) analysis.
         * Performing downstream (of target) analysis.
         * Analyzing the connection topology for DC=ForestDnsZones,DC=ourdomain,DC=com.
         * Performing upstream (of target) analysis.
         * Performing downstream (of target) analysis.
         * Analyzing the connection topology for CN=Schema,CN=Configuration,DC=ourdomain,DC=com.
         * Performing upstream (of target) analysis.
         * Performing downstream (of target) analysis.
         * Analyzing the connection topology for CN=Configuration,DC=ourdomain,DC=com.
         * Performing upstream (of target) analysis.
         * Performing downstream (of target) analysis.
         * Analyzing the connection topology for DC=ourdomain,DC=com.
         * Performing upstream (of target) analysis.
         * Performing downstream (of target) analysis.
         ......................... LE-DC-01 passed test Topology

      Starting test: VerifyEnterpriseReferences

         ......................... LE-DC-01 passed test

         VerifyEnterpriseReferences

      Starting test: VerifyReferences

         The system object reference (serverReference)

         CN=LE-DC-01,OU=Domain Controllers,DC=ourdomain,DC=com and backlink

         on

         CN=LE-DC-01,CN=Servers,CN=Burlingame,CN=Sites,CN=Configuration,DC=ourdomain,DC=com

         are correct. 
         The system object reference (serverReferenceBL)

         CN=LE-DC-01,CN=Domain System Volume (SYSVOL share),CN=File Replication Service,CN=System,DC=ourdomain,DC=com

         and backlink on

         CN=NTDS Settings,CN=LE-DC-01,CN=Servers,CN=Burlingame,CN=Sites,CN=Configuration,DC=ourdomain,DC=com

         are correct. 
         The system object reference (frsComputerReferenceBL)

         CN=LE-DC-01,CN=Domain System Volume (SYSVOL share),CN=File Replication Service,CN=System,DC=ourdomain,DC=com

         and backlink on

         CN=LE-DC-01,OU=Domain Controllers,DC=ourdomain,DC=com are correct. 
         ......................... LE-DC-01 passed test VerifyReferences

      Starting test: VerifyReplicas

         ......................... LE-DC-01 passed test VerifyReplicas

   
   Testing server: Burlingame\LE-DC-02

      Starting test: Advertising

         Fatal Error:DsGetDcName (LE-DC-02) call failed, error 5

         The Locator could not find the server.

         ......................... LE-DC-02 failed test Advertising

      Starting test: CheckSecurityError

         * Dr Auth:  Beginning security errors check!
         Found KDC LEDC01 for domain ourdomain.com in site Burlingame
         Checking machine account for DC LE-DC-02 on DC LEDC01.
         * LE-DC-02 Server Reference is incorrect!  Should be

         CN=LE-DC-02,CN=Computers,DC=ourdomain,DC=com, and is

         CN=LE-DC-02,OU=Domain Controllers,DC=ourdomain,DC=com.

         * LE-DC-02 Server Reference is incorrect

         Could not open pipe with [LE-DC-02]:failed with 1396:

         The target account name is incorrect.

         Could not get NetBIOSDomainName

         Failed can not test for HOST SPN

         Failed can not test for HOST SPN

         ldap_search_sW failed with 2:

         The system cannot find the file specified.

         Unable to verify the machine account

         (CN=LE-DC-02,CN=Computers,DC=ourdomain,DC=com) for LE-DC-02 on

         LEDC01.

         [LE-DC-02] No security related replication errors were found on this

         DC!  To target the connection to a specific source DC use

         /ReplSource:<DC>.           

Hi tcstrub,

There's a lot of errors here and based on the dcdiag output alone, I'm not sure which is the root cause versus being a knock-on effect. It's also worth mentioning that the output indicates you have four domain controllers at your main site, not three:

1. LECD01 at Burlingame.
2. LEDC03 at Burlingame.
3. LE-DC-01 at Burlingame.
4. LE-DC-02 at Burlingame.

And then the final domain controller:

1. LESEADC01 at Bellevue.

This report illustrates various issues to do with:

1. Connectivity.
2. Time synchronisation.
3. Computer state (i.e. LE-DC-01 and 02 being referenced in the CN=Computers container).

I'd start with running "dcdiag /c /v" on each of the domain controllers as well as an "ipconfig /all" and post the output up to something like OneDrive for us to take a look at. I'd also look to clean up any domain controller metadata (using "ntdsutil") if any of the above don't exist anymore.

Also remember that while you might be able to ping, RDP, etc. from your workstation or one of the domain controllers to the branch office domain controller, it also needs to be able to (by default) communicate back in the other direction to each of the domain controllers in your main site. If you don't have full physical connectivity because the network isn't fully routable or firewalls block particular forms of connectivity between all domain controllers then you're going to have to look at an option such as disabling automatic site bridging and manually specifying the site topology.

In addition to ping, telnet and tracert are your friends for diagnosing actual connectivity and points of communication breakdown.

In any case, I'd tackle your local issues in the main site before focusing too heavily on the branch office domain controller as it feels like its problems are only going to be a by-product of the faults at your main site.

Cheers,
Lain

Hi Lain,

Thanks so much for your message - I will endeavor to walk through the steps you detail here. I have placed the dcdiag files at the following dropbox location

https://www.dropbox.com/s/jbdwcwrobfqp81y/domain.zip?dl=0

Here is a bit more detail on the situation: this started when I created two new DCs in our main site (le-dc-01 and le-dc-02) to replace the older Windows Server 2003 ledc01. I did not initially realize that there were synchronization errors between leseadc01 and our main site, and have now been trying to determine where the issue began.

As I get additional information, I will post. Thanks again for all your assistance, it is a great help!

Here is a link for the ipconfig info as well

https://www.dropbox.com/sh/uql98kcpmder1vh/AADKTDkuG36EN8Mz2bJ9VVCOa?dl=0

Please remove ::1 from being your DCs DNS server in the DCs IPv6 settings. Also, I would recommend that you adjust the IP settings to what I recommended here: http://www.ahmedmalek.com/web/fr/articles.asp?artid=23

Please also make sure that there are not NAT devices between your DCs and that required AD ports for replication are opened in both directions between your sites.

Let's run a PortQRY to check if there are any closed ports.

PortQry GUI - Run the "Domains & Trusts" option between DCs, or between DCs and any machine (other servers you want to promote, or even from a client machine), that you want to test if there are any blocked AD ports. Post only errors with "NOTLISTENING," 0x00000001, and 0x00000002. You can ignore UDP 389 and UDP 88 messages. If you see TCP 42 errors, that just means WINS is not running on the target server.
       PortQryUI - GUI - Version 2.0 8/2/2004
http://www.microsoft.com/download/en/details.aspx?id=24009 

=

In addition to the dcdiag /c /v suggested by Lain, let's run the following to get some more info on replication. These are helpful AD tools I suggest to keep in your toolbox.

1. Download The Active Directory Replication Status Tool (ADREPLSTATUS):
   http://www.microsoft.com/en-us/download/details.aspx?id=30005
     This tool requires .Net Framework 4. If it's not installed, download and install it:
       Microsoft .NET Framework 4 (Web Installer)
       http://www.microsoft.com/en-us/download/details.aspx?id=17851
 

2. ReplDIAG:  (run it as repldiag > c:\repldiag.txt, then open it as a CSV in Excel choosing comma separated, to be able to clearly read the output).  Explained here:
     Troubleshooting replication with ReplDiag.exe [part 1 of 4], Rob Bolbotowski [MSFT], 13 Oct 2010 12:04 PM
     http://blogs.technet.com/b/robertbo/archive/2010/10/13/troubleshooting-replication-with-repldiag-exe-part-1-of-4.aspx
        ReplDiag Downloadable from:
        http://activedirectoryutils.codeplex.com/releases/view/13664

3. Third Party Utility: Dynamic AD Replication Checker Tool not only checks AD Replication for all domain controllers in your organization but also provides monitoring capabilities. For any non-working Domain Controller you can use the various options available to troubleshoot the issue.
 Dynamic AD Replication Checker Tool Version 2.0 Released
 http://blog.dynamicitkit.com/dynamic-ad-replication-checker-tool-version-2-0-released/
 Download Dynamic AD Replication Checker Tool Version 2.0 (part of "Dynamic Pack")
 http://www.dynamic-spotaction.com/index.html

4. Joe Richards' JoeWare's free GCChk tool will help find lingering objects a little faster and in situations where the Microsoft default tools may lack or or that some consider too complex to figure out:
http://www.joeware.net/freetools/tools/gcchk/index.htm

Hi tcstrub,

Sorry for taking a while to get back to you, however, there's some good advice above. That said, there's a lot of unknowns in which case here's some targeted things I'd be looking at first.

Step 1. Clean up the IP configuration.

Here's a couple of commands I tend to run when I'm configuring IPv4 on the domain controllers:

REM Disable IPv6 transitional protocols.
netsh int 6to4 set state disabled
netsh int isa set state disabled
netsh int ter set state disabled

REM Ensure IPv6 DNS entries have been cleared.
netsh int ipv6 set dns ethernet static none

Note, you'd want to change the interface name to "ethernet0" or "ethernet" as indicated in your ipconfig output on DropBox.

This leaves native IPv6 enabled should you wish to use it but disables the transitional technologies you're unlikely to require. It also gets rid of the ::1 DNS address. You can do the same thing with the GUI but the above is faster and also works on Server Core installations.

Step 1a. Standardise the primary DNS server on your domain controllers until the problem is sorted.

This is not required but it is a habit I got into in the early 2000's as a means of simplifying some diagnostic and recovery scenarios. Some people might do it, others not, but I'm going to mention it anyway.

I tend to set the primary DNS IP of any suspect servers to point back to what I know to be a trustworthy domain controller before I run around changing configurations as it helps realise the immediacy of some changes while ensuring at least that one domain controller is across things like DNS record changes and so on.

Once I'm done with recovering a situation, I'll put the primary DNS IP address back such that the server points to itself again (for writable domain controllers, in any case).

Step 2. Run "dcdiag /s:dc1.yourdomain.com /e /test:dns /dnsbasic"

This will give you an idea if basic connectivity and name resolution is up and running. Clearly, if it's not, you've got basic name resolution and connectivity troubleshooting to do (going back to the first post, things like routing and firewall issues). Telnet and tracert-d are your friends as you need to verify the connectivity between all domain controllers.

Step 3. Figure out if any domain controllers have passed the tombstone interval.

If any of your domain controllers haven't been replicating for quite some time then there's a chance that things might have been broken for long enough to have exceeded the tombstone interval (most likely 180 days unless it's an old forest) in which case that domain controller is dead in the water and cannot be resurrected. In such a case, you'll need to note any such domain controllers and add those to the list you need to remove as illustrated in the next step, step 3.

You can look for signs of this issue by:

1. Running "repadmin /showrepl" on each domain controller and looking for any errors indicating the tombstone timeframe has been exceeded as described here under the "repadmin /showrepl" section.
2. Checking the "NTDS Replication" log in Event Viewer and searching for any instances of event 2042, as discussed here.

Step 4.  Establish which domain controllers are the ones you expect to keep and remove the others (including any from step 2).

Once you have your list of domain controllers to remove, you need to use "ntdsutil" to remove the metadata - as well as your tool of choice to remove the "server" object(s) from your site configuration (i.e. use Sites and Service MMC if you're unfamiliar with the command line or PowerShell).

Here's a TechNet article on how to perform a metadata cleanup. If you have any issues, let us know and someone can walk you through it. You may well need to perform this on multiple / all of your domain controllers depending on just how broken replication is.

If you find any of the domain controllers you wanted to keep had to be removed, rebuild and re-promote them after the metadata cleanup has been finished.

This is only intended as a basic overview on how you might proceed. Let us know if you have further questions and we'll help as best we can.

Cheers,
Lain

PS: You've got way too many DNS IP addresses on host leseadc01.ourdomain.com - including loopback addresses for IPv4 and IPv6 which shouldn't be there.

You'd do well to remove all but two addresses as the longer a query takes to complete, the more likely whatever is using this server as a DNS server will experience client-side timeouts in any case (not everything waits for full iterations). Keep in mind what I said in Step 1a above as well insofar as I'd recommend pointing this domain controller to a known good DC until the diagnostic process is over and the problems are resolved (at which point change the primary address back to itself or another well connected DNS server).

Cheers,
Lain

 Starting portqry.exe -n 192.168.50.152 -e 135 -p TCP ...


Querying target system called:

 192.168.50.152

Attempting to resolve IP address to a name...


IP address resolved to LESEADC01.ourdomain.com

querying...

TCP port 135 (epmap service): LISTENING

Using ephemeral source port
Querying Endpoint Mapper Database...
Server's response:

UUID: d95afe70-a6d5-4259-822e-2c84da1ddb0d 
ncacn_ip_tcp:192.168.50.152[49152]

UUID: 50abc2a4-574d-40b3-9d66-ee4fd5fba076 
ncacn_ip_tcp:192.168.50.152[57234]

UUID: 6b5bdd1e-528c-422c-af8c-a4079be4fe48 Remote Fw APIs
ncacn_ip_tcp:192.168.50.152[57193]

UUID: 367abb81-9844-35f1-ad32-98f038001003 
ncacn_ip_tcp:192.168.50.152[57183]

UUID: 6bffd098-a112-3610-9833-46c3f874532d 
ncacn_ip_tcp:192.168.50.152[57178]

UUID: 5b821720-f63b-11d0-aad2-00c04fc324db 
ncacn_ip_tcp:192.168.50.152[57178]

UUID: f5cc59b4-4264-101a-8c59-08002b2f8426 NtFrs Service
ncacn_ip_tcp:192.168.50.152[57161]

UUID: d049b186-814f-11d1-9a3c-00c04fc9b232 NtFrs API
ncacn_ip_tcp:192.168.50.152[57161]

UUID: a00c021c-2be2-11d2-b678-0000f87a8f8e PERFMON SERVICE
ncacn_ip_tcp:192.168.50.152[57161]

UUID: c9ac6db5-82b7-4e55-ae8a-e464ed7b4277 Impl friendly name
ncacn_np:192.168.50.152[\\pipe\\lsass]

UUID: c9ac6db5-82b7-4e55-ae8a-e464ed7b4277 Impl friendly name
ncacn_ip_tcp:192.168.50.152[49155]

UUID: e3514235-4b06-11d1-ab04-00c04fc2dcd2 MS NT Directory DRS Interface
ncacn_np:192.168.50.152[\\pipe\\lsass]

UUID: e3514235-4b06-11d1-ab04-00c04fc2dcd2 MS NT Directory DRS Interface
ncacn_ip_tcp:192.168.50.152[49155]

UUID: e3514235-4b06-11d1-ab04-00c04fc2dcd2 MS NT Directory DRS Interface
ncacn_http:192.168.50.152[49166]

UUID: e3514235-4b06-11d1-ab04-00c04fc2dcd2 MS NT Directory DRS Interface
ncacn_np:192.168.50.152[\\pipe\\31f505083f0fa762]

UUID: f5cc5a18-4264-101a-8c59-08002b2f8426 MS NT Directory NSP Interface
ncacn_np:192.168.50.152[\\pipe\\lsass]

UUID: f5cc5a18-4264-101a-8c59-08002b2f8426 MS NT Directory NSP Interface
ncacn_ip_tcp:192.168.50.152[49155]

UUID: f5cc5a18-4264-101a-8c59-08002b2f8426 MS NT Directory NSP Interface
ncacn_http:192.168.50.152[49166]

UUID: f5cc5a18-4264-101a-8c59-08002b2f8426 MS NT Directory NSP Interface
ncacn_np:192.168.50.152[\\pipe\\31f505083f0fa762]

UUID: 12345778-1234-abcd-ef00-0123456789ab 
ncacn_np:192.168.50.152[\\pipe\\lsass]

UUID: 12345778-1234-abcd-ef00-0123456789ab 
ncacn_ip_tcp:192.168.50.152[49155]

UUID: 12345778-1234-abcd-ef00-0123456789ab 
ncacn_http:192.168.50.152[49166]

UUID: 12345778-1234-abcd-ef00-0123456789ab 
ncacn_np:192.168.50.152[\\pipe\\31f505083f0fa762]

UUID: 12345778-1234-abcd-ef00-0123456789ac 
ncacn_np:192.168.50.152[\\pipe\\lsass]

UUID: 12345778-1234-abcd-ef00-0123456789ac 
ncacn_ip_tcp:192.168.50.152[49155]

UUID: 12345778-1234-abcd-ef00-0123456789ac 
ncacn_http:192.168.50.152[49166]

UUID: 12345778-1234-abcd-ef00-0123456789ac 
ncacn_np:192.168.50.152[\\pipe\\31f505083f0fa762]

UUID: 12345778-1234-abcd-ef00-0123456789ac 
ncacn_ip_tcp:192.168.50.152[49167]

UUID: 0b1c2170-5732-4e0e-8cd3-d9b16f3b84d7 RemoteAccessCheck
ncacn_np:192.168.50.152[\\pipe\\lsass]


UUID: 98716d03-89ac-44c7-bb8c-285824e51c4a XactSrv service
ncacn_np:192.168.50.152[\\PIPE\\atsvc]

UUID: 98716d03-89ac-44c7-bb8c-285824e51c4a XactSrv service
ncacn_ip_tcp:192.168.50.152[49154]

UUID: 1a0d010f-1c33-432c-b0f5-8cf4e8053099 IdSegSrv service
ncacn_np:192.168.50.152[\\PIPE\\atsvc]

UUID: 1a0d010f-1c33-432c-b0f5-8cf4e8053099 IdSegSrv service
ncacn_ip_tcp:192.168.50.152[49154]

UUID: a398e520-d59a-4bdd-aa7a-3c1e0303a511 IKE/Authip API
ncacn_np:192.168.50.152[\\PIPE\\atsvc]

UUID: a398e520-d59a-4bdd-aa7a-3c1e0303a511 IKE/Authip API
ncacn_ip_tcp:192.168.50.152[49154]

UUID: a398e520-d59a-4bdd-aa7a-3c1e0303a511 IKE/Authip API
ncacn_np:192.168.50.152[\\PIPE\\srvsvc]

UUID: 552d076a-cb29-4e44-8b6a-d15e59e2c0af IP Transition Configuration endpoint
ncacn_np:192.168.50.152[\\PIPE\\atsvc]

UUID: 552d076a-cb29-4e44-8b6a-d15e59e2c0af IP Transition Configuration endpoint
ncacn_ip_tcp:192.168.50.152[49154]

UUID: 552d076a-cb29-4e44-8b6a-d15e59e2c0af IP Transition Configuration endpoint
ncacn_np:192.168.50.152[\\PIPE\\srvsvc]

UUID: 2e6035b2-e8f1-41a7-a044-656b439c4c34 Proxy Manager provider server endpoint
ncacn_np:192.168.50.152[\\PIPE\\atsvc]

UUID: 2e6035b2-e8f1-41a7-a044-656b439c4c34 Proxy Manager provider server endpoint
ncacn_ip_tcp:192.168.50.152[49154]

UUID: 2e6035b2-e8f1-41a7-a044-656b439c4c34 Proxy Manager provider server endpoint
ncacn_np:192.168.50.152[\\PIPE\\srvsvc]

UUID: c36be077-e14b-4fe9-8abc-e856ef4f048b Proxy Manager client server endpoint
ncacn_np:192.168.50.152[\\PIPE\\atsvc]

UUID: c36be077-e14b-4fe9-8abc-e856ef4f048b Proxy Manager client server endpoint
ncacn_ip_tcp:192.168.50.152[49154]

UUID: c36be077-e14b-4fe9-8abc-e856ef4f048b Proxy Manager client server endpoint
ncacn_np:192.168.50.152[\\PIPE\\srvsvc]

UUID: c49a5a70-8a7f-4e70-ba16-1e8f1f193ef1 Adh APIs
ncacn_np:192.168.50.152[\\PIPE\\atsvc]

UUID: c49a5a70-8a7f-4e70-ba16-1e8f1f193ef1 Adh APIs
ncacn_ip_tcp:192.168.50.152[49154]

UUID: c49a5a70-8a7f-4e70-ba16-1e8f1f193ef1 Adh APIs
ncacn_np:192.168.50.152[\\PIPE\\srvsvc]

UUID: 30b044a5-a225-43f0-b3a4-e060df91f9c1 
ncacn_np:192.168.50.152[\\PIPE\\atsvc]

UUID: 30b044a5-a225-43f0-b3a4-e060df91f9c1 
ncacn_ip_tcp:192.168.50.152[49154]

UUID: 30b044a5-a225-43f0-b3a4-e060df91f9c1 
ncacn_np:192.168.50.152[\\PIPE\\srvsvc]

UUID: c9ac6db5-82b7-4e55-ae8a-e464ed7b4277 Impl friendly name
ncacn_np:192.168.50.152[\\PIPE\\atsvc]

UUID: c9ac6db5-82b7-4e55-ae8a-e464ed7b4277 Impl friendly name
ncacn_ip_tcp:192.168.50.152[49154]

UUID: c9ac6db5-82b7-4e55-ae8a-e464ed7b4277 Impl friendly name
ncacn_np:192.168.50.152[\\PIPE\\srvsvc]

UUID: 201ef99a-7fa0-444c-9399-19ba84f12a1a AppInfo
ncacn_np:192.168.50.152[\\PIPE\\atsvc]

UUID: 201ef99a-7fa0-444c-9399-19ba84f12a1a AppInfo
ncacn_ip_tcp:192.168.50.152[49154]

UUID: 201ef99a-7fa0-444c-9399-19ba84f12a1a AppInfo
ncacn_np:192.168.50.152[\\PIPE\\srvsvc]

UUID: 5f54ce7d-5b79-4175-8584-cb65313a0e98 AppInfo
ncacn_np:192.168.50.152[\\PIPE\\atsvc]

UUID: 5f54ce7d-5b79-4175-8584-cb65313a0e98 AppInfo
ncacn_ip_tcp:192.168.50.152[49154]

UUID: 5f54ce7d-5b79-4175-8584-cb65313a0e98 AppInfo
ncacn_np:192.168.50.152[\\PIPE\\srvsvc]

UUID: fd7a0523-dc70-43dd-9b2e-9c5ed48225b1 AppInfo
ncacn_np:192.168.50.152[\\PIPE\\atsvc]

UUID: fd7a0523-dc70-43dd-9b2e-9c5ed48225b1 AppInfo
ncacn_ip_tcp:192.168.50.152[49154]

UUID: fd7a0523-dc70-43dd-9b2e-9c5ed48225b1 AppInfo
ncacn_np:192.168.50.152[\\PIPE\\srvsvc]

UUID: 58e604e8-9adb-4d2e-a464-3b0683fb1480 AppInfo
ncacn_np:192.168.50.152[\\PIPE\\atsvc]

UUID: 58e604e8-9adb-4d2e-a464-3b0683fb1480 AppInfo
ncacn_ip_tcp:192.168.50.152[49154]

UUID: 58e604e8-9adb-4d2e-a464-3b0683fb1480 AppInfo
ncacn_np:192.168.50.152[\\PIPE\\srvsvc]

UUID: f6beaff7-1e19-4fbb-9f8f-b89e2018337c Event log TCPIP
ncacn_np:192.168.50.152[\\pipe\\eventlog]

UUID: f6beaff7-1e19-4fbb-9f8f-b89e2018337c Event log TCPIP
ncacn_ip_tcp:192.168.50.152[49153]

UUID: 30adc50c-5cbc-46ce-9a0e-91914789e23c NRP server endpoint
ncacn_np:192.168.50.152[\\pipe\\eventlog]

UUID: 30adc50c-5cbc-46ce-9a0e-91914789e23c NRP server endpoint
ncacn_ip_tcp:192.168.50.152[49153]

UUID: 3c4728c5-f0ab-448b-bda1-6ce01eb0a6d6 DHCPv6 Client LRPC Endpoint
ncacn_np:192.168.50.152[\\pipe\\eventlog]

UUID: 3c4728c5-f0ab-448b-bda1-6ce01eb0a6d6 DHCPv6 Client LRPC Endpoint
ncacn_ip_tcp:192.168.50.152[49153]

UUID: 3c4728c5-f0ab-448b-bda1-6ce01eb0a6d5 DHCP Client LRPC Endpoint
ncacn_np:192.168.50.152[\\pipe\\eventlog]

UUID: 3c4728c5-f0ab-448b-bda1-6ce01eb0a6d5 DHCP Client LRPC Endpoint
ncacn_ip_tcp:192.168.50.152[49153]

UUID: 76f226c3-ec14-4325-8a99-6a46348418af 
ncacn_np:192.168.50.152[\\PIPE\\InitShutdown]

UUID: d95afe70-a6d5-4259-822e-2c84da1ddb0d 
ncacn_np:192.168.50.152[\\PIPE\\InitShutdown]

Total endpoints found: 98



==== End of RPC Endpoint Mapper query response ====
portqry.exe -n 192.168.50.152 -e 135 -p TCP exits with return code 0x00000000.
=============================================

 Starting portqry.exe -n 192.168.50.152 -e 389 -p BOTH ...


Querying target system called:

 192.168.50.152

Attempting to resolve IP address to a name...


IP address resolved to LESEADC01.ourdomain.com

querying...

TCP port 389 (ldap service): LISTENING

Using ephemeral source port
Sending LDAP query to TCP port 389...

LDAP query response:


currentdate: 06/08/2015 16:42:45 (unadjusted GMT)
subschemaSubentry: CN=Aggregate,CN=Schema,CN=Configuration,DC=ourdomain,DC=com
dsServiceName: CN=NTDS Settings,CN=LESEADC01,CN=Servers,CN=Bellevue,CN=Sites,CN=Configuration,DC=ourdomain,DC=com
namingContexts: DC=ourdomain,DC=com
defaultNamingContext: DC=ourdomain,DC=com
schemaNamingContext: CN=Schema,CN=Configuration,DC=ourdomain,DC=com
configurationNamingContext: CN=Configuration,DC=ourdomain,DC=com
rootDomainNamingContext: DC=ourdomain,DC=com
supportedControl: 1.2.840.113556.1.4.319
supportedLDAPVersion: 3
supportedLDAPPolicies: MaxPoolThreads
highestCommittedUSN: 762999
supportedSASLMechanisms: GSSAPI
dnsHostName: LESEADC01.ourdomain.com
ldapServiceName: ourdomain.com:leseadc01$@ourdomain.COM
serverName: CN=LESEADC01,CN=Servers,CN=Bellevue,CN=Sites,CN=Configuration,DC=ourdomain,DC=com
supportedCapabilities: 1.2.840.113556.1.4.800
isSynchronized: TRUE
isGlobalCatalogReady: TRUE
domainFunctionality: 2
forestFunctionality: 2
domainControllerFunctionality: 5


======== End of LDAP query response ========

UDP port 389 (unknown service): LISTENING or FILTERED

Using ephemeral source port
Sending LDAP query to UDP port 389...

LDAP query response:


currentdate: 06/08/2015 16:42:49 (unadjusted GMT)
subschemaSubentry: CN=Aggregate,CN=Schema,CN=Configuration,DC=ourdomain,DC=com
dsServiceName: CN=NTDS Settings,CN=LESEADC01,CN=Servers,CN=Bellevue,CN=Sites,CN=Configuration,DC=ourdomain,DC=com
namingContexts: DC=ourdomain,DC=com
defaultNamingContext: DC=ourdomain,DC=com
schemaNamingContext: CN=Schema,CN=Configuration,DC=ourdomain,DC=com
configurationNamingContext: CN=Configuration,DC=ourdomain,DC=com
rootDomainNamingContext: DC=ourdomain,DC=com
supportedControl: 1.2.840.113556.1.4.319
supportedLDAPVersion: 3
supportedLDAPPolicies: MaxPoolThreads
highestCommittedUSN: 762999
supportedSASLMechanisms: GSSAPI
dnsHostName: LESEADC01.ourdomain.com
ldapServiceName: ourdomain.com:leseadc01$@ourdomain.COM
serverName: CN=LESEADC01,CN=Servers,CN=Bellevue,CN=Sites,CN=Configuration,DC=ourdomain,DC=com
supportedCapabilities: 1.2.840.113556.1.4.800
isSynchronized: TRUE
isGlobalCatalogReady: TRUE
domainFunctionality: 2
forestFunctionality: 2
domainControllerFunctionality: 5


======== End of LDAP query response ========

UDP port 389 is LISTENING

portqry.exe -n 192.168.50.152 -e 389 -p BOTH exits with return code 0x00000000.
=============================================

 Starting portqry.exe -n 192.168.50.152 -e 636 -p TCP ...


Querying target system called:

 192.168.50.152

Attempting to resolve IP address to a name...


IP address resolved to LESEADC01.ourdomain.com

querying...

TCP port 636 (ldaps service): LISTENING
portqry.exe -n 192.168.50.152 -e 636 -p TCP exits with return code 0x00000000.
=============================================

 Starting portqry.exe -n 192.168.50.152 -e 3268 -p TCP ...


Querying target system called:

 192.168.50.152

Attempting to resolve IP address to a name...


IP address resolved to LESEADC01.ourdomain.com

querying...

TCP port 3268 (msft-gc service): LISTENING

Using ephemeral source port
Sending LDAP query to TCP port 3268...

LDAP query response:


currentdate: 06/08/2015 16:42:49 (unadjusted GMT)
subschemaSubentry: CN=Aggregate,CN=Schema,CN=Configuration,DC=ourdomain,DC=com
dsServiceName: CN=NTDS Settings,CN=LESEADC01,CN=Servers,CN=Bellevue,CN=Sites,CN=Configuration,DC=ourdomain,DC=com
namingContexts: DC=ourdomain,DC=com
defaultNamingContext: DC=ourdomain,DC=com
schemaNamingContext: CN=Schema,CN=Configuration,DC=ourdomain,DC=com
configurationNamingContext: CN=Configuration,DC=ourdomain,DC=com
rootDomainNamingContext: DC=ourdomain,DC=com
supportedControl: 1.2.840.113556.1.4.319
supportedLDAPVersion: 3
supportedLDAPPolicies: MaxPoolThreads
highestCommittedUSN: 762999
supportedSASLMechanisms: GSSAPI
dnsHostName: LESEADC01.ourdomain.com
ldapServiceName: ourdomain.com:leseadc01$@ourdomain.COM
serverName: CN=LESEADC01,CN=Servers,CN=Bellevue,CN=Sites,CN=Configuration,DC=ourdomain,DC=com
supportedCapabilities: 1.2.840.113556.1.4.800
isSynchronized: TRUE
isGlobalCatalogReady: TRUE
domainFunctionality: 2
forestFunctionality: 2
domainControllerFunctionality: 5


======== End of LDAP query response ========
portqry.exe -n 192.168.50.152 -e 3268 -p TCP exits with return code 0x00000000.
=============================================

 Starting portqry.exe -n 192.168.50.152 -e 3269 -p TCP ...


Querying target system called:

 192.168.50.152

Attempting to resolve IP address to a name...


IP address resolved to LESEADC01.ourdomain.com

querying...

TCP port 3269 (msft-gc-ssl service): LISTENING
portqry.exe -n 192.168.50.152 -e 3269 -p TCP exits with return code 0x00000000.
=============================================

 Starting portqry.exe -n 192.168.50.152 -e 53 -p BOTH ...


Querying target system called:

 192.168.50.152

Attempting to resolve IP address to a name...


IP address resolved to LESEADC01.ourdomain.com

querying...

TCP port 53 (domain service): LISTENING

UDP port 53 (domain service): LISTENING
portqry.exe -n 192.168.50.152 -e 53 -p BOTH exits with return code 0x00000000.
=============================================

 Starting portqry.exe -n 192.168.50.152 -e 88 -p BOTH ...


Querying target system called:

 192.168.50.152

Attempting to resolve IP address to a name...


IP address resolved to LESEADC01.ourdomain.com

querying...

TCP port 88 (kerberos service): LISTENING

UDP port 88 (kerberos service): LISTENING or FILTERED
portqry.exe -n 192.168.50.152 -e 88 -p BOTH exits with return code 0x00000002.
=============================================

 Starting portqry.exe -n 192.168.50.152 -e 445 -p TCP ...


Querying target system called:

 192.168.50.152

Attempting to resolve IP address to a name...


IP address resolved to LESEADC01.ourdomain.com

querying...

TCP port 445 (microsoft-ds service): LISTENING
portqry.exe -n 192.168.50.152 -e 445 -p TCP exits with return code 0x00000000.
=============================================

 Starting portqry.exe -n 192.168.50.152 -e 137 -p UDP ...


Querying target system called:

 192.168.50.152

Attempting to resolve IP address to a name...


IP address resolved to LESEADC01.ourdomain.com

querying...

UDP port 137 (netbios-ns service): LISTENING or FILTERED

Using ephemeral source port
Attempting NETBIOS adapter status query to UDP port 137...

Server's response: MAC address 00155d301400
UDP port: LISTENING
portqry.exe -n 192.168.50.152 -e 137 -p UDP exits with return code 0x00000000.
=============================================

 Starting portqry.exe -n 192.168.50.152 -e 138 -p UDP ...


Querying target system called:

 192.168.50.152

Attempting to resolve IP address to a name...


IP address resolved to LESEADC01.ourdomain.com

querying...

UDP port 138 (netbios-dgm service): LISTENING or FILTERED
portqry.exe -n 192.168.50.152 -e 138 -p UDP exits with return code 0x00000002.
=============================================

 Starting portqry.exe -n 192.168.50.152 -e 139 -p TCP ...


Querying target system called:

 192.168.50.152

Attempting to resolve IP address to a name...


IP address resolved to LESEADC01.ourdomain.com

querying...

TCP port 139 (netbios-ssn service): LISTENING
portqry.exe -n 192.168.50.152 -e 139 -p TCP exits with return code 0x00000000.
=============================================

 Starting portqry.exe -n 192.168.50.152 -e 42 -p TCP ...


Querying target system called:

 192.168.50.152

Attempting to resolve IP address to a name...


IP address resolved to LESEADC01.ourdomain.com

querying...

TCP port 42 (nameserver service): NOT LISTENING
portqry.exe -n 192.168.50.152 -e 42 -p TCP exits with return code 0x00000001.





Running the dc diag against LESEADC01 shows the problem with its dns:

Microsoft Windows [Version 6.2.9200]
(c) 2012 Microsoft Corporation. All rights reserved.

C:\Windows\system32>dcdiag /s:leseadc01.ourdomain.com /e /test:dns /dnsbasic

Directory Server Diagnosis

Performing initial setup:
   * Identified AD Forest.
   Ldap search capability attribute search failed on server LEDC03, return
   value = 81
   Got error while checking if the DC is using FRS or DFSR. Error:
   Win32 Error 81The VerifyReferences, FrsEvent and DfsrEvent tests might fail
   because of this error.
   Done gathering initial info.

Doing initial required tests

   Testing server: Burlingame\LEDC01
      Starting test: Connectivity
         ......................... LEDC01 passed test Connectivity

   Testing server: Burlingame\LEDC03
      Starting test: Connectivity
         Server LEDC03 resolved to these IP addresses: 192.168.0.42, but none
         of the addresses could be reached (pinged). Please check the network.
         Error: 0x2b02 "Error due to lack of resources."
         This error more often means that the targeted server is shutdown or
         disconnected from the network.
         Got error while checking LDAP and RPC connectivity. Please check your
         firewall settings.
         ......................... LEDC03 failed test Connectivity

   Testing server: Bellevue\LESEADC01
      Starting test: Connectivity
         ......................... LESEADC01 passed test Connectivity

   Testing server: Burlingame\LE-DC-01
      Starting test: Connectivity
         ......................... LE-DC-01 passed test Connectivity

   Testing server: Burlingame\LE-DC-02
      Starting test: Connectivity
         ......................... LE-DC-02 passed test Connectivity

Doing primary tests

   Testing server: Burlingame\LEDC01

   Testing server: Burlingame\LEDC03

   Testing server: Bellevue\LESEADC01

   Testing server: Burlingame\LE-DC-01

   Testing server: Burlingame\LE-DC-02

      Starting test: DNS

         DNS Tests are running and not hung. Please wait a few minutes...

               Starting test: DNS

                        Starting test: DNS

                                 Starting test: DNS

                                          Starting test: DNS
                                             ......................... LE-DC-02
                                             failed test DNS
                                    ......................... LE-DC-01 failed
                                    test DNS
                           ......................... LESEADC01 passed test DNS
                  ......................... LEDC01 passed test DNS
         ......................... LEDC03 failed test DNS

   Running partition tests on : DomainDnsZones

   Running partition tests on : ForestDnsZones

   Running partition tests on : Schema

   Running partition tests on : Configuration

   Running partition tests on : ourdomain

   Running enterprise tests on : ourdomain.com
      Starting test: DNS
         Test results for domain controllers:

            DC: ledc03.ourdomain.com
            Domain: ourdomain.com


               TEST: Authentication (Auth)
                  Error: Authentication failed with specified credentials

               TEST: Basic (Basc)
                  Error: No LDAP connectivity
                  Error: No WMI connectivity
                  No host records (A or AAAA) were found for this DC


            DC: LE-DC-01.ourdomain.com
            Domain: ourdomain.com


               TEST: Authentication (Auth)
                  Error: Authentication failed with specified credentials

               TEST: Basic (Basc)
                  Error: No WMI connectivity
                  No host records (A or AAAA) were found for this DC


            DC: le-dc-02.ourdomain.com
            Domain: ourdomain.com


               TEST: Authentication (Auth)
                  Error: Authentication failed with specified credentials

               TEST: Basic (Basc)
                  Error: No WMI connectivity
                  No host records (A or AAAA) were found for this DC

         Summary of DNS test results:

                                            Auth Basc Forw Del  Dyn  RReg Ext
            _________________________________________________________________
            Domain: ourdomain.com
               ledc03                       FAIL FAIL n/a  n/a  n/a  n/a  n/a
               LE-DC-01                     FAIL FAIL n/a  n/a  n/a  n/a  n/a
               le-dc-02                     FAIL FAIL n/a  n/a  n/a  n/a  n/a

         ......................... ourdomain.com failed test DNS

C:\Windows\system32>

Lain and Ace, many, many thanks for your input - it has been extremely helpful. I have cleaned up the DNS entries for the offending DC and run the script you provided Lain to remove IPv6 settings and have run the dns dcdiag and port queries (as listed above). There definitely is an issue with dns, but whether that is because of dns or overall connectivity, I am not sure. The problematic DC has been offline for about a month, so should not be in tombstone timeline as of yet. I would love to be able to resolve the issue, but if it comes to it, what would be necessary to bring up another DC at the site (and avoid these issues that are currently having)?

Again thanks so much - appreciate it.

Hi tcstrub,

Could you possibly update your DropBox location by removing the existing logs and replacing them with refreshed versions from the following commands run against each domain controller:

1. ipconfig /all
2. dcdiag /s:yourdc.ourdomain.com /e /test:dns /dnsbasic /v
3. repadmin /showrepl /all

If you can include only the results from the domain controllers on your Burlingame site at this stage, that would be appreciated as I expect you're going to need to talk to your network administrator about the Bellevue site.

What I'm looking for with a local Burlingame analysis is that:

1. All domain controllers in this site have IPv4 connectivity and can resolve each other's names.
2. That you've successfully navigated the metadata clean up.

The LDAP error 81 (or 51 hexadecimal) simply illustrates that leseadc01 cannot reach ledc03 for some reason. That may come back to the discussion you need to have with your network administrator but we'll get to that in due course.

Cheers,
Lain