Hello Pavan Maganti,
I can confirm that following below were enabled on the DAG NICs:
Link-Layer Topology Discovery Mapper I/O Driver
Link-Layer Topology Discovery Responder
I then checked the registry to ensure IPv6 was enabled.
I notice there was no "DisabledComponents" registry subkey on all Exchnage 2013 servers which made me suspect it could of been something with IPv6 ???
On on all the Exchnage 2013 servers I created a New DWORD (32-bit) Value called DisabledComponents in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip6\Parameters and set the Value data to 0 (this enabling all IPv6 components which apparently is
the Windows default setting)
I rebooted all the Exchnage 2013 servers and then tried to add the mailbox servers into the DAG again = It FAILED with the same error message.
Note: This is error log im getting from C:\ExchangeSetupLogs\DagTasks:
add-databaseavailabiltygroupserver started on machine EXCMB1.
[2014-01-02T23:29:02] add-dagserver started
[2014-01-02T23:29:02] commandline: $scriptCmd = {& $wrappedCmd @PSBoundParameters }
[2014-01-02T23:29:02] Option 'Identity' = 'EXCD13'.
[2014-01-02T23:29:02] Option 'MailboxServer' = 'EXCMB2'.
[2014-01-02T23:29:02] Option 'DatabaseAvailabilityGroupIpAddresses' = ''.
[2014-01-02T23:29:02] Option 'WhatIf' = ''.
[2014-01-02T23:29:02] Process: w3wp w3wp.exe:8452.
[2014-01-02T23:29:02] User context = 'NT AUTHORITY\SYSTEM'.
[2014-01-02T23:29:02] Member of group 'Everyone'.
[2014-01-02T23:29:02] Member of group 'EXCMB1\SophosUser'.
[2014-01-02T23:29:02] Member of group 'BUILTIN\Users'.
[2014-01-02T23:29:02] Member of group 'NT AUTHORITY\SERVICE'.
[2014-01-02T23:29:02] Member of group 'CONSOLE LOGON'.
[2014-01-02T23:29:02] Member of group 'NT AUTHORITY\Authenticated Users'.
[2014-01-02T23:29:02] Member of group 'NT AUTHORITY\This Organization'.
[2014-01-02T23:29:02] Member of group 'BUILTIN\IIS_IUSRS'.
[2014-01-02T23:29:02] Member of group 'LOCAL'.
[2014-01-02T23:29:02] Member of group 'IIS APPPOOL\MSExchangeECPAppPool'.
[2014-01-02T23:29:02] Member of group 'BUILTIN\Administrators'.
[2014-01-02T23:29:02] Updated Progress 'Validating the parameters.' 2%.
[2014-01-02T23:29:02] Working
[2014-01-02T23:29:02] Mailbox server: value passed in = EXCMB2, mailboxServer.Name = EXCMB2, mailboxServer.Fqdn = EXCMB2.contoso.local
[2014-01-02T23:29:02] LogClussvcState: clussvc is Stopped on EXCMB2.contoso.local. Exception (if any) = none
[2014-01-02T23:29:02] The IP addresses for the DAG are (blank means DHCP): 172.19.220.47
[2014-01-02T23:29:02] Looking up IP addresses for EXCD13.
[2014-01-02T23:29:04] Failure while trying to resolve EXCD13: threw a SocketException: No such host is known.
[2014-01-02T23:29:04] Looking up IP addresses for excmb2.
[2014-01-02T23:29:04] excmb2 = [ 172.19.220.46 ].
[2014-01-02T23:29:04] Looking up IP addresses for EXCMB2.contoso.local.
[2014-01-02T23:29:04] EXCMB2.contoso.local = [ 172.19.220.46 ].
[2014-01-02T23:29:04] DAG EXCD13 has 0 servers:
[2014-01-02T23:29:04] Updated Progress 'Checking if Mailbox server 'EXCMB2' is in a database availability group.' 4%.
[2014-01-02T23:29:04] Working
[2014-01-02T23:29:04] GetRemoteCluster() for the mailbox server failed with exception = An Active Manager operation failed. Error: An error occurred while attempting a cluster operation. Error: Cluster API '"OpenCluster(EXCMB2.contoso.local) failed with 0x6d9.
Error: There are no more endpoints available from the endpoint mapper"' failed.. This is OK.
[2014-01-02T23:29:04] Ignoring previous error, as it is acceptable if the cluster does not exist yet.
[2014-01-02T23:29:04] DumpClusterTopology: Opening remote cluster EXCD13.
[2014-01-02T23:29:04] DumpClusterTopology: Failed opening with Microsoft.Exchange.Cluster.Replay.AmGetFqdnFailedNotFoundException: An Active Manager operation failed with a transient error. Please retry the operation. Error: The fully qualified domain name
for node 'EXCD13' could not be found.
at Microsoft.Exchange.Cluster.ActiveManagerServer.AmServerNameCache.<>c__DisplayClass1.<ResolveFqdn>b__0(Object param0, EventArgs param1)
at Microsoft.Exchange.Cluster.ActiveManagerServer.AmHelper.RunADOperationEx(EventHandler ev)
at Microsoft.Exchange.Cluster.ActiveManagerServer.AmServerNameCache.ResolveFqdn(String shortNodeName, Boolean throwException)
at Microsoft.Exchange.Cluster.ActiveManagerServer.AmServerNameCache.GetFqdn(String shortNodeName, Boolean throwException)
at Microsoft.Exchange.Cluster.ActiveManagerServer.AmServerName..ctor(String serverName, Boolean throwOnFqdnError)
at Microsoft.Exchange.Management.SystemConfigurationTasks.DumpClusterTopology..ctor(String nameCluster, ITaskOutputHelper output)
[2014-01-02T23:29:04] Dumping the cluster by connecting to: EXCD13.
[2014-01-02T23:29:04] DumpClusterTopology: Not dumping, since the cluster could not be contacted.
[2014-01-02T23:29:04] The new server (EXCMB2) is not a member of a cluster, nor are the other servers (if there are any other servers).
[2014-01-02T23:29:04] The computer account EXCD13 does not exist.
[2014-01-02T23:29:04] According to GetNodeClusterState(), the server EXCMB2 is NotConfigured.
[2014-01-02T23:29:04] InternalValidate() done.
[2014-01-02T23:29:04] Updated Progress 'Adding server 'EXCMB2' to database availability group 'EXCD13'.' 6%.
[2014-01-02T23:29:04] Working
[2014-01-02T23:29:04] Connecting to server 'EXCMB2.contoso.local' via WMI...
[2014-01-02T23:29:05] Fetching the network adapters and including the ones without default gateways.
[2014-01-02T23:29:05] EXCMB2.contoso.local has an address: 172.19.220.46/24 default gateway(s)=172.19.220.250 [valid]
[2014-01-02T23:29:05] IP address derived that's suitable for clustering DHCP/IPv6: 172.19.220.0/24.
[2014-01-02T23:29:05] EXCMB2.contoso.local has an address: fe80::30a5:ab07:f05:d10c/64 default gateway(s)=172.19.220.250 [invalid]
[2014-01-02T23:29:05] EXCMB2.contoso.local has an address: 10.1.1.2/24 default gateway(s)=<none> [valid]
[2014-01-02T23:29:05] IP address derived that's suitable for clustering DHCP/IPv6: 10.1.1.0/24.
[2014-01-02T23:29:05] EXCMB2.contoso.local has an address: fe80::91ec:7d06:8415:73a3/64 default gateway(s)=<none> [invalid]
[2014-01-02T23:29:05] The address 172.19.220.47 falls under the network (172.19.220.0/24).
[2014-01-02T23:29:05] Updated Progress 'Forming cluster named 'EXCD13' on server 'EXCMB2'.' 8%.
[2014-01-02T23:29:05] Working
[2014-01-02T23:29:05] Forming cluster named 'EXCD13' on server 'EXCMB2'. The cluster address is/are [172.19.220.47], and the cluster network prefix length is/are [24].
[2014-01-02T23:29:07] The following log entry comes from a different process that's running on machine 'EXCMB2'. BEGIN
[2014-01-02T23:29:07] [2014-01-02T23:29:07] Updated Progress 'Forming cluster named 'EXCD13' on server 'excmb2'.' 2%.
[2014-01-02T23:29:07] Working
[2014-01-02T23:29:07] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseInitialize, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 5, szObjectName = EXCD13, dwStatus = 0x0 )
[2014-01-02T23:29:07] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseInitialize, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 5, szObjectName = , dwStatus = 0x0 )
[2014-01-02T23:29:07] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateNodeState, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 10, szObjectName = EXCMB2.contoso.local, dwStatus
= 0x0 )
[2014-01-02T23:29:08] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateNodeState, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 10, szObjectName = , dwStatus = 0x0 )
[2014-01-02T23:29:09] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseQueryClusterNameAccount, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 15, szObjectName = EXCD13, dwStatus = 0x0 )
[2014-01-02T23:29:09] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseQueryClusterNameAccount, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 15, szObjectName = , dwStatus = 0x0 )
[2014-01-02T23:29:09] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateClusterNameAccount, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 21, szObjectName = EXCD13, dwStatus = 0x0
)
[2014-01-02T23:29:09] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateClusterNameAccount, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseFatal, dwPercentComplete = 21, szObjectName = EXCD13, dwStatus = 0xc000018b )
[2014-01-02T23:29:09] Found a matching exception: Microsoft.Exchange.Cluster.Replay.DagTaskComputerAccountCouldNotBeValidatedException: A server-side database availability group administrative operation failed. Error Computer account 'EXCD13' couldn't be validated
by user 'NT AUTHORITY\SYSTEM'. Error: Unknown error (0xc000018b).
[2014-01-02T23:29:09] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseValidateClusterNameAccount, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseFatal, dwPercentComplete = 21, szObjectName = EXCD13, dwStatus = 0xc000018b )
[2014-01-02T23:29:09] Found a matching exception: Microsoft.Exchange.Cluster.Replay.DagTaskComputerAccountCouldNotBeValidatedException: A server-side database availability group administrative operation failed. Error Computer account 'EXCD13' couldn't be validated
by user 'NT AUTHORITY\SYSTEM'. Error: Unknown error (0xc000018b).
[2014-01-02T23:29:09] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseFailureCleanup, ePhaseType = ClusterSetupPhaseStart, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 21, szObjectName = EXCMB2.contoso.local, dwStatus =
0x0 )
[2014-01-02T23:29:09] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseFailureCleanup, ePhaseType = ClusterSetupPhaseContinue, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 21, szObjectName = EXCD13, dwStatus = 0x0 )
[2014-01-02T23:29:09] ClusterSetupProgressCallback( eSetupPhase = ClusterSetupPhaseFailureCleanup, ePhaseType = ClusterSetupPhaseEnd, ePhaseSeverity = ClusterSetupPhaseInformational, dwPercentComplete = 21, szObjectName = , dwStatus = 0x0 )
[2014-01-02T23:29:07] The operation wasn't successful because an error was encountered. You may find more details in log file "C:\ExchangeSetupLogs\DagTasks\dagtask_2014-01-02_23-29-02.426_add-databaseavailabiltygroupserver.log".
[2014-01-02T23:29:07] WriteError! Exception = Microsoft.Exchange.Cluster.Replay.DagTaskOperationFailedException: A server-side database availability group administrative operation failed. Error The operation failed. CreateCluster errors may result from incorrectly
configured static addresses. Error: Computer account 'EXCD13' couldn't be validated by user 'NT AUTHORITY\SYSTEM'. Error: Unknown error (0xc000018b). ---> Microsoft.Exchange.Cluster.Replay.DagTaskComputerAccountCouldNotBeValidatedException: A server-side
database availability group administrative operation failed. Error Computer account 'EXCD13' couldn't be validated by user 'NT AUTHORITY\SYSTEM'. Error: Unknown error (0xc000018b).
at Microsoft.Exchange.Cluster.ClusApi.AmCluster.CreateExchangeCluster(String clusterName, AmServerName firstNodeName, String[] ipAddress, UInt32[] ipPrefixLength, IClusterSetupProgress setupProgress, IntPtr context, Exception& failureException,
Boolean throwExceptionOnFailure)
at Microsoft.Exchange.Cluster.ClusApi.ClusterFactory.CreateExchangeCluster(String clusterName, AmServerName firstNodeName, String[] ipAddress, UInt32[] ipPrefixLength, IClusterSetupProgress setupProgress, IntPtr context, Exception& failureException,
Boolean throwExceptionOnFailure)
at Microsoft.Exchange.Cluster.Replay.DagHelper.CreateDagCluster(String clusterName, AmServerName firstNodeName, String[] ipAddresses, UInt32[] netmasks, String& verboseLog)
--- End of inner exception stack trace (Microsoft.Exchange.Cluster.Replay.DagTaskComputerAccountCouldNotBeValidatedException) ---
at Microsoft.Exchange.Cluster.Replay.DagHelper.ThrowDagTaskOperationWrapper(Exception exception)
at Microsoft.Exchange.Cluster.Replay.DagHelper.CreateDagCluster(String clusterName, AmServerName firstNodeName, String[] ipAddresses, UInt32[] netmasks, String& verboseLog)
at Microsoft.Exchange.Cluster.ReplayService.ReplayRpcServer.<>c__DisplayClass40.<RpcsCreateCluster>b__3f()
at Microsoft.Exchange.Data.Storage.Cluster.HaRpcExceptionWrapperBase`2.RunRpcServerOperation(String databaseName, RpcServerOperation rpcOperation)
--- End of stack trace on server (EXCMB2.contoso.local) ---
at Microsoft.Exchange.Data.Storage.Cluster.HaRpcExceptionWrapperBase`2.ClientRethrowIfFailed(String databaseName, String serverName, RpcErrorExceptionInfo errorInfo)
at Microsoft.Exchange.Cluster.Replay.ReplayRpcClientWrapper.RunCreateCluster(AmServerName rpcServerName, String clusterName, AmServerName firstNode, String[] ipaddrs, UInt32[] netmasks, String& verboseLog)
at Microsoft.Exchange.Management.SystemConfigurationTasks.AddDatabaseAvailabilityGroupServer.FormCluster()
[2014-01-02T23:29:07] Updated Progress 'Done!' 100%.
[2014-01-02T23:29:07] COMPLETED
add-databaseavailabiltygroupserver explicitly called CloseTempLogFile().