We have an odd issue when trying to move mailboxes to Databases with Exchange 2013 Build 847.32 on all servers affected. 

We have 2013 successfully deployed within the UK (3 servers, with a DAG) we are looking to deploy an additional 2 servers in the US across an MPLS WAN. The US has it's own DAG in the US and its own MAPI network (this MAPI network is tested routable to the UK) DAG own replication network not routable, and simply a cross over...

The Migration Arbitration mailbox is located within the UK DAG, it exists as do the other Arbitration mailboxes and everything works fine moving mailboxes across DBs in the UK...

As soon as I try and move a mailbox in the US between the two server (or even on the same server within the US) it comes up with the following error after about 20mins: Worth stressing that all migration within the UK DAG work perfectly fine... 

Error: MigrationTransientException: Failed to communicate with the mailbox database. --> Failed to communicate with the mailbox database. --> MapiExceptionMdbOffline: Unable to make connection to the server. (hr=0x80004005, ec=1142) Diagnostic context: Lid: 41192 dwParam: 0x1 Lid: 63464 Lid: 34792 StoreEc: 0x6AB Lid: 51176 StoreEc: 0x80040115 Lid: 48104 Lid: 39912 StoreEc: 0x80040115 Lid: 41192 dwParam: 0x2 Lid: 55272 Lid: 34792 StoreEc: 0x6BE Lid: 51176 StoreEc: 0x80040115 Lid: 48104 Lid: 39912 StoreEc: 0x80040115 Lid: 41192 dwParam: 0x0 Lid: 49064 dwParam: 0x1 Lid: 37288 StoreEc: 0x6AB Lid: 49064 dwParam: 0x2 Lid: 38439 EMSMDBPOOL.EcPoolConnect called [length=48] Lid: 54823 EMSMDBPOOL.EcPoolConnect returned [ec=0x476][length=20][latency=796] Lid: 53361 StoreEc: 0x476 Lid: 51859 Lid: 33649 StoreEc: 0x476 Lid: 43315 Lid: 58225 StoreEc: 0x476 Lid: 39912 StoreEc: 0x476 Lid: 54129 StoreEc: 0x476 Lid: 50519 Lid: 59735 StoreEc: 0x476 Lid: 59199 Lid: 27356 StoreEc: 0x476 Lid: 65279 Lid: 52465 StoreEc: 0x476 Lid: 60065 Lid: 33777 StoreEc: 0x476 Lid: 59805 Lid: 52487 StoreEc: 0x476 Lid: 19778 Lid: 27970 StoreEc: 0x476 Lid: 17730 Lid: 25922 StoreEc: 0x476

I assume this is looking for the Migration Arbitration mailbox in the UK, across the WAN and is failing but all other features seem to work fine... In fact if I try to migrate using the Shell on a US server, it works fine so this seems to be limited to the Console...

WAN connectivity is fine, I have done a TCP any any on the ASAs to make sure there is full communication and everything else works fine... 

Any ideas as to why it seems to be failing to see the Arbitration mailbox? I t is also worth mentioning I have re-run the ADPREP /Alldomains to ensure there is no corruption within the Arbitration mailboxes...

I have probably missed something very simple somewhere but for the life of me can't think where... 

Any help gratefully received... 

   

Worth also confirming that the Exchange servers, even though the US servers are on a separate Subnet across the WAN they are within the same Domain (Forest Root) as the UK servers... 

Hi,

From your description, when you migrate mailbox via EMS, it works well. But when you do it via EAC, the above error occurs. In your case, I would like to verify if the mailbox database status is mounted when this error occurs in EAC.

Best regards,

Yes thats correct... The Mailbox database is fully mounted on both sides, the source and the destination, as well as the store with the Arbitration mailbox (within the UK)

I can migrate perfectly fine with the EAC if I work with Mailbox Databases in the UK (Where the Arbitration mailboxes are located) the EAC does not seem to work within the Americas servers... 

Hi,

In your case, the issue should be related to ECP virtual directory in American Exchange server. I recommend you recreate ECP virtual directory and check the result.

For your reference:

New-EcpVirtualDirectory

https://technet.microsoft.com/en-us/library/dd351218%28v=exchg.150%29.aspx?f=255&MSPPError=-2147217396

Hope this can be helpful to you.

Best regards,

Hi,

Any update?

Best regards,

We have already tried deleting and re-creating the ECP sites to no avail... 

It seems that it is struggling to communicate with the Arbitration mailbox over in Europe... However we have fully tested that all ports are open between the two sites successfully... 

I understand that the ECP uses a different method of moving the mailboxes than the shell, the shell uses a newer method that does not search for the arbitration mailboxes... Which is why it works using the shell and not via the ECP...

This is so frustrating and is holding up our migration significantly... 

Hi,

Since the issue is related to the migration mailbox (arbitration mailbox), I recommend you check if the following account is existed in ADUC within American servers.

Migration.8f3e7716-2011-43e4-96b1-aba62d229136

If this account does not exist within American servers, you need to run setup /prepareAD and then enable this migration mailbox to check the result.

Hope this can be helpful to you.

Best regards,

There is only 1 Migration mailbox within the Organisation, and this is within the correct OU /USERS and has been deleted and re-created via setup.exe /p and the enable mailbox and set as management powershell commands... We are sure that this mailbox is perfectly fine as it works for the UK Servers... 

We actually recreated this mailbox on the American servers and we were unable to move a mailbox while it was there, so we know it is something strange with the American server being unable to do anything with the Arbitration mailbox

Hi,

Yes, the issue is related to the arbitration mailbox, so you can create an arbitration mailbox on American servers as a workaround.

Hope my clarification can be helpful to you.

Best re

There can only be the one Arbitration mailbox, so I can't create another... 

However, I have deleted the migration mailbox, and then re-ran Forest Prep and forced it to be recreated on the American server, this then meant that I couldn't move any mailboxes anywhere within the Organisation... So I had to delete it and then recreate it on the European servers, so that at least we can move European mailboxes around... 

I agree that it is relating to the arbitration mailbox, but more related to the way the EMC is calling the Arbitration mailbox, it is almost like it will not pass communication past it's own site... 

I have confirmed that all ports are open between the sites, and have confirmed that the port is open on both boxes, so we know it is not a network issue... 

Any other ideas? 

Just to let anyone else know that experiences this issue, the issue was caused by a DCERPC Inspect rule on the Cisco ASA... 

The Inspect rule is used on ASAs to dynamically open higher range RCP dynamic ports and then close them after a set timeout... It appears that the DCERPC rule as standard does not go high enough, so some of the Exchange port 135 traffic was being dropped that was higher range than the DCERPC rule would allow... 

Removed the DCERPC Inspect rule and everything works as expected... 

• Marked as answer by cSIX 20 hours 51 minutes ago