My understanding was that msexchangerepl.exe was responsible for database seeding and replicating in a multi-node environment. However, I have noticed that store.exe transfers an immense amount of data as well between servers in the DAG. What is this process doing on the network? Thanks. Edit for clarification. This traffic is going between a server that is running as a CAS and DAG and another server that is running only as a DAG with no active mailbox databases. I would expect that the majority of data between these two servers would just be standard DAG replication from msexchangerepl.exe.

In Exchange 2007, the Microsoft Exchange Replication service was responsible for replaying logs into passive database copies. When the passive copy was activated, the database cache that had been built by the Microsoft Exchange Replication service as a result of replay activity would be lost when the Microsoft Exchange Information Store service would mount the database. This put the database cache in a state known as a cold state. The database cache, which is used to cache read/write operations, is small in size (cold) during this period. Therefore, it has a significantly diminished ability to reduce read I/O operations. In Exchange 2010, the passive copy replay functionality previously performed by the Microsoft Exchange Replication service has been moved into the Microsoft Exchange Information Store service. As a result, a warm database cache is present and immediately available for use after a failover or switchover occurs. http://technet.microsoft.com/en-us/library/dd638137.aspxSatheshwaran Manoharan | Exchange 2003/2007/2010 | Blog:http://www.careexchange.in | Please mark it as an answer if it really helps you

I'm not sure that explains why the data store is actually transmitting data over the network. That seems to say that once a server with a passive database receives replication data, it then sends it to the data store for write I/O caching instead of using a cache within the local replication service. I need to know why store.exe is transferring data between DAG nodes and how to isolate that traffic for prioritization.

if you are looking to isolate traffic. Replication data passes by only 64327 port check if that helps Satheshwaran Manoharan | Exchange 2003/2007/2010 | Blog:http://www.careexchange.in | Please mark it as an answer if it really helps you

Thanks, I tried 64327 but there is also a significant amount of data passing from other ports between the servers. On a netstat it says these ports are associated with store.exe. I am trying to determine what this traffic is and how to isolate it as well. Could it be public folders?

See http://technet.microsoft.com/en-us/library/bb331973.aspx Hope it is helpful.Fiona Liao TechNet Community Support

Is this traffic SMB on port 445? I have noticed a massive amount of traffic between DAG nodes, this was identified as Content Indexing traffic by Microsoft Support. I've never found it documented anywhere. The source was system so may not be the one you are looking at but you can use netmon to find out. Use netmon to capture just smb on one of the DAG nodes, This command line just captures one minutes worth: nmcap /network * /capture tcp.port == 445 /file %computername%.chn:200M /terminatewhen /timeafter 60 sec /captureprocesses with the captureprocess switch you can see which exe is generating the traffic. You can change the port number to suit yourself. In my case I saw lots of traffic going to between DAG nodes, this had broken in my case and was causing WAN utilisation problems with over 70GB of SMB data going in a 2 hour period with empty databases. Cause was suggested as being the empty databases, fix was to recreate and add a dummy mailbox to each. This sorted out the spike but there is still a large proportion of the replication traffic using SMB on port 445 (content indexing) while the logs themselves go over port 64327 as previously mentioned.

Is this traffic SMB on port 445? I have noticed a massive amount of traffic between DAG nodes, this was identified as Content Indexing traffic by Microsoft Support. I've never found it documented anywhere. The source was system so may not be the one you are looking at but you can use netmon to find out. Use netmon to capture just smb on one of the DAG nodes, This command line just captures one minutes worth: nmcap /network * /capture tcp.port == 445 /file %computername%.chn:200M /terminatewhen /timeafter 60 sec /captureprocesses with the captureprocess switch you can see which exe is generating the traffic. You can change the port number to suit yourself. In my case I saw lots of traffic going to between DAG nodes, this had broken in my case and was causing WAN utilisation problems with over 70GB of SMB data going in a 2 hour period with empty databases. Cause was suggested as being the empty databases, fix was to recreate and add a dummy mailbox to each. This sorted out the spike but there is still a large proportion of the replication traffic using SMB on port 445 (content indexing) while the logs themselves go over port 64327 as previously mentioned. Will, I believe we found the same thing. I SPAN'ed all the ports on the switch that the servers are connected do and did a passive packet capture for roughly 2 minutes. This was the top data between the local Exchange servers and the far end Exchange server at our other site. (Node-1 is local, Node-2 is at the other site) Address A Port A Owning Application Address B Port Owning Application Packets Bytes Node-1 64327 msexchangerepl.exe Node-2 50692 msexchangerepl.exe 10862 9.94 MB Node-1 64327 msexchangerepl.exe Node-2 46170 msexchangerepl.exe 8213 7 MB Node-1 64327 msexchangerepl.exe Node-2 50697 msexchangerepl.exe 7870 7 MB Node-1 64327 msexchangerepl.exe Node-2 50665 msexchangerepl.exe 7485 7 MB Node-1 64327 msexchangerepl.exe Node-2 50684 msexchangerepl.exe 7283 7 MB Node-2 50643 Node-1 445 10204 5 MB Node-1 64327 msexchangerepl.exe Node-2 50698 msexchangerepl.exe 5547 5 MB Node-2 50652 store.exe Node-1 49903 Microsoft.Exchange.Search.ExSearch.exe 6494 3 MB Node-2 50679 store.exe Node-1 49903 MsFTEFD.exe 3247 2 MB Node-2 50680 store.exe Node-1 49903 MsFTEFD.exe 2443 1.8 MB Node-2 50653 store.exe Node-1 49903 Microsoft.Exchange.Search.ExSearch.exe 2369 0.7 MB Node-1 49903 store.exe Node-2 29731 MsFTEFD.exe 944 0.7 MB Node-2 50681 Node-1 49903 msexchangerepl.exe 718 0.6 MB Node-2 50682 Node-1 49903 msexchangerepl.exe 293 0.2 MB Node-2 50654 store.exe Node-1 49903 Microsoft.Exchange.Search.ExSearch.exe 723 0.2 MB A lot of it appears to be replication, then there is a 443 stream. While 5MB does not seem like a lot compared to replication, over a 2 minute period that is using more than a full T1. This is in addition to these store.exe processes that also appear to be part of the Exchange search indexer. I found a some brief documentation that mentions this, but it still doesn't add up. http://technet.microsoft.com/en-us/library/bb232132.aspx In organizations that have a database availability group (DAG), during the seeding process,DAG members with a passive mailbox database copy replicate the content index catalog fromthe DAG member that has the active mailbox database copy. The content index is typically10 percent the size of the mailbox database. After initial seeding, the server with thepassive database copy gets message data from the server with the active database andperforms content indexing locally. The bandwidth used for copying message content forindexing is in addition to the bandwidth used for replication of transaction logs.When planning a high availability deployment, you must consider the bandwidth usedby Exchange Search. Some other website specified that this type of replication occurs over microsoft-ds (445) but I am not able to find it right now. What is strange is that during the time of capture we were not doing a database seed, just normal log shipping.

you might want to take al look at this blog: http://blogs.technet.com/b/exchange/archive/2011/12/14/database-maintenance-in-exchange-2010.aspx Especially Database Checksumming: it reads ~5Mb/s for each actively scanning database (both active and passive database). If you monitor your mailbox servers (resmon.exe) you'll see the store.exe proces causing the Read IO / network load because it's reading large chunks of data from the databases for checksumming. Regards, Pascal

you might want to take al look at this blog: http://blogs.technet.com/b/exchange/archive/2011/12/14/database-maintenance-in-exchange-2010.aspx Especially Database Checksumming: it reads ~5Mb/s for each actively scanning database (both active and passive database). If you monitor your mailbox servers (resmon.exe) you'll see the store.exe proces causing the Read IO / network load because it's reading large chunks of data from the databases for checksumming. Regards, Pascal