Hello All
Facing a bit weird issue in environment and have been trying to fix it since few days now without any luck. We are running on Exchange 2010 SP2 UR3 (planning for SP3 UR 1) soon. Recently we observed following issues in environment:
- Outlook clients, disconnecting and reconnecting intermittently (no WAN accelerator in env)
- Databases going into disconnected & healthy state intermittently
- Total number of connections to mailbox server running very high compared to what they should be.
Now as you must be suspecting by now, we have narrowed down the issue being packet loss in environment. After lot of diagnosis, we were able to determine that cause of packet drip is RX pause packets sent from servers to switch which indicates that destination box is unable to process as fast as they are being received, so it tells switch to slow down.
To fix above issue, switch guy tell us to enable flow control on switch which will buffer the packets. However we already have flow control on our esx 4.1 farm on which exchange is running along with dedicated NIC for mapi, replication and backup.
But the bigger issue is Outlook connections. In network capture we can see CAS requesting fast packet re transmit after intermittent packet drop.
Currently all users have more than 35 connections on an average. Exchange store limits have already been bumped up. The issue I believe is due to fact that exchange is not closing previously disconnected sessions and keep accumulating them over period of day. (tcp keep alive tome is default 2 hrs). Increased connections along with network packet drop is creating intermittent pop up on outlook side it seems.
So my question being:
- How can I ensure these connections get closed beside restarting services or moving mailboxes. Should I decrease tcp kewp alive time on server?
- If anyone knows impact of enabling flow control on network switches even though they are enabled on esx host side? If it is required and any know gotchas or issues due to same?
Hopefully I covered most of background, please let me know if additional details required.
Thanks in advance !