store.exe won't gracefully shut down
Exchange 2007 SP1 enterprise, RU6 on Windows 2008 x64 Enterprise. Geographically dispersed CCR mailbox cluster with CAS/HTS on separate servers. From-scratch installation on dedicated hardware (HP BL460c with 2x E5430 and 16 GB RAM). Symantec Mail security for Exchange 6.0.7. No non-Exchange roles or services on the mailbox nodes. Approximately 150 users across three sites. We're still debugging some issues with the witness node in our CCR cluster which is causing failover when none is needed. This would be uneventful, except that store.exe won't shut down gracefully on the node which is active at the start of the failover. As such, it never releases/closes the E0x.log files, so the cluster replay service can't copy them to the node coming online. The information stores won't come online on the target node with a lossless failover. Because the cluster service has already stopped waiting for the information store instance to shut down, it's proceeded with the failover and left the cluster in a partially-failed state. Store.exe will stay running for hours after the failover is initiated. The workaround is to manually terminate the store.exe process, copy the E0x.log files to their respective target locations, then bring each of the information stores online on the now-active server. We never have any data loss, but the manual aspect of this failover is causing us considerable grief. The only notable thing about the store.exe process is that the failure to shut down is more likely to happen the longer the information store instance has been running. On the active node, store.exe will eventually consume nearly all the RAM on the server (14 out of 16 GB), but CPU is nearly idle and users have no complaints about responsiveness. If the information store has been running for more than about a week straight, failover is guaranteed to be a problem. We need to solve the witness node issue, but there are still going to be times when we want a graceful shutdown for maintenance, et cetera. What can we do to troubleshoot this issue with the information store?
July 9th, 2009 5:06pm
I agree, you should not have to manually terminate the store. Do all of the databases get transferred propery and mounted on the other server during a move or failover. Can you upgrade to RU8? One thought I had was what would happen if you disabled Symantec Mail Security? When I have used SMS in the past, it gobbles up a TON of memory and puts a lot ofhooks in to the store. Jim McBee - Blog - http://mostlyexchange.blogspot.com
Free Windows Admin Tool Kit Click here and download it now
July 9th, 2009 9:13pm