We have a clustered (active/passive) Exchange 2007 mailbox server, patched up to date (Exchange and the OS - Server 2003 R2). Since the early hours of this morning, store.exe keeps failing about every 5 minutes. It ramps up to around 3.5Gb of memory usage (it's usually over 6) and then restarts. This isn't enough to automatically fail over the cluster, but when we did that manually, the problem moved to the other node.Users with mailboxes on that server seem to get their backed-up mail delievered just before the server fails again. There are no public folders and no AV on the server. Other mailbox server in the organization are fine. Event Viewer doesn't appear to offer any useful insight.Has anyone seen anything like this, or have any suggestions?

A bit more detail...The Event ID is 9659:"The Store.exe process stops responding on a server that is running Microsoft Exchange Server 2007. Additionally, the following event is logged in the event log: Event ID : 9659 Source : MSExchangeIS Type : Error Description : The Microsoft Exchange Information Store encountered an unexpected exception 0xC0000005 at address <var><address></var> while processing a request for user <var><OTHER_SERVER$></var>."http://support.microsoft.com/kb/941655 states this was fixed in Exchange 2007 Rollup 6 or Exchange 2007 SP1 Rollup 1, but this server is at SP2 with Rollup 2.

And more...This article is exactly what we're seeing: http://support.microsoft.com/kb/960495/en-usIt shouldn't be an issue on our SP/Patch level.

That's a generic access violation. While a bug was fixed that causes one in that KB, that very much so doesn't mean the source of all of these crashes was fixed, unfortunately. You need to call PSS.Active Directory, 4th Edition - www.briandesmond.com/ad4/

Hi,Try running ExMon on the problematic cluster and find the user consuming most of the memory. Also, try moving such user to a stable cluster/ exchange server and check if the issue persists. __________________________ Sanjay

Hi Brian,Yep, we've got a call in for this. Working with them to isolate the exact location of the problem. Just wondered if anyone else had seen this. Like you say, the error is fairly generic, but I can find no reference to those events on this patch level.

Sanjay, thanks for your suggestion. ExMon didn't identify the location of this problem, although I was able to narrow it down to a particular mailbox by cross-referencing the queues on the hub-transport and the mail tracking logs when the store.exe failures occurred - God bless PowerShell! ;-)It turns out that messages going to this particular mailbox from outside the organization were causing the Information Store Service to restart. The problem followed the mailbox to another server and was repeatable when a specific message was replayed through the hub transport.It's still not clear exactly what the specific cause of the failure was because it was triggered by one message to the mailbox yesterday and a different one today. Different content, different senders, same recipient. Other messages sent from inside and outside the organization to that user's mailbox were delivered just fine. It seems to be something to do with rules that the user had setup, but since you can't explicitly make a rule that says "when new message contains x, crash the store.exe process and screw up the mailbox server for all its users", it may be that the same circumstances could happen again.What we can do if it happens again is check the queues on the hub transport servers, find the message that is stuck at the top of a queue beyond store.exe failures, suspend that message. Check the message's destination and...- If we want to confirm it is a repeat of this problem:Move the mailbox to a test server and un-suspend that message. Since it's already been categorized by that time it will try to go to the original mailbox server, find the mailbox has moved, go to the submission queue, sit there for 30 minutes, then be put in the right queue for the mailbox's new location. If it is a repeat of this issue, it would then crash store.exe on the new mailbox server.While the message is suspended on a queue, you can export it to a .eml file that you can play with and replay as often as you'd like to repeat the issue.or- If we just want to get the message delivered and move on:Clean or remove the rules on the target mailbox and let the queue retry.

Hi Jonathan, I don't see this issue before...Thank you for your detail problem analysis. I suggest you keep on working with our PSS support, once you get it fixed, would you please share with us? It will benifit more people here. Thank you.