Hi all,

I have TMG 2010 SP1 (without update 1 installed-there's a problem with this I'll describe in a new thread), installed in the domain with Exchange 2007 SP3 (with latest rollups) Edge role. Everything's been running fine for nearly a year, until it came to renewing the Edge subscription (certificate was about to expire). That went well enough, but at the same time the Exchange server ran out of disk space, so I had to migrate the user mailboxes elsewhere. Since then, I've had this recurring problem where the Control service on TMG crashes (not sure if the subscription/mailbox move is relevant, but I'm providing as much info as possible).

In summary, I've managed to narrow the failure down to being caused by any firewall rules that use our External web listener (that is, services such as OWA, ActiveSync, and SharePoint). Internal access using an Internal web listener work fine (the only difference between the 2 listeners is Internal is configured to listen on the Internal network, External on the External network, both listeners use the same wildcard cert).

When a user accesses the services above, they can log in but once they try to do something the session stops because the Control service has crashed (I'm presented with a 'Send Report to Microsoft' box on the server), the Firewall service stops itself and nothing comes in or goes out. When I restart the Firewall service, the TMG Managed Control service crashes which I then have to restart too.

I've been running with these firewall rules disabled for a little while now, and haven't had the service crash (yet), but that's not a good solution as users can't check their emails from home. I have tried creating a new listener and new rules, but that still causes a crash. Here are some logs from the Event Viewer;

When the Control Service crashes;

 

Log Name:  Application
Source:  Application Error
Date:   25/02/2011 17:33:49
Event ID:  1000
Task Category: (100)
Level:   Error
Keywords:  Classic
User:   N/A
Computer:  server.domain.local
Description:
Faulting application mspadmin.exe, version 7.0.8108.200, time stamp 0x4c17aca0, faulting module ncrypt.dll, 
version 6.0.6002.18005, time stamp 0x49e0419b, exception code 0xc0000005, fault offset 0x000000000000310e, 
process id 0xb9c, application start time 0x01cbd50653ba6415.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
 <System>
 <Provider Name="Application Error" />
 <EventID Qualifiers="0">1000</EventID>
 <Level>2</Level>
 <Task>100</Task>
 <Keywords>0x80000000000000</Keywords>
 <TimeCreated SystemTime="2011-02-25T17:33:49.000Z" />
 <EventRecordID>311227</EventRecordID>
 <Channel>Application</Channel>
 <Computer>server.domain.local</Computer>
 <Security />
 </System>
 <EventData>
 <Data>mspadmin.exe</Data>
 <Data>7.0.8108.200</Data>
 <Data>4c17aca0</Data>
 <Data>ncrypt.dll</Data>
 <Data>6.0.6002.18005</Data>
 <Data>49e0419b</Data>
 <Data>c0000005</Data>
 <Data>000000000000310e</Data>
 <Data>b9c</Data>
 <Data>01cbd50653ba6415</Data>
 </EventData>
</Event>

This appears to be caused by 'ncrypt.dll', searching for this revealed that a 'fix' may be to copy a good version of this file, I have done this and it's made no difference.

Then, the TMG Firewall stops;

 

Log Name:  Application
Source:  Microsoft Forefront TMG Firewall
Date:   25/02/2011 17:34:01
Event ID:  14182
Task Category: None
Level:   Information
Keywords:  Classic
User:   N/A
Computer:  server.domain.local
Description:
The Firewall service was stopped gracefully.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
 <System>
 <Provider Name="Microsoft Forefront TMG Firewall" />
 <EventID Qualifiers="16384">14182</EventID>
 <Level>4</Level>
 <Task>0</Task>
 <Keywords>0x80000000000000</Keywords>
 <TimeCreated SystemTime="2011-02-25T17:34:01.000Z" />
 <EventRecordID>311228</EventRecordID>
 <Channel>Application</Channel>
 <Computer>server.domain.local</Computer>
 <Security />
 </System>
 <EventData>
 </EventData>
</Event>

 

And then, if I restart the Firewall while Managed Control is running;

 

Log Name:  Application
Source:  Application Error
Date:   28/02/2011 09:39:20
Event ID:  1000
Task Category: (100)
Level:   Error
Keywords:  Classic
User:   N/A
Computer:  server.domain.local
Description:
Faulting application IsaManagedCtrl.exe, version 7.0.8108.200, time stamp 0x4c17ac26, faulting module unknown, 
version 0.0.0.0, time stamp 0x00000000, exception code 0xc0000005, fault offset 0x000000006fe6f37c, 
process id 0x%9, application start time 0x%10.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
 <System>
 <Provider Name="Application Error" />
 <EventID Qualifiers="0">1000</EventID>
 <Level>2</Level>
 <Task>100</Task>
 <Keywords>0x80000000000000</Keywords>
 <TimeCreated SystemTime="2011-02-28T09:39:20.000Z" />
 <EventRecordID>313832</EventRecordID>
 <Channel>Application</Channel>
 <Computer>server.domain.local</Computer>
 <Security />
 </System>
 <EventData>
 <Data>IsaManagedCtrl.exe</Data>
 <Data>7.0.8108.200</Data>
 <Data>4c17ac26</Data>
 <Data>unknown</Data>
 <Data>0.0.0.0</Data>
 <Data>00000000</Data>
 <Data>c0000005</Data>
 <Data>000000006fe6f37c</Data>
 </EventData>
</Event>

 

I have found that I can typically stop the Managed Control service first, then restart the Firewall with no problems. I also have an ongoing alert that is new;

 

Log Name:  Application
Source:  Microsoft Forefront TMG Control
Date:   28/02/2011 11:29:39
Event ID:  32572
Task Category: None
Level:   Error
Keywords:  Classic
User:   N/A
Computer:  server.domain.local
Description:
Cache log failure: Failed to write content to the cache log; this may interfere with cache utilization monitoring. 




The failure is due to error: Category does not exist.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
 <System>
 <Provider Name="Microsoft Forefront TMG Control" />
 <EventID Qualifiers="49152">32572</EventID>
 <Level>2</Level>
 <Task>0</Task>
 <Keywords>0x80000000000000</Keywords>
 <TimeCreated SystemTime="2011-02-28T11:29:39.000Z" />
 <EventRecordID>314616</EventRecordID>
 <Channel>Application</Channel>
 <Computer>server.domain.local</Computer>
 <Security />
 </System>
 <EventData>
 <Data>Category does not exist.</Data>
 </EventData>
</Event>

 

I don't know if this is relevant, a topic for another thread, or nothing to worry about.

Thanks

• Edited by Matthew Peters_UK Monday, February 28, 2011 12:14 PM cleaned up code

Matthew,

The key to the post about the crash is that you had an expiring certificate.  Was that replaced with a new one and did you delete the old one?  If the old one is in place, I would suggest removing it from the system totally.  This may be the root cause due to conflicting certificates.  If you want for more in depth troubleshooting, you may have to open a case with Microsoft Support.  One of the tools we would use to isolate the root cause wold be DebugDiag, which can be downloaded from http://www.microsoft.com/downloads/en/details.aspx?FamilyID=28bd5941-c458-46f1-b24d-f60151d875a3&displaylang=en. 

 

Thanks

Hi Brennan,

Thanks for your reply! Initially, I had tried to renew the subscription, which ended in failure, so I created a new one. During the process, I had trouble generating a subscription using the TMG console, so I ended up generating the subscription with the EMS. Having looked at the certificates installed, there were two with the same name, same expiry date, but physically different (different thumbprints, amongst other things).

To make things simpler, I removed the edge subscription and disabled connectivity in TMG, then proceeded to remove any associated certificates to start from scratch. I enabled connectivity in TMG, then had a problem generating the files (error message saying the Managed Control Service wasn't responding, even though it was). I was able to get around this by creating a new certificate, enabling SMTP by running

New-ExchangeCertificate –Services SMTP –DomainName server.domain.local

I'm not sure I formed the command correctly first time round, so I'm going to try again tomorrow and see what happens.

Thanks

Ok, seems like the above was correct, I ran

New-ExchangeCertificate –Services SMTP –DomainName mail.domain.com, edgeserver, hubserver

to generate the SMTP cert, then was able to generate the subscription file. I'd admit, the amount of information I managed to find about the above command was scarce (wrt the correct parameters), so I took a best guess. I have since been through all certs on the edge and hub servers and removed any that were not required (including one that was for mail.domain.com that has been superceeded). I've had no errors on either server about missing SMTP certificates, there is mail flowing in and out and the event log shows edgesync running and succeeding every time, so it seem edgesync is working fine.

I've re-enabled the Activesync rule and those devices that use it seem to be working fine, with no crashes on the server so far (note the external Activesync rule uses the same SSL listener as the other rules), I have also re-enabled OWA access, but am yet to test it.

OWA still causes the service to crash, I have attempted to run the DebugDiag tool, but that hasn't come back with anything that appears useful.

Hi Metthew,

I would propose to open call in CSS at this point. Service crash is not something we can deal with on forum.

Thanks,

 

I think I've got somewhere towards what may be causing the issue here. I have recently had to move the mailbox database to an external drive (the server was low on disk space). This external drive is connected to the LAN and seen on the Exchange server as an iSCSI device (therefore seen as a local disk with drive letter etc). I fear now I'm getting into 'unsupported' territory here, and that this fault may not be with TMG directly.

This morning I have tested Outlook Anywhere remote connections, these work fine without crashing the service. Autodiscover also doesn't cause a crash, and I've had ActiveSync running since yesterday without problems.

I created a new mailbox database (and storage group) on the old drive (the one that was full) and moved a mailbox to it. I connected using OWA to that mailbox with no problems. On moving the mailbox to the iSCSI device, the Control service crashed on access. I then created a new storage group/mailbox database on the iSCSI device (in case the existing one was hosed when it was moved around) and made a mailbox there, which on access crashed the Control service. So it seems there's a problem when the mailbox is on the iSCSI device.

This may now be a question for the Exchange forum, but with the crash apparently caused by ncrypt.dll, this looks like some sort of authentication thing going awry, and caused by the iSCSI box hosting the databases. Any thoughts what might be going wrong?

Thanks

I have been fighting this error (Event ID 1000) since August.  i have opened 2 separate cases with MS and still no resolution.  First call they were geared to a certificate issue we ended by upgrading to 7.0.8108.200..Second call was more focused on an Exchange HUB/CAS issue...after a month with no resolution in site they recommend a reload from scratch!!

We are running Forefront TMG in an array configuration so we have 2 servers that load balance the requests.  We get this error on both of them a lot.  bad thing about it is it takes services down till the service comes back online or you down the offending box.

Reloading is not an option since it is happening on both boxes..I can see this being a solution if one of the boxes were popping this error.

please let me know if you find anything..I am pulling my hair out here!!

thanks!

Hi Matthew,

 

Unfortunately, due to the complexity of this issue we are unable to effectively assist with this request in the forum.

 

I would like to suggest that you contact Microsoft Product Support Services via telephone so that a dedicated Support Professional can assist with this request.

 

To obtain the phone numbers for specific technology request please take a look at the web site listed below.

 

http://support.microsoft.com/default.aspx?scid=fh;EN-US;PHONENUMBERS

 

If you are outside the US please see http://support.microsoft.com for regional support phone numbers.

 

Thank you for your patience and understanding.

 

Regards,

Hey Matthew,

I have opened another case with MS to see if we can again hunt this error down.  It seems to now be crashing 2 times a day.  I will update this forum with any useful information that may arise.

 

 

Hi i have same issue for last month and finally i was able to solve it

because detailed steps .. who needs solution contact me : mohamed_15@hotmail.com