Here is a hotfix for AFD.sys on Server 2008 that mentions a high handle count / memory leak issue

http://support.microsoft.com/kb/977332

That or possibly some other hotfix that updates AFD.sys might be a route to pursue.

• Edited by KevinMJohnston Tuesday, August 12, 2014 8:51 PM
• Proposed as answer by Joyce LMicrosoft contingent staff, Moderator Tuesday, August 19, 2014 10:32 AM

I have recently been deploying a new SCCM 2012 R2 environment in a place where there's never been SMS/SCCM or any other management in place. 

Over the last few months of doing this we've had a few cases of servers losing connectivity to the domain so you need to log in using the local admin password.  Shares are unavailable.  Netlogon errors in the event log.  It's happened to about 8-10 Windows 2008 servers so far.  I don't have a hard number because we didn't put together that it was a pattern right away.  After lots of head scratching I found one in that state and saw that CcmExec.exe had about 17,000 handles, as opposed to other systems I looked at that may have a few hundred.  Killing CcmExec.exe instantly brought the server back to its normal state, complete domain connectivity.   The next time it happened on a different system, still Windows 2008, it was the same thing.  This one around 13,000 handles.   I looked and it was all connections to \Device\Afd .  I have one I'm looking at right now with CcmExec at 11,895 handles, but it hasn't caused a problem yet.  I know I can restart the service and it'll go back down, but that's not fixing it. 

I'm not seeing it everywhere, but it's now happened enough that had to stop the rollout of SCCM clients until we can figure out what's going on.  Anyone have any suggestions on where to go from here? 

CSS would be the place to go for something like this. That does not sound normal at all.

Here is a hotfix for AFD.sys on Server 2008 that mentions a high handle count / memory leak issue

http://support.microsoft.com/kb/977332

That or possibly some other hotfix that updates AFD.sys might be a route to pursue.

• Edited by KevinMJohnston Tuesday, August 12, 2014 8:51 PM
• Proposed as answer by Joyce LMicrosoft contingent staff, Moderator Tuesday, August 19, 2014 10:32 AM

As an update to this.  I did open a case, but he hadn't gotten anywhere with it yet.

I found what was triggering this, if not the underlying cause. I'm waiting for the support engineer to hopefully verify.  But I'll share in case anyone ever comes across it.

The systems affected all had one thing in common.  Client push install grabbed and used a PKI certificate that was installed locally for another purpose.  My site was set to use PKI certificate when available.  We aren't using PKI certs for clients yet, but were keeping the option open for the future so I hadn't imported any trusted root certificate authorities yet. 

When I unchecked "use PKI certificate when available" the affect clients stopped creating file handles.  These were all building around 3000 handles a day from CcmExec.exe, they stopped overnight. My best guess is that the client using a PKI certificate that isn't for SCCM caused the clients to keep trying to validate that cert, and exposed an issue with either CcmExec or AFD.  I'm going to try to recreate this in a lab environment, and hope the support engineer is able to do the same. But it stopped entirely on all of them as soon as I made that one change.

• Edited by PMall Tuesday, August 19, 2014 7:49 PM

As an update to this.  I did open a case, but he hadn't gotten anywhere with it yet.

I found what was triggering this, if not the underlying cause. I'm waiting for the support engineer to hopefully verify.  But I'll share in case anyone ever comes across it.

The systems affected all had one thing in common.  Client push install grabbed and used a PKI certificate that was installed locally for another purpose.  My site was set to use PKI certificate when available.  We aren't using PKI certs for clients yet, but were keeping the option open for the future so I hadn't imported any trusted root certificate authorities yet. 

When I unchecked "use PKI certificate when available" the affect clients stopped creating file handles.  These were all building around 3000 handles a day from CcmExec.exe, they stopped overnight. My best guess is that the client using a PKI certificate that isn't for SCCM caused the clients to keep trying to validate that cert, and exposed an issue with either CcmExec or AFD.  I'm going to try to recreate this in a lab environment, and hope the support engineer is able to do the same. But it stopped entirely on all of them as soon as I made that one change.

• Edited by PMall Tuesday, August 19, 2014 7:49 PM

ConfigMgr does not have a specific PKI cert, as I recall, it just uses a workstation authentication cert. So it would find any with that purpose, though it does expect some specific configuration of the cert (well described in the docs on PKI cert requirements for Configuration Manager).

No solution yet. I have the same problem. My servers are crashing because there are to many handles created by ccmexe

same problem :(

I have 4 terminal servers with Windows 2008 R2 installed from one vhd-image

I have problem with many handles on 3 of 4 servers

Temporary solution: restart ccmexec service every night with task scheduler

• Edited by FighterZP 6 hours 12 minutes ago

same problem :(

I have 4 terminal servers with Windows 2008 R2 installed from one vhd-image

I have problem with many handles on 3 of 4 servers

Temporary solution: restart ccmexec service every night with task scheduler

• Edited by FighterZP 6 hours 14 minutes ago

same problem :(

I have 4 terminal servers with Windows 2008 R2 installed from one vhd-image

I have problem with many handles on 3 of 4 servers

Temporary solution: restart ccmexec service every night with task scheduler

• Edited by FighterZP 22 hours 59 minutes ago

I tried this trick and it works

disabling BGB 

No idea what functionality I miss . Now we need to create a case at MS for this workaround

same problem :(

I have 4 terminal servers with Windows 2008 R2 installed from one vhd-image

I have problem with many handles on 3 of 4 servers

Temporary solution: restart ccmexec service every night with task scheduler

• Edited by FighterZP Thursday, March 05, 2015 9:38 AM

same problem :(

I have 4 terminal servers with Windows 2008 R2 installed from one vhd-image

I have problem with many handles on 3 of 4 servers

Temporary solution: restart ccmexec service every night with task scheduler

• Edited by FighterZP Thursday, March 05, 2015 9:38 AM

I tried this trick and it works

disabling BGB 

No idea what functionality I miss . Now we need to create a case at MS for this workaround

• Proposed as answer by DanielRatliff 15 hours 5 minutes ago

I wouldn't call that a "trick", I would call that disabling functionality.

Client Notification (aka BGB) is exactly what its name says it is, the ability to notify a client to perform an action from/by the site.

Typically, intermittent issues like this are caused by the AV product in use.

When you say "your servers", which servers *exactly* are you talking about?

We are also facing the similar issue, we have implemented a workaround that will restart the ccmexec service if the process handles increase beyond 6000. We are working with MS for the permanent fix.

http://msexchange.me/2015/03/18/restart-service-to-avoid-process-handle-leaks/

Here is the Microsoft's answer to my case:

Please do following on SCCM Server:
Uncheck the box  Client Notification (TCP) from site configuration which uses 10123 port

This will have no effect on Clients. All clients will switch automatically to port 80.
Please after doing the above step wait 1 hour and then restart the SMS_Notification_Server Service and SMS_Agent_Host-Service on the Server, this will reset the services back to normal.

The cause:
This is BUG from SCCM and will be fixed with CU5 or SP2. The workaround we are using (revert back to port 80) is very simple and can be also left without changing it back after Cu5 or SP2.

Here is the Microsoft's answer to my case:

Please do following on SCCM Server:
Uncheck the box  Client Notification (TCP) from site configuration which uses 10123 port

This will have no effect on Clients. All clients will switch automatically to port 80.
Please after doing the above step wait 1 hour and then restart the SMS_Notification_Server Service and SMS_Agent_Host-Service on the Server, this will reset the services back to normal.

The cause:
This is BUG from SCCM and will be fixed with CU5 or SP2. The workaround we are using (revert back to port 80) is very simple and can be also left without changing it back after Cu5 or SP2.

• Proposed as answer by DanielRatliff 15 hours 5 minutes ago

Can you please tell me if:

1. Those servers are Management Points?

2. You are using the 'Client Notification' feature?