SCOM error - ECP Health Set unhealthy - Failure Reason - CafeFailure

Hi,

I hope someone can help. I am continually getting SCOM alterts with the above appearing for all 3 of our CAS servers. Also, intermittently, I am unable to logon to the ECP (either using the DNS round robin name given to all the servers, or the server URLs themselves).  The full email is pasted below. Really not got much of a clue on this one and, it would appear, neither does google. Any ideas anyone?

Alert: Health Set unhealthy
Source: CAS01 - ECP
Path: CAS01.domain;CAS01.domain
Last modified by: System
Last modified time: 6/9/2015 3:15:51 AM
Alert description: Exchange Admin Center logon is failing on Client Access Server CAS01.
Availability has dropped to 50REPLACE_PERCENT_SIGN. You can find protocol level traces for the failures on E:\Program Files\Microsoft\Exchange Server\V15\Logging\Monitoring\ECP\EacCafeLogonProbe.

Incident start time: 06/06/2015 20:15:30

Last failed result:
Failing Component - Ecp
Failure Reason - CafeFailure

Exception: 
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> Microsoft.Exchange.Net.MonitoringWebClient.ScenarioException: Microsoft.Exchange.Net.MonitoringWebClient.ScenarioException: 
Failure source: Ecp
Failure reason: CafeFailure
Failing component:Ecp
Exception hint: CafeErrorPage: CafeFailure 500 Inner exception: Microsoft.Exchange.Net.MonitoringWebClient.CafeErrorPageException
ErrorPageFailureReason: CafeFailure, RequestFailureContext: FailurePoint=BackEnd, HttpStatusCode=500, Error=500, Details=System.Net.WebException: The remote server returned an error: (500) Internal Server Error.
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at Microsoft.Exchange.HttpProxy.ProxyRequestHandler.<>c__DisplayClass2c.<OnResponseReady>b__2b(), HttpProxySubErrorCode=, WebExceptionStatus=ProtocolError, LiveIdAuthResult=
Microsoft.Exchange.Net.MonitoringWebClient.CafeErrorPageException: An error occurred on the Client Access server while processing the request
WebExceptionStatus: Success
GET https://cas01/ecp/error.aspx?aspxerrorpath=/ecp/ HTTP/1.1
User-Agent: Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.1; MSEXCHMON; ACTIVEMONITORING)
Accept: */*
Cache-Control: no-cache
Cookie: ClientId=EJNL9BZUCOUCBOMFW; $Path=/; PBack=0; $Path=; cadata=Vr1cSpzZQnZCeDsc12CPAVbTO...; $Path=/; cadataTTL=TVOpPUWwl83RbPqZUVqrdw==; $Path=/; cadataKey=kB+UhSQVw9u2loxd+mxyvARsw...; $Path=/; cadataIV=lmNHgpiEs+eTyDaU048Vw5K0P...; $Path=/; cadataSig=LrqS26oXzq38cMY+lxzOLzDDE...; $Path=/; X-BackEndCookie=S-1-5-21-1515194898-11533...; $Path=/ecp
X-FeToBeTimeout: 100


HTTP/1.1 500 InternalServerError
request-id: aca0e1d9-54a5-4c79-a55b-c22b0db77aa6
X-CalculatedBETarget: MBX02.domain
X-DiagInfo: SHR2EXMBX02
X-BEServer: SHR2EXMBX02
X-UA-Compatible: IE=10
X-FailureContext: BackEnd;500;NTAw;U3lzdGVtLk5ldC5XZWJFeGNlcHRpb246IFRoZSByZW1vdGUgc2VydmVyIHJldHVybmVkIGFuIGVycm9yOiAoNTAwKSBJbnRlcm5hbCBTZXJ2ZXIgRXJyb3IuDQogICBhdCBTeXN0ZW0uTmV0Lkh0dHBXZWJSZXF1ZXN0LkVuZEdldFJlc3BvbnNlKElBc3luY1Jlc3VsdCBhc3luY1Jlc3VsdCkNCiAgIGF0IE1pY3Jvc29mdC5FeGNoYW5nZS5IdHRwUHJveHkuUHJveHlSZXF1ZXN0SGFuZGxlci48PmNfX0Rpc3BsYXlDbGFzczJjLjxPblJlc3BvbnNlUmVhZHk+Yl9fMmIoKQ==;;ProtocolError;
X-FEServer: CAS01
Content-Length: 1766
Cache-Control: private
Content-Type: text/html; charset=utf-8
Date: Sat, 06 Jun 2015 21:08:16 GMT
Set-Cookie: X-BackEndCookie=S-1-5-21-1515194898-11533...; expires=Mon, 06-Jul-2015 21:08:16 GMT; path=/ecp; secure; HttpOnly
Server: Microsoft-IIS/8.5
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET

<!DOCTYPE html>
<html>
    <head>
        <title>Runtime Error</title>
        <meta name="viewport" content="width=device-width" />
        <style>
         body {font-family:"Verdana";font-weight:normal;font-size: .7em;color:black;} 
         p {font-family:"Verdana";font-weight:normal;color:black;margin-top: -5px}
         b {font-family:"Verdana";font-weight:bold;color:black;margin-top: -5px}
         H1 { font-family:"Verdana";font-weight:normal;font-size:18pt;color:red }
         H2 { font-family:"Verdana";font-weight:normal;font-size:14pt;color:maroon }
         pre {font-family:"Consolas","Lucida Console",Monospace;font-size:11pt;margin:0;padding:0.5em;line-height:14pt}
         .marker {font-weight: bold; color: black;text-decoration: none;}
         .version {color: gray;}
         .error {margin-bottom: 10px;}
         .expandable { text-decoration:underline; font-weight:bold; color:navy; cursor:hand; }
         @media screen and (max-width: 639px) {
          pre { width: 440px; overflow: auto; white-space: pre-wrap; word-wrap: break-word; }
         }
         @media screen and (max-width: 479px) {
          pre { width: 280px; }
         }
        </style>
    </head>

    <body bgcolor="white">



-------------------------------------------------------------------------------

States of all monitors within the health set:
Note: Data may be stale. To get current data, run: Get-ServerHealth -Identity 'CAS01' -HealthSet 'ECP'

State               Name                                    TargetResource                     HealthSet                     AlertValue     ServerComponent     
-----               ----                                    --------------                     ---------                     ----------     ---------------     
NotApplicable       EacCtpMonitor                                                              ECP                           Unhealthy      None                
NotApplicable       MaintenanceFailureMonitor.ECP                                              ECP                           Healthy        None                
NotApplicable       MaintenanceTimeoutMonitor.ECP                                              ECP                           Healthy        None                


States of all health sets:
Note: Data may be stale. To get current data, run: Get-HealthReport -Identity 'CAS01'

State               HealthSet                     AlertValue     LastTransitionTime       MonitorCount        
-----               ---------                     ----------     ------------------       ------------        
NotApplicable       ActiveSync                    Healthy        24/05/2015 03:51:41      2                   
NotApplicable       AD                            Healthy        24/05/2015 03:55:36      9                   
NotApplicable       ECP                           Unhealthy      06/06/2015 20:15:30      3                   
Online              FrontendTransport             Healthy        24/05/2015 04:07:00      11                  
Online              HubTransport                  Unhealthy      24/05/2015 03:53:40      14                  
NotApplicable       Monitoring                    Healthy        24/05/2015 03:53:07      5                   
Online              UM.CallRouter                 Healthy        24/05/2015 03:51:17      5                   
NotApplicable       UM.Protocol                   Healthy        24/05/2015 03:52:34      2                   
NotApplicable       OAB                           Healthy        24/05/2015 03:53:32      2                   
NotApplicable       EAS                           Healthy        24/05/2015 03:51:17      2                   
NotApplicable       Autodiscover                  Healthy        06/06/2015 11:41:58      4                   
NotApplicable       Autodiscover.Protocol         Healthy        24/05/2015 03:52:55      2                   
NotApplicable       FreeBusy                      Healthy        24/05/2015 03:53:16      2                   
NotApplicable       BitlockerDeployment           Healthy        24/05/2015 03:50:22      2                   
NotApplicable       ClientAccess.Proxy            Healthy        24/05/2015 03:53:45      330                 
NotApplicable       Compliance                    Healthy        24/05/2015 04:33:27      6                   
NotApplicable       DLExpansion                   Healthy        24/05/2015 03:50:10      2                   
Online              Transport                     Healthy        24/05/2015 03:53:21      2                   
NotApplicable       EDS                           Healthy        24/05/2015 04:27:44      25                  
NotApplicable       EventAssistants               Healthy        24/05/2015 03:53:40      2                   
NotApplicable       EWS                           Healthy        06/06/2015 11:42:00      4                   
NotApplicable       EWS.Protocol                  Healthy        24/05/2015 03:52:43      2                   
NotApplicable       ExtendedReportWeb             Healthy        24/05/2015 03:53:34      2                   
NotApplicable       FEP                           Healthy        24/05/2015 03:51:08      2                   
NotApplicable       FfoRws                        Healthy        24/05/2015 03:50:54      2                   
NotApplicable       Places                        Healthy        24/05/2015 03:52:37      2                   
NotApplicable       RemoteMonitoring              Healthy        27/05/2015 04:34:08      6                   
NotApplicable       DataProtection                Healthy        24/05/2015 03:54:23      3                   
NotApplicable       Search                        Healthy        24/05/2015 03:53:24      2                   
NotApplicable       Inference                     Healthy        24/05/2015 03:49:29      2                   
NotApplicable       OWA                           Healthy        24/05/2015 03:51:20      3                   
NotApplicable       MailboxSpace                  Healthy        06/06/2015 11:44:54      2                   
NotApplicable       MRS                           Healthy        06/06/2015 11:46:46      2                   
NotApplicable       MessageTracing                Healthy        06/06/2015 11:46:28      2                   
NotApplicable       Network                       Healthy        06/06/2015 11:45:56      4                   
NotApplicable       Calendaring                   Healthy        06/06/2015 11:42:53      2                   
NotApplicable       PeopleConnect                 Healthy        06/06/2015 11:45:14      2                   
NotApplicable       IMAP                          Healthy        06/06/2015 11:44:33      2                   
NotApplicable       POP                           Healthy        06/06/2015 11:44:07      2                   
NotApplicable       ProcessIsolation              Healthy        06/06/2015 11:46:26      27                  
NotApplicable       Provisioning                  Healthy        06/06/2015 11:46:55      2                   
NotApplicable       Psws                          Healthy        06/06/2015 11:43:22      2                   
NotApplicable       PublicFolders                 Healthy        06/06/2015 11:43:47      2                   
NotApplicable       PushNotifications             Healthy        06/06/2015 11:44:24      2                   
NotApplicable       Outlook                       Healthy        06/06/2015 11:46:33      3                   
NotApplicable       RPS                           Healthy        06/06/2015 11:44:14      3                   
NotApplicable       Security                      Healthy        06/06/2015 11:45:38      3                   
NotApplicable       SiteMailbox                   Healthy        06/06/2015 11:45:31      2                   
NotApplicable       Store                         Healthy        06/06/2015 11:43:38      2                   
NotApplicable       OutlookMapiHttp               Healthy        24/05/2015 03:53:43      1                   
NotApplicable       UserThrottling                Healthy        24/05/2015 04:19:32      7                   
Online              Autodiscover.Proxy            Healthy        24/05/2015 03:48:40      1                   
Online              ActiveSync.Proxy              Healthy        24/05/2015 03:48:40      1                   
Online              ECP.Proxy                     Healthy        24/05/2015 03:48:40      1                   
Online              EWS.Proxy                     Healthy        24/05/2015 03:48:40      1                   
Online              OutlookMapiHttp.Proxy         Healthy        24/05/2015 03:48:40      1                   
Online              OAB.Proxy                     Healthy        24/05/2015 03:48:40      1                   
Online              OWA.Proxy                     Healthy        24/05/2015 03:48:40      2                   
Online              RPS.Proxy                     Healthy        24/05/2015 03:48:40      1                   
Online              RWS.Proxy                     Healthy        24/05/2015 03:48:40      1                   
NotApplicable       Antimalware                   Healthy        24/05/2015 03:51:17      2                   
NotApplicable       MailboxTransport              Healthy        24/05/2015 03:53:39      2                   
Online              Outlook.Proxy                 Healthy        24/05/2015 03:48:40      1                   
NotApplicable       EmailManagement               Healthy        24/05/2015 03:52:29      2                   
NotApplicable       FIPS                          Healthy        24/05/2015 03:51:18      2                   
NotApplicable       FfoCentralAdmin               Healthy        24/05/2015 03:52:03      2                   
NotApplicable       FfoMonitoring                 Healthy        06/06/2015 11:46:33      6                   
NotApplicable       OfficeGraph                   Healthy        06/06/2015 11:43:24      2                   
NotApplicable       HxService.Mail                Healthy        06/06/2015 11:46:56      2                   
NotApplicable       RemoteStore                   Healthy        06/06/2015 11:45:19      2                   
Online              SharedCache                   Healthy        06/06/2015 11:46:08      2                   
NotApplicable       StreamingOptics               Healthy        06/06/2015 11:43:35      2                   
NotApplicable       TimeBasedAssistants           Healthy        06/06/2015 11:46:47      2                   


Note: Subsequent detected alerts are suppressed until the health set is healthy again.

Knowledge: http://technet.microsoft.com/en-us/library/ms.exch.scom.ECP(EXCHG.150).aspx?v=15.0.1076.9

Alert view link: "http://Server02/OperationsManager?DisplayMode=Pivot&AlertID=%7b09897636-0d7e-41b2-b703-cd647763112e%7d"
Notification subscription ID generating this message: {ECB7E49D-DF13-2F63-F884-56CECD2EB2B2}


Thanks in advance...


  • Edited by wiggum83 22 hours 29 minutes ago
June 9th, 2015 5:00am

Hi Wiggum,

Thank you for your question.

We could run the following command to get Exchange version:

Get-ExchangeServer | FT Name.*Version*

For example if we will see version 15.0 <Build 1076.9>, then we will consider that Exchange version is 15.0.1076.9.

Then run the following command on all CAS server to check if the issue persist:

Add-GlobalMonitoringOverride -Identity "ECP\ EacCtpMonitor " -ItemType Responder -PropertyName Enabled -PropertyValue 0 -ApplyVersion "15.0.1076.9"  

If there are any questions regarding this issue, please be free to let me know.

Best Regard,

Jim

Free Windows Admin Tool Kit Click here and download it now
June 10th, 2015 2:09am

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics