exchange 2013 SP1 with a 2 member DAG. As the subject suggests, no external email is flowing. Internal email flows fine and we can send email but incoming email is not working. Here's what i did last:
put 1 server in maintenance mode and performed windows updates. There were a total of 13; 1 exchange spam definiftion and the rest were windows updates. Reboot the server. Windows did its thing with "configuring updates". Logged back on and took
the server out of maintenance mode.
Logged into the other server and repeated the above steps.
Now, no external email is being received. I have restarted the exchange transport service on both servers. I have rebooted both servers. A connectivity test says not listening on port 25 so i look at the firewall and all looks fine there. Went ahead and
rebooted the firewall. Looking at event viewer on one of the servers:
msexchange common ID = 106:
Performance counter updating error. Counter name is Per-Tenant KeyToRemoveBudgets Cache Size, category name is MSExchangeRemotePowershell. Optional code: 3. Exception: The exception thrown is : System.InvalidOperationException: The requested Performance Counter is not a custom counter, it has to be initialized as ReadOnly.
at System.Diagnostics.PerformanceCounter.InitializeImpl()
at System.Diagnostics.PerformanceCounter.get_RawValue()
at Microsoft.Exchange.Diagnostics.ExPerformanceCounter.set_RawValue(Int64 value)
Last worker process info : System.ArgumentException: Process with an Id of 17004 is not running.
at System.Diagnostics.Process.GetProcessById(Int32 processId)
at Microsoft.Exchange.Diagnostics.ExPerformanceCounter.GetLastWorkerProcessInfo()
Processes running while Performance counter failed to update:
8272 MSExchangeDelivery
588 winlogon
3148 MSExchangeMigrationWorkflow
1768 FrameworkService
2552 sftracing
384 smss
684 svchost
9640 svchost
1364 spoolsv
8060 w3wp
2740 WMSvc
8844 MSExchangeMailboxReplication
1172 MSExchangeDagMgmt
1552 fms
6476 scanningprocess
12504 TBMon
8640 w3wp
560 wininit
1544 naPrdMgr
9620 svchost
752 svchost
5676 postgres
11056 w3wp
552 csrss
7052 Microsoft.Exchange.AntispamUpdateSvc
352 svchost
5288 MSExchangeSubmission
8736 Microsoft.Exchange.EdgeSyncSvc
3696 noderunner
9408 Microsoft.Exchange.Store.Worker
3300 rhs
2708 svchost
11572 explorer
9584 postgres
4276 rhs
11816 csrss
3876 noderunner
8996 MSExchangeFrontendTransport
13132 msdtc
720 svchost
8008 UMWorkerProcess
2444 rundll32
12140 dwm
1404 svchost
1696 MaseRulesUpdater
4452 conhost
5436 scanningprocess
10664 conhost
4444 RPCServ
2872 cmd
8908 Microsoft.Exchange.Search.Service
3020 Microsoft.Exchange.Directory.TopologyService
3652 msexchangerepl
860 dwm
1876 MSExchangeHMHost
496 csrss
2464 noderunner
9752 svchost
13612 w3wp
12584 w3wp
3248 postgres
3016 MSExchangeMailboxAssistants
2260 conhost
3604 noderunner
13880 TiWorker
12680 postgres
3240 postgres
1860 updateservice
5756 MSExchangeTransportLogSearch
6388 w3wp
5204 Microsoft.Exchange.ServiceHost
11844 winlogon
3232 postgres
6972 w3wp
2636 svchost
2240 rundll32
3224 postgres
13068 w3wp
9132 Microsoft.Exchange.Store.Service
9388 VSSVC
2432 SMSvcHost
4400 Microsoft.Exchange.RpcClientAccess.Service
2232 mqsvc
3216 postgres
4284 rhs
2076 rdpclip
652 lsass
848 LogonUI
2816 clussvc
9316 svchost
6588 SMSvcHost
2908 postgres
644 services
3400 WmiPrvSE
13104 w3wp
440 svchost
1424 bedbg
832 Microsoft.Exchange.Diagnostics.Service
3392 WmiPrvSE
5164 wksprt
13240 WmiPrvSE
10284 taskhostex
1812 ForefrontActiveDirectoryConnector
6240 w3wp
3976 MSExchangeTransport
11832 ParserServer
12868 mmc
1212 svchost
6136 umservice
5740 Microsoft.Exchange.UM.CallRouter
420 svchost
1576 hostcontrollerservice
1600 McSACore
7316 w3wp
2780 MSExchangeHMWorker
4748 conhost
4156 conhost
11796 w3wp
2212 SAFeService
1672 inetinfo
7412 ServerManager
13956 ParserServer
4028 svchost
4740 RPCServ
6512 scanningprocess
4120 EdgeTransport
4736 MSExchangeThrottling
6508 TrustedInstaller
10032 svchost
1384 conhost
4 System
2748 conhost
5124 beremote
0 Idle
Performance Counters Layout information: FileMappingNotFoundException for category MSExchangeRemotePowershell : Microsoft.Exchange.Diagnostics.FileMappingNotFoundException: Cound not open File mapping for name Global\netfxcustomperfcounters.1.0msexchangeremotepowershell. Error Details: 2
at Microsoft.Exchange.Diagnostics.FileMapping..ctor(String name, Boolean writable)
at Microsoft.Exchange.Diagnostics.PerformanceCounterMemoryMappedFile.Initialize(String fileMappingName, Boolean writable)
at Microsoft.Exchange.Diagnostics.ExPerformanceCounter.GetAllInstancesLayout(String categoryName)
ASP.NET 4.0.30319.0 event id = 1309
Event code: 3001
Event message: The request has been aborted.
Event time: 8/10/2015 1:37:38 PM
Event time (UTC): 8/10/2015 5:37:38 PM
Event ID: 1d3405fee9a1462e8c74b588192d2926
Event sequence: 2
Event occurrence: 1
Event detail code: 0
Application information:
Application domain: /LM/W3SVC/2/ROOT/mapi/emsmdb-1-130837017375363922
Trust level: Full
Application Virtual Path: /mapi/emsmdb
Application Path: C:\Program Files\Microsoft\Exchange Server\V15\ClientAccess\mapi\emsmdb\
Machine name: HOSTNAME
Process information:
Process ID: 6388
Process name: w3wp.exe
Account name: NT AUTHORITY\SYSTEM
Exception information:
Exception type: HttpException
Exception message: Request timed out.
Request information:
Request URL: https://hostname.domain.tld:444/mapi/emsmdb/?mailboxId=14970465-5102-4158-a0be-92ab799ae411@domain.tld
Request path: /mapi/emsmdb/
User host address: 192.168.123.9
User: DOMAIN\EXCHANGE$
Is authenticated: True
Authentication Type: Negotiate
Thread account name: NT AUTHORITY\SYSTEM
Thread information:
Thread ID: 9
Thread account name: NT AUTHORITY\SYSTEM
Is impersonating: False
Stack trace:
Custom event details:
msexchange common event id = 4999
Watson report about to be sent for process id: 3604, with parameters: E12, c-RTL-AMD64, 16.00.0847.030, NodeRunner#ContentEngineNode1, M.Exchange.StoreProvider, Microsoft.Exchange.Search.Mdb.StoreSessionCache.CreateStoreSession, IllegalCrossServerConnectionEx4: Client=CI, 5d440b9a, 15.0.0.0.
ErrorReportingEnabled: False
msexchangetransport event id = 7004
The activation of all modules took longer than expected to complete. Total Load Time: 00:00:25.2628377 Total Start Time: 00:00:00.0620390 Breakdown:
<LoadTimings>
<Component Name="TransportConfiguration" Elapsed="00:00:00.4738625" />
<Component Name="Certificate, RemoteDelivery and Database components" Elapsed="00:00:11.2755427">
<Component Name="Database and dependents" Elapsed="00:00:11.2754665">
<Component Name="MessagingDatabase" Elapsed="00:00:10.6266077" />
<Component Name="ResourceManager" Elapsed="00:00:00.0109835" />
<Component Name="Microsoft.Exchange.Transport.Storage.IPFiltering.Database" Elapsed="00:00:00.6377793" />
</Component>
<Component Name="Microsoft.Exchange.Transport.CertificateComponent" Elapsed="00:00:00.0038076" />
<Component Name="RemoteDelivery" Elapsed="00:00:00.0728482" />
</Component>
<Component Name="SmtpReceiveAgents" Elapsed="00:00:02.9254297" />
<Component Name="AD Configuration Readers" Elapsed="00:00:05.3481238">
<Component Name="Microsoft.Exchange.Transport.Components+TransportMailItemLoader" Elapsed="00:00:00.0018326" />
<Component Name="Microsoft.Exchange.Transport.OrarGenerator" Elapsed="00:00:00.0147869" />
<Component Name="Microsoft.Exchange.Transport.RecipientAPI.IsMemberOfResolverComponent`1[Microsoft.Exchange.Data.Transport.RoutingAddress]" Elapsed="00:00:00.1303095" />
<Component Name="Microsoft.Exchange.Transport.RecipientAPI.IsMemberOfResolverComponent`1[System.String]" Elapsed="00:00:00.0042433" />
<Component Name="PerfCounters" Elapsed="00:00:00.1587449" />
<Component Name="Microsoft.Exchange.Transport.Components+LoggingComponent" Elapsed="00:00:00.1306421" />
<Component Name="Microsoft.Exchange.Transport.PoisonMessage" Elapsed="00:00:00.0014586" />
<Component Name="Microsoft.Exchange.Transport.DsnGenerator" Elapsed="00:00:00.0433187" />
<Component Name="Microsoft.Exchange.Transport.Components+DirectTrustLoader" Elapsed="00:00:00.0191300" />
<Component Name="Microsoft.Exchange.Transport.Components+MicrosoftExchangeRecipientLoader" Elapsed="00:00:00.0442465" />
<Component Name="Microsoft.Exchange.Transport.Components+ServicePrincipalNameRegistrar" Elapsed="00:00:00.0357243" />
<Component Name="RoutingAgents" Elapsed="00:00:05.3455610" />
<Component Name="StorageAgents" Elapsed="00:00:00.0017384" />
<Component Name="Microsoft.Exchange.Transport.MessageThrottling.MessageThrottlingComponent" Elapsed="00:00:00.0003576" />
<Component Name="RmsClientManager" Elapsed="00:00:00.0000116" />
<Component Name="ShadowRedundancy" Elapsed="00:00:00.2398865" />
<Component Name="SafetyNet" Elapsed="00:00:00.1383299" />
</Component>
<Component Name="AD/Database Dependent Components" Elapsed="00:00:00.2642467">
<Component Name="Microsoft.Exchange.Transport.RemoteDelivery.QueueManager" Elapsed="00:00:00.2641110" />
</Component>
<Component Name="Microsoft.Exchange.Transport.Categorizer.RoutingComponent" Elapsed="00:00:00.1480142" />
<Component Name="Dns" Elapsed="00:00:00.0764196" />
<Component Name="Microsoft.Exchange.Transport.Categorizer.UnhealthyTargetFilterComponent" Elapsed="00:00:00.0041697" />
<Component Name="Categorizer" Elapsed="00:00:00.1036018" />
<Component Name="Categorizer Dependent Components" Elapsed="00:00:04.5629851">
<Component Name="Microsoft.Exchange.Transport.Components+RpcServerComponent" Elapsed="00:00:00.0000119" />
<Component Name="Microsoft.Exchange.Protocols.Smtp.SmtpInComponent" Elapsed="00:00:00.0584332" />
<Component Name="Microsoft.Exchange.Transport.Components+StoreDriverLoader" Elapsed="00:00:00.3434350" />
<Component Name="Microsoft.Exchange.Transport.Pickup.PickupComponent" Elapsed="00:00:00.0000271" />
<Component Name="Microsoft.Exchange.Transport.Storage.Messaging.Utah.BootScanner" Elapsed="00:00:01.3006153" />
<Component Name="SmtpOut" Elapsed="00:00:00.0022416" />
<Component Name="Microsoft.Exchange.Transport.RemoteDelivery.NonSmtpGatewayConnectionHandler" Elapsed="00:00:00.0455598" />
<Component Name="DeliveryAgents" Elapsed="00:00:00.0026155" />
<Component Name="Microsoft.Exchange.Transport.Components+AggregatorLoader" Elapsed="00:00:04.5599261" />
<Component Name="ProcessingQuota" Elapsed="00:00:00.0508139" />
</Component>
<Component Name="Microsoft.Exchange.Transport.BackgroundProcessingThread" Elapsed="00:00:00.0000164" />
</LoadTimings>
<StartTimings>
<Component Name="Certificate, RemoteDelivery and Database components" Elapsed="00:00:00.0000480">
<Component Name="Database and dependents" Elapsed="00:00:00.0000080" />
<Component Name="RemoteDelivery" Elapsed="00:00:00.0000188" />
</Component>
<Component Name="AD Configuration Readers" Elapsed="00:00:00.0003325">
<Component Name="ShadowRedundancy" Elapsed="00:00:00.0000085" />
<Component Name="SafetyNet" Elapsed="00:00:00.0003189" />
</Component>
<Component Name="AD/Database Dependent Components" Elapsed="00:00:00.0000010" />
<Component Name="Categorizer" Elapsed="00:00:00.0000062" />
<Component Name="Categorizer Dependent Components" Elapsed="00:00:00.0607315">
<Component Name="Microsoft.Exchange.Transport.Components+RpcServerComponent" Elapsed="00:00:00.0593132" />
<Component Name="Microsoft.Exchange.Protocols.Smtp.SmtpInComponent" Elapsed="00:00:00.0010827" />
<Component Name="Microsoft.Exchange.Transport.Components+StoreDriverLoader" Elapsed="00:00:00.0000079" />
<Component Name="Microsoft.Exchange.Transport.Pickup.PickupComponent" Elapsed="00:00:00.0000115" />
<Component Name="Microsoft.Exchange.Transport.Storage.Messaging.Utah.BootScanner" Elapsed="00:00:00.0000046" />
<Component Name="SmtpOut" Elapsed="00:00:00.0001810" />
<Component Name="Microsoft.Exchange.Transport.RemoteDelivery.NonSmtpGatewayConnectionHandler" Elapsed="00:00:00.0000064" />
<Component Name="DeliveryAgents" Elapsed="00:00:00.0000073" />
<Component Name="Microsoft.Exchange.Transport.Components+AggregatorLoader" Elapsed="00:00:00.0000710" />
</Component>
<Component Name="Microsoft.Exchange.Transport.BackgroundProcessingThread" Elapsed="00:00:00.0005464" />
</StartTimings>
msexchangediagnostics
The EDS Job manager failed to start the following jobs:
Job: 'TransportSyncHealthHubLog' is poisoned.
Any ideas? Should i uninstall the updates?