Hello,

I have a newly-installed Exchange 2013 SP1 server (all roles installed on one server) that I installed CU6 on, and now the incoming mail flow has stopped.  Incoming mail is rejected at SMTP time with the following message:

"451 4.7.0 Temporary server error. Please try again later. PRX5"

From the logs, I traced it back to the Transport Service "Default SERVERNAME" receive connector, which is configured to listen on 0.0.0.0:2525.  Here are the relevant log snippets:

D:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\Logs\FrontEnd\ProtocolLog\SmtpReceive\RECV20150321-1.LOG:
2015-03-21T17:13:07.697Z,excambio\Default Frontend EXCAMBIO,08D2320EEE2DB8E8,30,127.0.0.1:25,127.0.0.1:23768,*,,Proxy destination(s) obtained from OnProxyInboundMessage event
2015-03-21T17:13:07.697Z,excambio\Default Frontend EXCAMBIO,08D2320EEE2DB8E8,31,127.0.0.1:25,127.0.0.1:23768,*,,NextHopFqdn property is null or whitespace when creating InboundProxyLayer
2015-03-21T17:13:08.743Z,excambio\Default Frontend EXCAMBIO,08D2320EEE2DB8E8,32,127.0.0.1:25,127.0.0.1:23768,*,,"Message or connection acked with status Retry and response 441 4.4.1 Error encountered while communicating with primary target IP address: ""Failed to connect. Winsock error code: 10061, Win32 error code: 10061."" Attempted failover to alternate host, but that did not succeed. Either there are no alternate hosts, or delivery failed to all alternate hosts. The last endpoint attempted was 192.168.27.25:2525"
2015-03-21T17:13:08.743Z,excambio\Default Frontend EXCAMBIO,08D2320EEE2DB8E8,33,127.0.0.1:25,127.0.0.1:23768,>,451 4.7.0 Temporary server error. Please try again later. PRX5 ,

D:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\Logs\FrontEnd\ProtocolLog\SmtpSend\SEND20150321-1.LOG:
2015-03-21T17:08:07.749Z,Inbound Proxy Internal Send Connector,08D2320EEE2DB8E2,0,,192.168.27.25:2525,*,,attempting to connect
2015-03-21T17:08:08.778Z,Inbound Proxy Internal Send Connector,08D2320EEE2DB8E2,1,,192.168.27.25:2525,*,,"Failed to connect. Winsock error code: 10061, Win32 error code: 10061, Error Message: No connection could be made because the target machine actively refused it 192.168.27.25:2525"
2015-03-21T17:08:12.397Z,Client Proxy Send Connector,08D2320EEE2DB8E4,0,,192.168.27.25:465,*,,attempting to connect. Client proxy session for HealthMailbox9301776481cb41068c2afd4f62f32744@jonheese.com

D:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\Logs\FrontEnd\Connectivity\CONNECTLOG20150321-1.LOG:
2015-03-21T17:13:07.697Z,08D2320EEE2DB8E9,SMTP,internalproxy,+,Undefined 00000000-0000-0000-0000-000000000000;QueueLength=<no priority counts>
2015-03-21T17:13:07.697Z,08D2320EEE2DB8E9,SMTP,internalproxy,>,excambio.jonheese.local[192.168.27.25]
2015-03-21T17:13:08.743Z,08D2320EEE2DB8E9,SMTP,internalproxy,>,Failed connection to 192.168.27.25:2525 (ConnectionRefused:0000274D)[TargetHost:excambio.jonheese.local:2525|MarkedUnhealthy|FailureCount:5|NextRetryTime:2015-03-21T17:13:08.778Z][TargetIPAddress:192.168.27.25:2525|MarkedUnhealthy|FailureCount:5|NextRetryTime:2015-03-21T17:13:08.778Z]
2015-03-21T17:13:08.743Z,08D2320EEE2DB8E9,SMTP,internalproxy,-,Messages: 0 Bytes: 0 (Retry : Unable to connect)

When I try telnet'ing to localhost:2525 (or the actual IP of the exchange server), I either get an initial connection that drops after I hit any key, or I get connection refused.

I have confirmed that the "Default SERVERNAME" connector is configured appropriately, under the HubTransport role, listening on 0.0.0.0:2525:

[PS] C:\Windows\system32>Get-ReceiveConnector "excambio\Default EXCAMBIO" | fl


RunspaceId                              : 73ca4d69-3185-41ac-abff-a82be444730c
AuthMechanism                           : Tls, Integrated, BasicAuth, BasicAuthRequireTLS, ExchangeServer
Banner                                  :
BinaryMimeEnabled                       : True
Bindings                                : {[::]:2525, 0.0.0.0:2525}
ChunkingEnabled                         : True
DefaultDomain                           :
DeliveryStatusNotificationEnabled       : True
EightBitMimeEnabled                     : True
SmtpUtf8Enabled                         : False
BareLinefeedRejectionEnabled            : False
DomainSecureEnabled                     : False
EnhancedStatusCodesEnabled              : True
LongAddressesEnabled                    : False
OrarEnabled                             : False
SuppressXAnonymousTls                   : False
ProxyEnabled                            : False
AdvertiseClientSettings                 : False
Fqdn                                    : excambio.jonheese.local
ServiceDiscoveryFqdn                    :
TlsCertificateName                      :
Comment                                 :
Enabled                                 : True
ConnectionTimeout                       : 00:10:00
ConnectionInactivityTimeout             : 00:05:00
MessageRateLimit                        : Unlimited
MessageRateSource                       : IPAddress
MaxInboundConnection                    : 5000
MaxInboundConnectionPerSource           : 20
MaxInboundConnectionPercentagePerSource : 2
MaxHeaderSize                           : 128 KB (131,072 bytes)
MaxHopCount                             : 60
MaxLocalHopCount                        : 12
MaxLogonFailures                        : 3
MaxMessageSize                          : 35 MB (36,700,160 bytes)
MaxProtocolErrors                       : 5
MaxRecipientsPerMessage                 : 200
PermissionGroups                        : ExchangeUsers, ExchangeServers, ExchangeLegacyServers
PipeliningEnabled                       : True
ProtocolLoggingLevel                    : Verbose
RemoteIPRanges                          : {::-ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff, 0.0.0.0-255.255.255.255}
RequireEHLODomain                       : False
RequireTLS                              : False
EnableAuthGSSAPI                        : False
ExtendedProtectionPolicy                : None
LiveCredentialEnabled                   : False
TlsDomainCapabilities                   : {}
Server                                  : excambio
TransportRole                           : HubTransport
SizeEnabled                             : Enabled
TarpitInterval                          : 00:00:05
MaxAcknowledgementDelay                 : 00:00:30
AdminDisplayName                        :
ExchangeVersion                         : 0.1 (8.0.535.0)
Name                                    : Default EXCAMBIO
DistinguishedName                       : CN=Default EXCAMBIO,CN=SMTP Receive
                                          Connectors,CN=Protocols,CN=excambio,CN=Servers,CN=Exchange Administrative
                                          Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=JonHeese,CN=Microsoft
                                          Exchange,CN=Services,CN=Configuration,DC=jonheese,DC=local
Identity                                : excambio\Default EXCAMBIO
Guid                                    : 018e86b8-351e-4ba5-92f9-f527f8200c36
ObjectCategory                          : jonheese.local/Configuration/Schema/ms-Exch-Smtp-Receive-Connector
ObjectClass                             : {top, msExchSmtpReceiveConnector}
WhenChanged                             : 3/21/2015 12:43:50 PM
WhenCreated                             : 3/21/2015 11:57:44 AM
WhenChangedUTC                          : 3/21/2015 4:43:50 PM
WhenCreatedUTC                          : 3/21/2015 3:57:44 PM
OrganizationId                          :
OriginatingServer                       : artemis.jonheese.local
IsValid                                 : True
ObjectState                             : Unchanged

And I've already tried the hosts file trick, with no change to this behavior.

It's probably worth noting too that this is *not* intermittent behavior -- this happens with every single email that is sent to the frontend (TCP/25) receive connector.

I've also confirmed that my components are all active:

[PS] C:\Windows\system32>Get-ServerComponentState excambio

Server                                  Component                               State
------                                  ---------                               -----
excambio.jonheese.local                 ServerWideOffline                       Active
excambio.jonheese.local                 HubTransport                            Active
excambio.jonheese.local                 FrontendTransport                       Active
excambio.jonheese.local                 Monitoring                              Active
excambio.jonheese.local                 RecoveryActionsEnabled                  Active
excambio.jonheese.local                 AutoDiscoverProxy                       Active
excambio.jonheese.local                 ActiveSyncProxy                         Active
excambio.jonheese.local                 EcpProxy                                Active
excambio.jonheese.local                 EwsProxy                                Active
excambio.jonheese.local                 ImapProxy                               Active
excambio.jonheese.local                 OabProxy                                Active
excambio.jonheese.local                 OwaProxy                                Active
excambio.jonheese.local                 PopProxy                                Active
excambio.jonheese.local                 PushNotificationsProxy                  Active
excambio.jonheese.local                 RpsProxy                                Active
excambio.jonheese.local                 RwsProxy                                Active
excambio.jonheese.local                 RpcProxy                                Active
excambio.jonheese.local                 UMCallRouter                            Active
excambio.jonheese.local                 XropProxy                               Active
excambio.jonheese.local                 HttpProxyAvailabilityGroup              Active
excambio.jonheese.local                 ForwardSyncDaemon                       Active
excambio.jonheese.local                 ProvisioningRps                         Active
excambio.jonheese.local                 MapiProxy                               Active
excambio.jonheese.local                 EdgeTransport                           Active
excambio.jonheese.local                 HighAvailability                        Active
excambio.jonheese.local                 SharedCache                             Active

Any assistance at this point is greatly appreciated.  I've spent 3-4 days just getting this system up and importing 37GB of mailboxes to it -- reinstalling at this point is not something I'm looking forward to doing (not to mention the fact that all incoming email is just queuing up at a backup MX while I'm waiting for this to go live...).  Thanks!

Regards,

Jon Heese

• Edited by Jon Heese (Work) Saturday, March 21, 2015 5:27 PM

Hello all,

First, I apologize for the long delay in my reply -- my wife gave birth to our second child in the intervening days, so I haven't had much time to get back to this project lately.

So, I have resolved this issue, and it was indeed the event logs that led me to the problem.  I've spent way too much time recently administering Linux/UNIX servers -- everything's in the log files on disk! -- I totally forgot to even check the event logs.

First, things it wasn't:

1. I had already (before posting this thread) found the myriad threads suggesting to use the local hosts file to point the Exchange server at its own IP -- that specific variant of this issue wasn't indicated by the log files, and that didn't change the behavior at all.

2. Running netstat -aon indicated that the ports 25 and 2525 were indeed being bound (on 0.0.0.0) by processes (more on that in a second).

3. There is no antivirus software installed on this server (yet).

4. The hard disks on this server had plenty (10+GB) of free space, so that wasn't it.

The event logs led me to discover that edgetransport.exe was crashing and restarting every few minutes (which explains why I could sometimes telnet to 2525 and sometimes could not, but never got any response from anything sent).

As to why edgetransport.exe was crashing, I did a little more Googling on that, and discovered someone on reddit post a similar issue which was resolved by removing a buggy Transport Agent.  That's when I realized that I had installed the CatchAllAgent Transport Agent on this server right around the time this issue started occurring (shame on me for not realizing that earlier!)...

When I disabled that Transport Agent, mail would flow without issue.  Without going into too much detail, the issue with the Transport Agent turned out to be that the installs .dll files associated with it had "streams" attached to them (because the files were copied from another computer over the network) that caused Windows to be unable to load them properly.  I downloaded streams.exe from SysInternals, deleted the streams, and everything started working properly again.

Thanks again to all who replied.

Regards,
Jon Heee

• Marked as answer by Jon Heese (Work) 12 hours 15 minutes ago