SMTP from Exchange server getting dropped
I've started getting log messages my ASA firewall, similar to the following:
Dec 23 2010 09:14:47:
%ASA-4-108004: ESMTP Classification: Dropped
connection for ESMTP Request
from Inside:172.16.173.31 (k9.company.com)
/12964 to Outside:206.190.54.127 (mta-v1.mail.vip.re4.yahoo.com)
/25; matched Class 4: header
line length gt 998
The K9 server is the Exchange server responsible for processing outbound SMTP. I realize from research that the 'header line length gt 998' is an issue with messages being outside the SMTP RFC, but my question is how can I track down where these
messages are originating? They have to be coming from inside the exchange server someplace.
December 23rd, 2010 9:36am
Sorry - forgot to mention - Exchange 2003.
Free Windows Admin Tool Kit Click here and download it now
December 23rd, 2010 9:38am
Well, I wouldnt run that SMTP Inspection stuff myself as it always seems to cause problems, but if you enable SMTP protocol logging and/or increase diags on the Transport properties in ESM, it should tell which messages are being dropped.
http://technet.microsoft.com/en-us/library/bb124469(EXCHG.65).aspx
December 23rd, 2010 9:44am
It looks like NCSA Common Log File Format was already on, so I went in and checked the logs for an example:
74.125.43.27 - OutboundConnectionCommand [23/Dec/2010:10:09:07 -0500] EHLO -?k9.company.com SMTP 0 4
74.125.43.27 - OutboundConnectionResponse [23/Dec/2010:10:09:07 -0500] - -?250-mx.google.com at your service, [72.240.36.42] SMTP 0
49
74.125.43.27 - OutboundConnectionCommand [23/Dec/2010:10:09:07 -0500] MAIL -?FROM:<user@company.com> SIZE=181210 SMTP 0 4
74.125.43.27 - OutboundConnectionResponse [23/Dec/2010:10:09:08 -0500] - -?250 2.1.0 OK l10si24293017bkw.10 SMTP 0 32
74.125.43.27 - OutboundConnectionCommand [23/Dec/2010:10:09:08 -0500] RCPT -?TO:<user@gmail.com> SMTP 0 4
74.125.43.27 - OutboundConnectionResponse [23/Dec/2010:10:09:08 -0500] - -?250 2.1.5 OK l10si24293017bkw.10 SMTP 0 32
74.125.43.27 - OutboundConnectionCommand [23/Dec/2010:10:09:08 -0500] DATA - SMTP 0 4
74.125.43.27 - OutboundConnectionResponse [23/Dec/2010:10:09:08 -0500] - -?354 Go ahead l10si24293017bkw.10 SMTP 0 33
I don't see anything here that would indicate a problem, other than there's nothing after the "Go ahead". This message matches the time stamp and server address of one of the connections the ASA logged as a drop. Should I turn on a
different type of logging?
Free Windows Admin Tool Kit Click here and download it now
December 23rd, 2010 10:47am
I would look for messages to yahoo:
ESMTP Classification: Dropped connection
for ESMTP Request from
Inside:172.16.173.31 (k9.company.com) /12964
to Outside:206.190.54.127 (mta-v1.mail.vip.re4.yahoo.com)
When the connection is dropped, there should be a 5xx error in the log and a NDR returned to the sender. I am assuming that the ASA will handle the connection drop that way so the senders know it never went through.
December 23rd, 2010 10:56am
I don't think it does, because there are no drops in the log file. I do see some messages to yahoo and gmail in the mail queue with a retry status on them. I'm wondering if exchange is eventually delivering a NDR to the user and he's just never
reporting it to us.
Free Windows Admin Tool Kit Click here and download it now
December 23rd, 2010 11:02am
Those could be the ones. Look for 4xx SMTP responses as well. What does the retry status show in the queues?
December 23rd, 2010 11:08am
On Thu, 23 Dec 2010 14:31:54 +0000, jmickens wrote:
>
>
>I've started getting log messages my ASA firewall, similar to the following:
>
>Dec 23 2010 09:14:47: %ASA-4-108004: ESMTP Classification: Dropped connection for ESMTP Request from Inside:172.16.173.31 (k9.company.com) /12964 to Outside:206.190.54.127 (mta-v1.mail.vip.re4.yahoo.com) /25; matched Class 4: header line length gt 998
The limit s/b 1000 (including the CRLF pair at the end of the line).
Does that 998 value represent the TOTAL line length or just the part
without the CRLF?
>The K9 server is the Exchange server responsible for processing outbound SMTP. I realize from research that the 'header line length gt 998' is an issue with messages being outside the SMTP RFC, but my question is how can I track down where these messages
are originating? They have to be coming from inside the exchange server someplace.
Your SMTP protocol logs will show you the SMTP conversation.
---
Rich Matheisen
MCSE+I, Exchange MVP
--- Rich Matheisen MCSE+I, Exchange MVP
Free Windows Admin Tool Kit Click here and download it now
December 23rd, 2010 11:31am
Those could be the ones. Look for 4xx SMTP responses as well. What does the retry status show in the queues?
I don't see any 5xx or 4xx errors in the logs. At least not associated with any of these dropped messages. Status of the messages in the queue just says "Retry", though the retry count at this point is 55. If the firewall is dropping
the packet, then SMTP
December 23rd, 2010 11:52am
Those could be the ones. Look for 4xx SMTP responses as well. What does the retry status show in the queues?
I don't see any 5xx or 4xx errors in the logs. At least not associated with any of these dropped messages. Status of the messages in the queue just says "Retry", though the retry count at this point is 55. If the firewall is dropping
the packet, then SMTP just doesn't see a reply, I guess.
Free Windows Admin Tool Kit Click here and download it now
December 23rd, 2010 11:52am
The limit s/b 1000 (including the CRLF pair at the end of the line).
Does that 998 value represent the TOTAL line length or just the part
without the CRLF?
Sorry Rich, I don't know. Whatever the default setting the ASA has for SMTP inspection is what is in place.
December 23rd, 2010 12:46pm
I would think about disabling the SMTP inspection stuff and testing.
Free Windows Admin Tool Kit Click here and download it now
December 23rd, 2010 3:39pm
On Thu, 23 Dec 2010 15:42:46 +0000, jmickens wrote:
>
>
>It looks like NCSA Common Log File Format was already on, so I went in and checked the logs for an example:
>
>
>
>74.125.43.27 - OutboundConnectionCommand [23/Dec/2010:10:09:07 -0500] EHLO -?k9.company.com SMTP 0 4 74.125.43.27 - OutboundConnectionResponse [23/Dec/2010:10:09:07 -0500] - -?250-mx.google.com at your service, [72.240.36.42] SMTP 0 49 74.125.43.27 -
OutboundConnectionCommand [23/Dec/2010:10:09:07 -0500] MAIL -?FROM:<user@company.com> SIZE=181210 SMTP 0 4 74.125.43.27 - OutboundConnectionResponse [23/Dec/2010:10:09:08 -0500] - -?250 2.1.0 OK l10si24293017bkw.10 SMTP 0 32 74.125.43.27 - OutboundConnectionCommand
[23/Dec/2010:10:09:08 -0500] RCPT -?TO:<user@gmail.com> SMTP 0 4 74.125.43.27 - OutboundConnectionResponse [23/Dec/2010:10:09:08 -0500] - -?250 2.1.5 OK l10si24293017bkw.10 SMTP 0 32 74.125.43.27 - OutboundConnectionCommand [23/Dec/2010:10:09:08 -0500] DATA
- SMTP 0 4 74.125.43.27 - OutboundConnectionResponse
>[23/Dec/2010:10:09:08 -0500] - -?354 Go ahead l10si24293017bkw.10 SMTP 0 33
The message is supposedly from someone in your company. Have you asked
them how they sent the message? Were they using Outlook with a RPC
connection (or OWA), or were they using SMTP to send the message (and,
perhaps, some e-mail client other than Outlook)?
If they were using SMTP to send the message they it's probably the
e-mail client that's at fault.
>I don't see anything here that would indicate a problem, other than there's nothing after the "Go ahead".
Well, that *is* the problem! :-)
If the firewall's just dropping the connection then, eventually,
there'll be a DSN sent to the message originator informing them of a
delayed delivery and, after a longer wait, a NDR informing them that
the message couldn't be delivered.
>This message matches the time stamp and server address of one of the connections the ASA logged as a drop. Should I turn on a different type of logging?
I'd switch to the W3C Extended log format -- and then check ALL the
boxes on the "Advanced" tab to collect the most information possible.
Of course, that's not going to be much help here. To see what's going
on you'll need a network monitor (WireShark is a good one). Once you
identify which session was the one that failed you can isolate the
packets associated with it and examine just those.
---
Rich Matheisen
MCSE+I, Exchange MVP
--- Rich Matheisen MCSE+I, Exchange MVP
December 23rd, 2010 3:44pm
On Thu, 23 Dec 2010 15:52:14 +0000, AndyD_ wrote:
>
>
>I would look for messages to yahoo:
>
>ESMTP Classification: Dropped connection for ESMTP Request from Inside:172.16.173.31 (k9.company.com) /12964 to Outside:206.190.54.127 (mta-v1.mail.vip.re4.yahoo.com)
>
>When the connection is dropped, there should be a 5xx error in the log and a NDR returned to the sender.
Well, maybe after two days. ;-) They should receive a DSN informing
them of the delayed delivery in a shorter time.
>I am assuming that the ASA will handle the connection drop that way so the senders know it never went through.
The dropped connection will just put the message into the retry state.
---
Rich Matheisen
MCSE+I, Exchange MVP
--- Rich Matheisen MCSE+I, Exchange MVP
Free Windows Admin Tool Kit Click here and download it now
December 23rd, 2010 3:46pm
Narrowing down who exactly is sending out the bad messages was how this whole thread started. In going through the log files, all the examples I've chosen have all come from one person (so far). Knowing where the log files are has been half the
battle. Once I found that out I figured out who they were coming from. I tried tracking the messages through the Exchange system (message tracking center), but I'm getting a lot of "Message transferred to **************************************"
errors.
I'm going to have to go back to the user who's email address these are originating from. They're either automated messages coming from some SMTP engine using his From address, or possibly they're being rule forwarded to some users' home email addresses,
because they all seem to be going to the same group of folks.
Thanks to you both for steering me in the right direction. At least now I know where to look.
December 23rd, 2010 4:02pm
On Thu, 23 Dec 2010 20:57:35 +0000, jmickens wrote:
>
>
>Narrowing down who exactly is sending out the bad messages was how this whole thread started. In going through the log files, all the examples I've chosen have all come from one person (so far). Knowing where the log files are has been half the battle.
Once I found that out I figured out who they were coming from. I tried tracking the messages through the Exchange system (message tracking center), but I'm getting a lot of "Message transferred to **************************************" errors.
Those are probably just messages sent to other e-mail systems outside
your Exchange organization -- or to Exchange 2007/2010 servers within
your organization.
>I'm going to have to go back to the user who's email address these are originating from. They're either automated messages coming from some SMTP engine using his From address, or possibly they're being rule forwarded to some users' home email addresses,
because they all seem to be going to the same group of folks.
Easy enough to find -- just look for the MAIL FROM address in the SMTP
protocol logs that belongs to that person that comes from an IP
address that's outside the addresses used by any Exchange servers.
>
>Thanks to you both for steering me in the right direction. At least now I know where to look.
---
Rich Matheisen
MCSE+I, Exchange MVP
--- Rich Matheisen MCSE+I, Exchange MVP
Free Windows Admin Tool Kit Click here and download it now
December 23rd, 2010 4:51pm
Narrowing down who exactly is sending out the bad messages was how this whole thread started. In going through the log files, all the examples I've chosen have all come from one person (so far). Knowing where the log files are has been half the
battle. Once I found that out I figured out who they were coming from. I tried tracking the messages through the Exchange system (message tracking center), but I'm getting a lot of "Message transferred to **************************************" errors.
I'm going to have to go back to the user who's email address these are originating from. They're either automated messages coming from some SMTP engine using his From address, or possibly they're being rule forwarded to some users' home email addresses,
because they all seem to be going to the same group of folks.
Thanks to you both for steering me in the right direction. At least now I know where to look.
Those **************************************" in the logs are to be expected when using the Cisco SMTP inspection stuff. :)
December 23rd, 2010 4:52pm
On Thu, 23 Dec 2010 21:48:36 +0000, AndyD_ wrote:
>
>
>Narrowing down who exactly is sending out the bad messages was how this whole thread started. In going through the log files, all the examples I've chosen have all come from one person (so far). Knowing where the log files are has been half the battle.
Once I found that out I figured out who they were coming from. I tried tracking the messages through the Exchange system (message tracking center), but I'm getting a lot of "Message transferred to **************************************" errors.
>
>I'm going to have to go back to the user who's email address these are originating from. They're either automated messages coming from some SMTP engine using his From address, or possibly they're being rule forwarded to some users' home email addresses,
because they all seem to be going to the same group of folks.
>
>Thanks to you both for steering me in the right direction. At least now I know where to look.
>
>Those **************************************" in the logs are to be expected when using the Cisco SMTP inspection stuff. :)
Hmmm . . . I thought he was using "*******" to mean "redacted"! :-/
---
Rich Matheisen
MCSE+I, Exchange MVP
--- Rich Matheisen MCSE+I, Exchange MVP
Free Windows Admin Tool Kit Click here and download it now
December 23rd, 2010 8:13pm
How's the issue now? Have you located the senders?Please remember to click Mark as Answer on the post that helps you, and to click Unmark as Answer if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.
December 27th, 2010 10:53pm
The user was on vacation, so I believe it was an automated process that sent the messages out. I won't be able to track the messages down until next week. I wound up disabling the ESMTP inspection.
Free Windows Admin Tool Kit Click here and download it now
December 28th, 2010 8:43am