Identifying internal origin of spam
I have a small company network with Exchange Server 2003 and clients running Outlook 2003, 2007 and 2010. We've had repeat occurrences of spam originating from our network. The other day, for example, there were over 6,000 spam messages queued under the
server's SMTP connector. We ran antivirus and malware scans on the server and all client computers. Malware was detected on one computer and cleaned, but today the same spam has reappeared in the queue. At this point all computers are scanning
as clean, so I need some other way to determine which computer is sending the spam.
By looking at the properties of a spam message in the queue, how can I determine which computer it originated from? The sender is forged, so that is no help. Can I enable some level of logging that will identify which client is sending the
forged messages?
October 2nd, 2010 7:51pm
On Sat, 2 Oct 2010 23:47:10 +0000, Richard Koett wrote:
>
>
>I have a small company network with Exchange Server 2003 and clients running Outlook 2003, 2007 and 2010. We've had repeat occurrences of spam originating from our network. The other day, for example, there were over 6,000 spam messages queued under the
server's SMTP connector. We ran antivirus and malware scans on the server and all client computers. Malware was detected on one computer and cleaned, but today the same spam has reappeared in the queue. At this point all computers are scanning as clean, so
I need some other way to determine which computer is sending the spam.
>
>By looking at the properties of a spam message in the queue, how can I determine which computer it originated from? The sender is forged, so that is no help. Can I enable some level of logging that will identify which client is sending the forged messages?
If the e-mail client that sent the message is using SMTP then the
"Received:" headers in the message will point out the IP address.
If the client is using MAPI/RPC, and you have one of those messages
then the message tracking logs might be you best shot. Look for the
messages that have that subject.
Because the "From:" header isn't correct I'm guessing it's a SMTP
client. If that's the case then shut off the use of your SMTP server
to all IP addresses except the ones belonging to your Exchange
servers. That'll stop it.
---
Rich Matheisen
MCSE+I, Exchange MVP
--- Rich Matheisen MCSE+I, Exchange MVP
Free Windows Admin Tool Kit Click here and download it now
October 2nd, 2010 10:39pm
I should be clear that when I referred to the sender being forged, I was searching for messages in the queue in Exchange System Manager, right-clicking the message and selecting "Properties", which lists a field labelled "Sender:". I'm
not sure if this is exactly equivalent to the "From:" field in the message header. Since I deleted the previous mass of messages from the queue, I can't examine those particular messages now.
I do have a single suspicious looking message queued right now, however, that may be of interest. (It is stuck in the queue because the upstream smarthost is currently blocking our outbound e-mail).
From ESM, under SmallBusinessSMTP Connector, the message properties are as follows:
Message ID: < TSI-DCBbKP7Ie8tlMaz0000097c@our.domain.ca >
Sender: "www.free.fr" < visaeurope@service.fr >
Subject: <subject is hidden>
Priority: Normal
Message size (bytes): 5,730
Number of body recipients: N/A
Recipients: Envelope Recipients;
SMTP:uhk@hotmail.fr;
Time of submission: 03/10/2010 11:31 AM
Time received by server: 03/10/2010 11:31 AM
Time expires: 03/10/2010 11:31 AM
Delivery failures: 2
Status: Retry
The file C:\Program Files\Exchsrvr\TSI-DC.log\20101003.log contains corresponding entries like:
# Message Tracking Log File
# Exchange System Attendant Version 6.5.7638.1
# Date Time client-ip Client-hostname Partner-Name Server-hostname server-IP Recipient-Address Event-ID MSGID Priority Recipient-Report-Status total-bytes Number-Recipients Origination-Time Encryption service-Version Linked-MSGID Message-Subject Sender-Address
2010-10-3 18:31:15 GMT 219.95.59.126 7w7uy - TSI-DC 192.168.0.254 uhk@hotmail.fr 1019 TSI-DCBbKP7Ie8tlMaz0000097c@our.domain.ca 3 0 5730 1 2010-10-3 18:31:13 GMT 0 Version: 6.0.3790.3959 - - visaeurope@service.fr -
The actual message in C:\Program Files\Exchsrvr\Mailroot\vsi 1\Queue\NTFS_28fb6df701cb6329000010cf.EML begins with:
Received: from 7w7uy ([219.95.59.126]) by our.server.ca with Microsoft SMTPSVC(6.0.3790.3959); Sun, 3 Oct 2010 11:31:14 -0700
From: "www.free.fr" <visaeurope@service.fr>
Subject: =?ISO-8859-1?Q?L'acc=E8s?= a votre compte Free.fr est totalement=?ISO-8859-1?Q?restaur=E8?=
To: uhk@hotmail.fr
Content-Type: multipart/alternative;
boundary="=_NextPart_2rfkindysadvnqw3nerasdf";
charset="US-ASCII"
MIME-Version: 1.0
Reply-To: visaeurope@service.fr
Date: Mon, 4 Oct 2010 02:35:16 +0800
X-Priority: 3
Return-Path: visaeurope@service.fr
Message-ID: <TSI-DCBbKP7Ie8tlMaz0000097c@our.domain.ca>
X-OriginalArrivalTime: 03 Oct 2010 18:31:14.0725 (UTC)
FILETIME=[2942E550:01CB6329]
This is a multi-part message in MIME format
--=_NextPart_2rfkindysadvnqw3nerasdf
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable
--=_NextPart_2rfkindysadvnqw3nerasdf
Content-Type: text/html
Content-Transfer-Encoding: quoted-printable
<head>
<cleaned_tagcontent=3D"fr" http-equiv=3D"Content-Language" />
<cleaned_tagcontent=3D"text/html; charset=3Dutf-8" http-equiv=3D"Content=
-Type" />
</head>
My interpretation of the preceeding information would be that a message was queued for delivery from 219.95.59.126 addressed to hotmail.fr, which would indicate an open relay. However, the SMTP connector properties do not appear to allow such
relaying. Furthermore, I tested the server manually as follows:
telnet our.server.ca 25
220 our.server.ca Microsoft ESMTP MAIL Service, Version: 6.0.3790.3959 ready at Sun, 3 Oct 2010 14:06:40 -0700
helo other.host.ca
250 our.server.ca Hello [x.x.x.x]
mail from:visaeurope@service.fr
250 2.1.0 visaeurope@service.fr....Sender OK
rcpt to:uhk@hotmail.com
550 5.7.1 Unable to relay for uhk@hotmail.com
This seems to confirm that open relaying is not allowed, so my interpretaion of the data must not be correct. Perhaps this single piece of mail is some sort of backscatter? The address 219.95.59.126 reverse maps to "server.petersen.local", which also seems
bogus.
I would appreciate any help in determining how these messages are originating. Our smarthost will not unblock our outbound mail until we can provide details of preventative measures.
October 3rd, 2010 6:04pm
On Sun, 3 Oct 2010 22:01:56 +0000, Richard Koett wrote:
>
>
>I should be clear that when I referred to the sender being forged, I was searching for messages in the queue in Exchange System Manager, right-clicking the message and selecting "Properties", which lists a field labelled "Sender:". I'm not sure if this
is exactly equivalent to the "From:" field in the message header. Since I deleted the previous mass of messages from the queue, I can't examine those particular messages now.
[ snip ]
>The actual message in C:\Program Files\Exchsrvr\Mailroot\vsi 1\Queue\NTFS_28fb6df701cb6329000010cf.EML begins with: Received: from 7w7uy ([219.95.59.126]) by our.server.ca with Microsoft SMTPSVC(6.0.3790.3959); Sun, 3 Oct 2010 11:31:14 -0700
[snip ]
>y interpretation of the preceeding information would be that a message was queued for delivery from 219.95.59.126 addressed to hotmail.fr, which would indicate an open relay. However, the SMTP connector properties do not appear to allow such relaying.
Furthermore, I tested the server manually as follows:
>
>telnet our.server.ca 25 220 our.server.ca Microsoft ESMTP MAIL Service, Version: 6.0.3790.3959 ready at Sun, 3 Oct 2010 14:06:40 -0700 helo other.host.ca 250 our.server.ca Hello [x.x.x.x] mail from:visaeurope@service.fr 250 2.1.0 visaeurope@service.fr....Sender
OK rcpt to:uhk@hotmail.com 550 5.7.1 Unable to relay for uhk@hotmail.com This seems to confirm that open relaying is not allowed, so my interpretaion of the data must not be correct. Perhaps this single piece of mail is some sort of backscatter? The address
219.95.59.126 reverse maps to "server.petersen.local", which also seems bogus.
>
>I would appreciate any help in determining how these messages are originating. Our smarthost will not unblock our outbound mail until we can provide details of preventative measures.
Given that the connectioncame directly from a machiner outside your
organization I'd guess that you do allow the use of your server as a
SMTP relay for some purposes. Perhaps you allow authenticated users to
relay? If you do then maybe you have an easily guessable password on
an account (or no password at all)? Try disaling all SMTP relay and
see if your problem stops.
Authenticated sessions in the Exchange 2003 SMTP protocol log are easy
to see. Check your log for connectins from that IP address and see
what's going on.
The IP address 219.95.59.126 address is in Kuala Lempur, Malaysia:
http://www.trustedsource.org/query/219.95.59.126
http://www.senderbase.org/senderbase_queries/detailip?search_string=219.95.59.126
Why your DNS query returned an name in a .local TLD I can only guess.
Does your DNS do any sort of wild-card lookups or return addresses in
our domain for queries that produce no answer?
---
Rich Matheisen
MCSE+I, Exchange MVP
--- Rich Matheisen MCSE+I, Exchange MVP
Free Windows Admin Tool Kit Click here and download it now
October 3rd, 2010 10:49pm
I think you may be on the right track, as the SMTP connector does permit relaying from authenticated computers. Also, although I no longer have any of the large wave of spam messages to examine, there are traces in the log files. I believe
the messages started at 06:58 PDT (13:58 GMT) on October 2nd. The file C:\Program Files\Exchsrvr\TSI-DC.log\20101002.log contains entries like:
2010-10-2 13:58:50 GMT 75.147.129.61 3lyw4 - TSI-DC 192.168.0.254 paris_hena11@yahoo.fr 1019
TSI-DCRf6Ci6nx8dyDd00000295@our.server.ca 3 0 3262 1 2010-10-2 13:58:50 GMT 0 Version: 6.0.3790.3959 - - service@freebox.net -
2010-10-2 13:58:50 GMT 75.147.129.61 3lyw4 - TSI-DC 192.168.0.254 aminos_vp@hotmail.com 1019
TSI-DCRf6Ci6nx8dyDd00000294@our.server.ca 3 0 3262 1 2010-10-2 13:58:50 GMT 0 Version: 6.0.3790.3959 - - service@freebox.net -
In the server's security event log, there are logon events about 1 second prior:
Event Type: Success Audit
Event Source: Security
Event Category: Account Logon
Event ID: 680
Date: 02/10/2010
Time: 6:58:49 AM
User: DOMAIN\Usernname
Computer: TSI-DC
Description:
Logon attempt by: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0
Logon account: Username
Source Workstation: TSI-DC
Error Code: 0x0
Event Type: Success Audit
Event Source: Security
Event Category: Logon/Logoff
Event ID: 552
Date: 02/10/2010
Time: 6:58:49 AM
User: NT AUTHORITY\SYSTEM
Computer: TSI-DC
Description:
Logon attempt using explicit credentials:
Logged on user:
User Name: TSI-DC$
Domain: DOMAIN
Logon ID: (0x0,0x3E7)
Logon GUID: {4afb4ad3-dd5f-bb05-7432-1195c1234ef8}
User whose credentials were used:
Target User Name: Username
Target Domain: DOMAIN
Target Logon GUID: -
Target Server Name: localhost
Target Server Info: localhost
Caller Process ID: 1928
Source Network Address: -
Source Port: -
Event Type: Success Audit
Event Source: Security
Event Category: Logon/Logoff
Event ID: 540
Date: 02/10/2010
Time: 6:58:49 AM
User: DOMAIN\Username
Computer: TSI-DC
Description:
Successful Network Logon:
User Name: Username
Domain: DOMAIN
Logon ID: (0x0,0xF90A1D0)
Logon Type: 3
Logon Process: Advapi
Authentication Package: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0
Workstation Name: TSI-DC
Logon GUID: -
Caller User Name: TSI-DC$
Caller Domain: DOMAIN
Caller Logon ID: (0x0,0x3E7)
Caller Process ID: 1928
Transited Services: -
Source Network Address: -
Source Port: -
The process ID 1928 corresponds to inetinfo.exe, so I guess that's consistent with someone authenticating prior to relaying mail?
The IP address 75.147.129.61 belongs to Comcast in the US, and also reverse maps to server.petersen.local via more than one DNS server I tried. I'm not sure what to make of that, but it's interesting that 219.95.59.129 also mapped to that name.
In any case, since I now have the username used to authenticate, I'll be changing the password for that account, but would also like to tighten up relay restrictions in general. The Default SMTP Virtual Server Properties show:
Relay Restrictions
Select which computers may relay through this virtual server:
* Only the list below
Access
IP Address (Mask) / Domain Name
Granted 192.168.0.254 (255.255.255.0)
Granted 127.0.0.1
* All all computers which successfully authenticate to relay, regardless of the list above.
I note that the right to relay is granted to computers that authenticate, not
users. I've now unchecked that last option, but considering the logon events detailed above, it looks to me like the logon is taking place locally on the server itself (192.168.0.254). Will this option have any effect
on users who authenticate via inetinfo.exe?
Thanks again for your advice.
October 4th, 2010 5:38am
On Mon, 4 Oct 2010 09:36:07 +0000, Richard Koett wrote:
>
>
>I think you may be on the right track, as the SMTP connector does permit relaying from authenticated computers. Also, although I no longer have any of the large wave of spam messages to examine, there are traces in the log files. I believe the messages
started at 06:58 PDT (13:58 GMT) on October 2nd. The file C:\Program Files\Exchsrvr\TSI-DC.log\20101002.log contains entries like:
>
>2010-10-2 13:58:50 GMT 75.147.129.61 3lyw4 - TSI-DC 192.168.0.254 paris_hena11@yahoo.fr 1019 TSI-DCRf6Ci6nx8dyDd00000295@our.server.ca 3 0 3262 1 2010-10-2 13:58:50 GMT 0 Version: 6.0.3790.3959 - - service@freebox.net -
>
>2010-10-2 13:58:50 GMT 75.147.129.61 3lyw4 - TSI-DC 192.168.0.254 aminos_vp@hotmail.com 1019 TSI-DCRf6Ci6nx8dyDd00000294@our.server.ca 3 0 3262 1 2010-10-2 13:58:50 GMT 0 Version: 6.0.3790.3959 - - service@freebox.net -
Those are from the message tracking log files. I'd start in the SMTP
protocol logs and look for the rows that have the IP address in the
"Received" header. The look for authentication commands in that subset
of data. IIRC, with Exchange 2003 the base64-encoded account and
password are stored in the log file so it'll be easy to identify which
account has been compromised.
I'd start by setting secure passwords on the administrator, admin, sa,
webmaster, hostmaster, abc, www, data, server, backup, test, root,
IWAM_machine and IUSR_machine (be careful with these), etc. accounts
-- the ones that are commonly used on lots of machines. And make sure
that the Guest account isn't enabled!
To find those authentication events in the application log, set the
diagnostics logging level on the MSExchangeTransport \ SMTP Protocol
to "maximum" and then watch for EventID 1708. You'll see the machine
name (if that's interesting, which it probably isn't) and the account
name used to authenticate.
>In the server's security event log, there are logon events about 1 second prior:
It's a lot easier to find this with the SMTP protocol and the
MsexchangeTransport events than it is with the security log. But if
you have the account name now's the time to make sure the password's
change to something more secure. It may also be time to rethink your
domain password polcy.
[ snip ]
>The process ID 1928 corresponds to inetinfo.exe, so I guess that's consistent with someone authenticating prior to relaying mail?
Or using OWA or ActiveSync or RPC-over-HTTPS or POP or IMAP or FTP or
anything else related to protocols managed by that executable.
>The IP address 75.147.129.61 belongs to Comcast in the US, and also reverse maps to server.petersen.local via more than one DNS server I tried. I'm not sure what to make of that, but it's interesting that 219.95.59.129 also mapped to that name.
You can never tell what other people allow in their DNS.
>In any case, since I now have the username used to authenticate, I'll be changing the password for that account, but would also like to tighten up relay restrictions in general. The Default SMTP Virtual Server Properties show:
>
>Relay Restrictions Select which computers may relay through this virtual server:
>* Only the list below
>Access
>IP Address (Mask) / Domain Name Granted 192.168.0.254 (255.255.255.0) Granted 127.0.0.1
192.168.0.254 is the wrong IP address. You want 192.168.0.0 if you
want the network 192.168.0.0/16 to relay. And I think that's not such
a good policy, either. Do you have any wireless access points on your
network? Are they on the 192.168.0.0/16 network?
Why is 127.0.0.1 allowed to relay?
>* All all computers which successfully authenticate to relay, regardless of the list above.
Turn that off.
>I note that the right to relay is granted to computers that authenticate, not users.
SMTP is usually a protocol used by servers, not individual users.
Exchange 2007 added port 587 in accordance with new "best practices".
Port 587 is a "submission" port to be used by individuals to submit
e-mail for delivery. Port 25 is a "delivery" port used by servers to
send e-mail.
>I've now unchecked that last option, but considering the logon events detailed above, it looks to me like the logon is taking place locally on the server itself (192.168.0.254). Will this option have any effect on users who authenticate via inetinfo.exe?
For what reason are they using SMTP to send e-mail? They don't need it
to send e-mail using Outlook or OWA.
---
Rich Matheisen
MCSE+I, Exchange MVP
--- Rich Matheisen MCSE+I, Exchange MVP
Free Windows Admin Tool Kit Click here and download it now
October 4th, 2010 10:28pm
Thanks for your thoughtful answers. Since I'm not the person who originally setup this server, I'm not sure about the reasons behind some of those settings. For example, 192.168.0.254 is the address of the server itself. I think the server's own address
and the loopback address are allowed to relay by default, since I've seen them included on other SBS servers. Perhaps this is needed when using a smarthost instead of DNS to route mail?
You are right that the password policy could have been better. All user passwords are now being changed to something stronger.
One final question: What is the best way to configure SMTP logging? There appear to be two choices:
I found the settings you referred to in ESM by drilling down to SERVER > Properties > Diagnostics Logging, where I can select MSExchangeTransport and set SMTP Protocol logging level to maximum. If I understand correctly, this will log information under
Event Viewer > Application.
Another option I found is to drill down to SERVER > Protocols > SMTP > Default SMTP Virtual Server > Properties > General, where there is a check box for "Enable Logging", and a set of "Extended logging options" under the advanced
properties for that. If I understand correctly, this will create log files under %SystemRoot%\System32\LogFiles.
Do you see any advantages or disadvantages to choosing one method over the other?
October 5th, 2010 2:05am
On Tue, 5 Oct 2010 06:03:19 +0000, Richard Koett wrote:
>
>
>Thanks for your thoughtful answers. Since I'm not the person who originally setup this server, I'm not sure about the reasons behind some of those settings. For example, 192.168.0.254 is the address of the server itself.
If that's true then the network mask should be 255.255.255.255 (i.e. a
single 32-bit address), not 255.255.255.0 (i.e. the 254 hosts on the
192.168.0.0\24 network).
>I think the server's own address and the loopback address are allowed to relay by default, since I've seen them included on other SBS servers. Perhaps this is needed when using a smarthost instead of DNS to route mail?
The only reason for including any address is if that address will be
allowed to use your machine as a SMTP relay. If you run other
applications on the machine that need a SMTP relay then using just the
servers IP address (not a network) is sufficient. Using 127.0.0.1 is
most likely unnecessary.
>You are right that the password policy could have been better. All user passwords are now being changed to something stronger.
>
>One final question: What is the best way to configure SMTP logging? There appear to be two choices:
>
>I found the settings you referred to in ESM by drilling down to SERVER > Properties > Diagnostics Logging, where I can select MSExchangeTransport and set SMTP Protocol logging level to maximum. If I understand correctly, this will log information under
Event Viewer > Application.
That's correct. But that not a log of SMTP conversations.
>Another option I found is to drill down to SERVER > Protocols > SMTP > Default SMTP Virtual Server > Properties > General, where there is a check box for "Enable Logging", and a set of "Extended logging options" under the advanced properties for that.
If I understand correctly, this will create log files under %SystemRoot%\System32\LogFiles.
Also correct.
>Do you see any advantages or disadvantages to choosing one method over the other?
The stuff that goes into to the event logs isn't usually used except
for diagnostic work. Leaving the logging level too high on too many
objects will rapidly fill the event logs and drive you nuts because by
the time you realize you need to look for a problem the information's
already been overwritten.
The protocol logs are invaluable when trying to puzzle out why mail
wasn't deivered (in either direction) or the origin of an e-mail. The
log files can be quite large (I usually set them to create a new log
every hour) and then can take up a lot of space. But they're just text
files and they're easy to manage (just delete the ones older than,
say, 14 days).
Both logs server their own purpose, but for the day-to-day "what
happened to that e-mail from customerX" sort of questions I'd take the
protocol log over the application log any day -- at least as a
starting point.
---
Rich Matheisen
MCSE+I, Exchange MVP
--- Rich Matheisen MCSE+I, Exchange MVP
Free Windows Admin Tool Kit Click here and download it now
October 5th, 2010 7:57pm