Outlook Anywhere has stopped working.
Active Directory = Windows 2003 Server SP2 - W2K3 DFL
Mail server OS = Windows 2008 Server SP2
Exchange 2007 SP2 (since August 12, 2010) - note: no RU installed yet.
Outlook 2007 SP2
Yes... Outlook Anywhere had been working for 16 months (installed the system in April 2009).
This weekend, two users noticed they could not access their email from home using Outlook (Anywhere) on their work laptops.
They could access their mail via OWA.
They do not remember if they could or could not access email via Outlook Anywhere the first weekend after SP2 for Exchange 2007 was installed.
Outlook seems to be working onsite (LAN) with possibly some OAB issues for two users but no complaints elsewhere.
EMC - Exchange Tools - Mail tracking shows that mail reaches their mailbox and once again, they can access via OWA.
When I attempted the reproduce the problem from an external connection (offsite) using Outlook Anywhere, this error message displayed about a minute after I entered my username and password (domainName\Username, then password):
"Microsoft Exchange is unavailable".
If I select Retry, I can enter Outlook but Status is "Offline".
This did work "before". Once again, first time anyone noticed a problem for the last 16 months was this weekend.
I was able to access OWA immediately after the above test and using the same connection. Send and Receive from OWA was successfull.
The following error (seen in Even Viewer) seems to coincide with my attempts to connect via Outlook Anywhere. 4 failed attempts and 4 entries as follows, produced within a minute of the attempt to connect:
EventID 11 CAPI2
Failed extract of third-party root list from auto update cab at: <http://www.download.windowsupdate.com/msdownload/update/v3/static/trustedr/en/authrootstl.cab>
with error: A required certificate is not within its validity period when verifying against the current system clock or the timestamp in the signed file.
Server side
Test-SystemHealth
Displays no errors about certificates however (some drivers are older than two years, the Filter Pack is not installed, Active Server Pages is not detected or allowed, but that's it).
Test-ServiceHealth
All is fine here.
Test-OWAConnectivity
Sometimes passes, sometimes fails (this never happened before SP2 and yes, I right click and run EMS as aministrator).
When it fails, this is the error message:
[PS] C:\>Test-OwaConnectivity -TrustAnySSLCertificate | fl
WARNING: The test was unable to log on to Outlook Web Access because the SSL certificate did not validate. You can force the cmdlet to proceed by re-running it and specifying the ?TrustAnySSLCertificate parameter.
AuthenticationMethod :
ClientAccessServer : MS1.mydomain.local
Scenario : Logon
ScenarioDescription : Log on to Outlook Web Access and verify the response page.
PerformanceCounterName : Logon Latency
Result : Skipped
MailboxServer : MS1.mydomain.local
StartTime : 8/23/2010 12:01:46 PM
Latency : 00:00:00.0156001
SecureAccess : True
Error : The test was unable to log on to Outlook Web Access because the SSL certificate did not validate. You can force the cmdlet to proceed by re-running
it and specifying the -TrustAnySSLCertificate parameter.
UserName : CAS_00xxxxxxxxxxxb
VirtualDirectoryName : owa (Default Web Site)
Url :
https://mail.mydomain.org/owa/
UrlType : Internal
EventType : Warning
Port : 0
ConnectionType : Plaintext
Yet at other times, with no apparent pattern, the test is successful. This was the result a minute or so before the failure above. Same credentials, same logon with right click and "Run as Administrator", same EMS session:
[PS] C:\>Test-OwaConnectivity
ClientAccessServer MailboxServer URL Scenario Result Latency Error
(ms)
------------------ ------------- --- -------- ------
- ------ -----
MS1 MS1
https://mail.mydomain.org/owa Logon Success 62.4
Same phenomenon with..
Test-WebServicesConnectivity
I just tried it - All Success (Pass).
But other times, it fails with a reference to the SSL cert not validating.
******************
EMC BPA Connectivity Test is a Pass (no errors).
EMC BPA Health Test shows same Warnings about old NIC and Storage drivers that the Test-SystemHealth check does. No errors. Nothing about certificates.
What do I do next?
I'm guessing it has something to do with the certificates?
August 23rd, 2010 12:20pm
Here are the ECRA results - only domain name and user name changed, IP address replaced with xx.xx.xx.xx
https://www.testexchangeconnectivity.com/
__________________________________________________
ExRCA is testing RPC/HTTP connectivity.
The RPC/HTTP test failed
.
Test Steps
Attempting to test Autodiscover for testuser1@MyDomain.org
Autodiscover was tested successfully.
Test Steps
ExRCA is attempting each method of contacting the Autodiscover service.
The Autodiscover service was tested successfully.
Test Steps
Attempting to test potential AutoDiscover URL https://MyDomain.org/AutoDiscover/AutoDiscover.xml
Testing of this potential Autodiscover URL failed.
NOTE: WE DO NOT USE THIS ADDRESS - OTHER AUTODISCOVER URL IS OK - SEE BELOW
Test Steps
Attempting to resolve the host name MyDomain.org in DNS.
Host successfully resolved
Additional Details
IP(s) returned: xx.xx.xx.xx
Testing TCP Port 443 on host MyDomain.org to ensure it is listening and open.
The port was opened successfully.
ExRCA is testing the SSL certificate to make sure it's valid.
The SSL certificate failed one or more certificate validation checks.
Test Steps
The certificate name is being validated.
Certificate name validation failed.
Tell me more about this issue and how to resolve it
Additional Details
Host name MyDomain.org does not match any name found on the server certificate E=info@plesk.com, CN=plesk, OU=Plesk, O="SWsoft, Inc.", L=Herndon, S=Virginia, C=US
NOTE: I HAVE NO IDEA WHAT THIS INFO IS ABOUT - THIS IS NOT MY DOMAIN. IT USED TO COME UP IN BPA HEALTH CHECKS - BUT NOT SINCE SP2
Attempting to test potential AutoDiscover URL https://autodiscover.MyDomain.org/AutoDiscover/AutoDiscover.xml
Testing of the Autodiscover URL was successful.
Test Steps
Attempting to resolve the host name autodiscover.MyDomain.org in DNS.
Host successfully resolved
Additional Details
IP(s) returned: xx.xx.xx.xx
Testing TCP Port 443 on host autodiscover.MyDomain.org to ensure it is listening and open.
The port was opened successfully.
ExRCA is testing the SSL certificate to make sure it's valid.
The certificate passed all validation requirements.
Test Steps
The certificate name is being validated.
Successfully validated the certificate name
Additional Details
Found hostname autodiscover.MyDomain.org in Certificate Subject Alternative Name entry
Certificate trust is being validated.
The certificate is trusted and all certificates are present in the chain.
Additional Details
The Certificate chain has be validated up to a trusted root. Root = E=info@valicert.com, CN=http://www.valicert.com/, OU=ValiCert Class 2 Policy Validation Authority, O="ValiCert, Inc.", L=ValiCert Validation Network
The certificate date is being confirmed to ensure the certificate is valid.
Date validation passed. The certificate hasn't expired.
Additional Details
Certificate is valid: NotBefore = 2/25/2010 6:12:11 PM, NotAfter = 3/23/2011 8:07:10 PM"
The IIS configuration is being checked for client certificate authentication.
Client certificate authentication wasn't detected.
Additional Details
Accept/Require Client Certificates not configured.
ExRCA is attempting to send an Autodiscover POST request to potential Autodiscover URLs.
Successfully Retrieved AutoDiscover Settings by sending AutoDiscover POST.
Test Steps
Attempting to Retrieve XML AutoDiscover Response from url https://autodiscover.MyDomain.org/AutoDiscover/AutoDiscover.xml for user
testuser1@MyDomain.org
The Autodiscover XML response was successfully retrieved.
Additional Details
AutoDiscover Account Settings
XML Response:
[...]
Autodiscover settings for Outlook Anywhere are being validated.
Outlook Anywhere Autodiscover Settings validated
Attempting to resolve the host name mail.MyDomain.org in DNS.
Host successfully resolved
Additional Details
IP(s) returned: xx.xx.xx.xx
Testing TCP Port 443 on host mail.MyDomain.org to ensure it is listening and open.
The port was opened successfully.
ExRCA is testing the SSL certificate to make sure it's valid.
The certificate passed all validation requirements.
Test Steps
The certificate name is being validated.
Successfully validated the certificate name
Additional Details
Found hostname mail.MyDomain.org in Certificate Subject Common name
Certificate trust is being validated.
The certificate is trusted and all certificates are present in the chain.
Additional Details
The Certificate chain has be validated up to a trusted root. Root = E=info@valicert.com, CN=http://www.valicert.com/, OU=ValiCert Class 2 Policy Validation Authority, O="ValiCert, Inc.", L=ValiCert Validation Network
The certificate date is being confirmed to ensure the certificate is valid.
Date validation passed. The certificate hasn't expired.
Additional Details
Certificate is valid: NotBefore = 2/25/2010 6:12:11 PM, NotAfter = 3/23/2011 8:07:10 PM"
The IIS configuration is being checked for client certificate authentication.
The test passed with some warnings encountered. Please expand the additional details.
Additional Details
Client Certificate Authentication could not be determined due to an unexpected failure. WinHttpSendRequest failed with error 12002.
Testing Http Authentication Methods for URL https://mail.MyDomain.org/rpc/rpcproxy.dll
The HTTP authentication test failed.
Additional Details
An HTTP 500 response was returned from Unknown
Free Windows Admin Tool Kit Click here and download it now
August 23rd, 2010 2:52pm
I regret that the above is unreadable - font was x-small and when I changed it to small that killed the formatting.
That's a lot to reformat. This seems to be the most pertinent:
---------------------------------------------------
The IIS configuration is being checked for client certificate authentication.
The test passed with some warnings encountered. Please expand the additional details.
Additional Details
Client Certificate Authentication could not be determined due to an unexpected failure. WinHttpSendRequest failed with error 12002.
Testing Http Authentication Methods for URL https://mail.MyDomain.org/rpc/rpcproxy.dll
The HTTP authentication test failed.
Additional Details
An HTTP 500 response was returned from Unknown
August 23rd, 2010 3:22pm
FYI:
ERCA Autodiscover test passed with and without SSL Trust.
Free Windows Admin Tool Kit Click here and download it now
August 23rd, 2010 3:36pm
Outlook /RPC
Doesn't tell us much.
Status is "Connecting" for a Type "Referral" and then "Directory", references the mail server (mailserver.mydomain.tld), then a domain
controller (dc1.mydomain.tld).
No results are displayed. No Pass, no Fail, nothing.
Outlook still displays "Microsoft Exchange is unavailable".
August 23rd, 2010 5:23pm
RPCPing?
(Is there I troubleshooting tool I have not thought of?).
Question:
Is RPCPing encrypted over the Internet (I think not)? Is it safe to send real user credentials?
Or should I create a test user as I did for
https://www.testexchangeconnectivity.com/
Pending your responses, I'm going to try with a test user. Not sure how valid that will be?
Free Windows Admin Tool Kit Click here and download it now
August 23rd, 2010 5:27pm
I tried this on the mailserver itself - just to see what would happen. Looks like I have to resolve some issues before trying it one the client machine. What is the RPCProxy? I thought it was the
ExternalHostName from Get-OutlookAnywhere? I also tried some other combinations:
C:\>RPCPing -t ncacn_http -o RPCProxy=mail.myDomain.org -u 10 -a connect -v 3 -E -P "testUser1,SecretPa$$Word,myDomain," -H 1 -F 3
Invalid BindingOption (RPCProxy=mail.myDomain.org). You must specify the RpcProxy
C:\>RPCPing -t ncacn_http -o RPCProxy=MailServer1.myDomain.local -u 10 -a connect -v 3 -E -P "testUser1,SecretPa$$Word,myDomain," -H 1 -F 3
Invalid BindingOption (RPCProxy=MailServer1.myDomain.local). You must specify the RpcProxy
C:\>RPCPing -t ncacn_http -o RPCProxy=MailServer1 -u 10 -a connect -v 3 -E -P "testUser1,SecretPa$$Word,myDomain," -H 1 -F 3
Invalid BindingOption (RPCProxy=MailServer1). You must specify the RpcProxy
August 23rd, 2010 5:56pm
Authentication settings on virtual IIS folders are as described below, except for EWS which also has
Basic enabled:
http://blogs.technet.com/b/ferris/archive/2010/03/30/default-authentication-settings-exchange-2007-2010-iis-application-virtual-directories.aspx
RPCwithCert = All = Disabled
Free Windows Admin Tool Kit Click here and download it now
August 23rd, 2010 6:24pm
All that for no response?
MSFT - any suggestions?
August 24th, 2010 9:56am
Please browse the URL below, you should get a blank page after authentication
https://mail.MyDomain.org/rpc/rpcproxy.dll
Does the redirection set on the /RPC virtual directory?
Please run the cmdlet below
Get-Outlookprovider EXPR |Fl CertPrincipalName,Server
Get-OutlookAnywhere | FlJames Luo
TechNet Subscriber Support (http://technet.microsoft.com/en-us/subscriptions/ms788697.aspx)
If you have any feedback on our support, please contact tngfb@microsoft.com
Free Windows Admin Tool Kit Click here and download it now
August 25th, 2010 4:22am
Thank you James. Good to hear from you!
----------------------------------------------------
https://mail.MyDomain.org/rpc/rpcproxy.dll
HTTP 500 Internal Server Error
The website cannot display the page.
--------------------------------------------------------------
[PS] C:\>Get-OutlookProvider EXPR | fl CertPrincipalName, server
CertPrincipalName :
Server :
That does not look right ^
[PS] C:\>Get-OutlookAnywhere | fl
ServerName : MS1
SSLOffloading : False
ExternalHostname : mail.myDomain.org
ClientAuthenticationMethod : Basic
IISAuthenticationMethods : {Basic, Ntlm}
MetabasePath : IIS://MS1.myDomain.loc/W3SVC/1/ROOT/Rpc
Path : C:\Windows\System32\RpcProxy
Server : MS1
AdminDisplayName :
ExchangeVersion : 0.1 (8.0.535.0)
Name : Rpc (Default Web Site)
DistinguishedName : CN=Rpc (Default Web Site),CN=HTTP,CN=Protocols,CN=MS1,CN=Servers,CN=Exchange Administr
ative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=myDomain,CN=Microsoft Exchange,CN=Se
rvices,CN=Configuration,DC=myDomain,DC=loc
Identity : MS1\Rpc (Default Web Site)
Guid : 86ea098d-f473-4f71-8f8a-c5e2e95cc74d
ObjectCategory : myDomain.loc/Configuration/Schema/ms-Exch-Rpc-Http-Virtual-Directory
ObjectClass : {top, msExchVirtualDirectory, msExchRpcHttpVirtualDirectory}
WhenChanged : 3/28/2009 6:58:53 PM
WhenCreated : 3/28/2009 6:58:38 PM
OriginatingServer : dc1.myDomain.loc
IsValid : True
-----------------------------------------------------------------
Once again, OWA, for example, is working fine.
And, for a comparison, I can get this - after authentication - (the EWS/exchange.asmx file is accessible as well).
https://mail.myDomain.org/autodiscover/autodiscover.xml
<?xml version="1.0" encoding="utf-8"
?>
-
<Autodiscover xmlns="http://schemas.microsoft.com/exchange/autodiscover/responseschema/2006">
-
<Response>
-
<Error Time="08:24:33.1753685"
Id="1668824430">
<ErrorCode>600</ErrorCode>
<Message>Invalid Request</Message>
<DebugData
/>
</Error>
</Response>
</Autodiscover>
August 25th, 2010 8:55am
As I saw above, “mail.myDomain.org” is the common name in the certificate, right? If so, the output is correct in the EXPR provider
I assume redirection isn’t configured on the virtual directory
Please check the SSL settings of /RPC virtual directory
·
Require SSL (Checked)
·
Require 128-bit SSL (Unchecked)
·
Client certificates: Ignore
Does the error information same no matter whether you browsed the virtual directory internally and externally? Please open the IIS log, check the
sub status code after 500
HTTP 500.x -- Internal Server Error Codes
Per my research, it seems that RPC proxy component has corrupted on the CAS server, please re-install it and test the URL again:
1.
Disable outlook anywhere via EMC
2.
Remove RPC proxy component via PowerShell
3.
Command: servermanagercmd -r rpc-over-http-proxy
4.
Reboot the server
5.
Install RPC proxy component via PowerShell
6.
Command: servermanagercmd -i rpc-over-http-proxy
7.
Enable outlook anywhere
8.
Restart Microsoft active directory Topology service
9.
Check the issue againJames Luo
TechNet Subscriber Support (http://technet.microsoft.com/en-us/subscriptions/ms788697.aspx)
If you have any feedback on our support, please contact tngfb@microsoft.com
Free Windows Admin Tool Kit Click here and download it now
August 26th, 2010 5:19am
As I saw above, “mail.myDomain.org” is the common name in the certificate, right?
I think so:
[PS] C:\>Get-ExchangeCertificate
Thumbprint Services Subject
---------- -------- -------
8606EFxxxxxxxxx62xxxxxxxxxx IP.WS CN=mail.myDomain.org, OU=Domain Control Validated, O=mail.myDomain.org
I assume redirection isn’t configured on the virtual directory
Correct. We did not configure redirction on any virtual directory, including OWA. I realize this is an option but we opted against it: keep things simple - just deploy a desktop shortcut
via GPO so users can access OWA, for example, without entering a long URL (or any URL for that matter). At home, they enter it once, then create a shortcut or add it to favorites.
Please check the SSL settings of /RPC virtual directory
·
Require SSL (Checked) -
SAME
·
Require 128-bit SSL (Unchecked) -
mine is checked, as it is for autodiscover, OWA and others.
·
Client certificates: Ignore -
SAME
James - my settings are underlined above. They are as you recommend, except for 128 bit SSL which, on my server, is checked.
Does the error information same no matter whether you browsed the virtual directory internally and externally?
Yes - I just verified: HTTP 500
This is an example of what I found in the IIS logs - not easy to find things in there - I searched for "500" and found entries like this:
2010-08-23 01:43:46 10.0.x.x RPC_IN_DATA /rpc/rpcproxy.dll MS1.abc.loc:6001 443 - x.x.185.29 MSRPC 500 0 21 45037
2010-08-23 01:43:46 10.0.x.x RPC_OUT_DATA /rpc/rpcproxy.dll MS1.abc.loc:6001 443 - x.x.185.29 MSRPC 500 0 21 45053
2010-08-23 01:44:32 10.0.x.x RPC_IN_DATA /rpc/rpcproxy.dll MS1.abc.loc:6002 443 - x.x.185.29 MSRPC 500 0 21 45037
2010-08-23 01:44:32 10.0.x.x RPC_OUT_DATA /rpc/rpcproxy.dll MS1.abc.loc:6002 443 - x.x.185.29 MSRPC 500 0 21 45037
2010-08-23 01:45:17 10.0.x.x RPC_IN_DATA /rpc/rpcproxy.dll MS1.abc.loc:6001 443 - x.x.185.29 MSRPC 500 0 21 45037
2010-08-23 01:45:17 10.0.x.x RPC_OUT_DATA /rpc/rpcproxy.dll MS1.abc.loc:6001 443 - x.x.185.29 MSRPC 500 0 21 45037
Please open the IIS log, check the sub status code after 500
Am I looking for something after 500? If so, I cannot find any 0 or 21 sub status codes in the link you provided.
Note: I am running Windows 2008 Server SP2, so IIS 7.
Per my research, it seems that RPC proxy component has corrupted on the CAS server, please re-install it and test the URL
again:
I will attempt to schedule the necessary downtime as soon as possible, hopefully Saturday morning.
Thank you so much for your assistance!
By the way, does the information above confirm your idea about a corrupt rpcproxy component?
August 26th, 2010 9:26am
I had some similar issues with Outlook Anywhere recently. For the 500 error, a simple reboot of our Exchange server fixed that. But for the other issues, we also had a problem with .NET Framework v4.0 and ended up having to uninstall .NET 4.0
and then we applied Server 2008 SP2 and also Exchange 2007 SP3.
Free Windows Admin Tool Kit Click here and download it now
August 26th, 2010 6:24pm
Quote: “I cannot find any 0 or 21 sub status codes in the link you provided”
21 is Win32 error code, ERROR_NOT_READY (The device is not ready)
Win32 Error Codes
Quote: “does the information above confirm your idea about a corrupt rpcproxy
component?”
I have seen several similar outlook anywhere cases that appears such error (500 0 21) when browse the RPC proxy component. It seems the possible
cause is the corrupted rpcproxy.dll fileJames Luo
TechNet Subscriber Support (http://technet.microsoft.com/en-us/subscriptions/ms788697.aspx)
If you have any feedback on our support, please contact tngfb@microsoft.com
August 26th, 2010 10:24pm
Thank you James - and Frosty.
We are planning to implement your recommendations as soon as possible.
I will keep you posted.
Free Windows Admin Tool Kit Click here and download it now
August 27th, 2010 8:36am
How's the issue currently?James Luo
TechNet Subscriber Support (http://technet.microsoft.com/en-us/subscriptions/ms788697.aspx)
If you have any feedback on our support, please contact tngfb@microsoft.com
August 31st, 2010 9:46pm
James,
I should be able to attempt the solution you recommended above, this Saturday, 7:00 AM local time.
Of course, I will keep you posted.
Oh yes! I'll do a full server backup (we are on Windows 2008) of the OS drive and a backup of the database (on separate drive) before the "operation".
But this should normally have no effect on the other components, right?
I don't want to make things worse.
Free Windows Admin Tool Kit Click here and download it now
September 2nd, 2010 4:14pm
James,
The solution seems to have worked - 3 out of the 4 users in question are now able to use Outlook Anywhere again.
I have not been able to contact the 4th user to confirm that they too can once again use Outlook Anywhere (on vacation).
As a bonus, it seems to have resolved the OAB problem in my other thread: "Outlook Connectivity- problems receiving email":
http://social.technet.microsoft.com/Forums/en-US/exchangesvradmin/thread/51903d4d-c2ae-4c20-b61b-27efaa34a5d4
Thank you for your help! You were right on the mark!
September 5th, 2010 5:28pm
Awesome : )James Luo
TechNet Subscriber Support (http://technet.microsoft.com/en-us/subscriptions/ms788697.aspx)
If you have any feedback on our support, please contact tngfb@microsoft.com
Free Windows Admin Tool Kit Click here and download it now
September 5th, 2010 9:25pm
so would this fix apply to Outlook 2007 (mapi)?
April 12th, 2011 10:47am
I did have the same problem. Some other programs someone installed fu**** up the application pool setting for DefaultAppPool, the identity was changed, when i replaced it with Networkservice everything works ok.
Free Windows Admin Tool Kit Click here and download it now
September 4th, 2012 2:22pm